How-to guides

Task-focused recipes for getting a specific job done. If you're brand new, run a Tutorial first; if you're looking up a config option, jump to the Reference.

Build a pipeline

Turn a sample file into a validated pipeline.yaml.

• Author a pipeline — interactive terminal UI: infer, review, choose write mode and connector, validate
• Inspect a file — sample a file and generate a columns: block
• Preview a file — view rows as a table, jump to any row
• Validate a file — dry-run a file against a config before writing data

Run & operate

Ingest files and keep the pipeline healthy in production.

• Run a pipeline — retry-safe commits, write modes, scheduling
• Scale ingestion — large files, many files, parallel workers, backfills
• Compact small files — merge many small files into fewer large ones
• Requeue failed files — move terminal failures back to PENDING after remediation
• Quarantine bad rows — let good rows land, set bad rows aside, investigate and re-drop them
• Healthcheck — probe the audit DB and destination without writing rows
• Observability — logs, metrics, and tracing
• Deploy Filedge — container image, docker-compose, and Kubernetes CronJobs
• Export an audit site — generate a read-only HTML site for audit stakeholders

Connect sources

Bring upstream systems in through the same File contract.

• API sources — the Fetcher pattern and the filedge-fetch reference companion
• Stripe API to DuckDB — local golden path for API Source → File → audited load
• API Source adapters — extend the Fetcher to a new API
• Queue sources — the Materializer pattern and the filedge-materialize reference companion
• Source manifests — upstream lineage for API / Queue / SFTP / vendor exports
• CDC files — apply change-data-capture files as SCD Type 1 changes

File formats

• Overview — CSV, NDJSON, Parquet, Excel, and fixed-width at a glance
• Fixed-width files — ingest fixed-width text with a declared column layout