Skip to content

Getting Started

This guide walks you from a raw CSV file to a running pipeline in about 5 minutes.

Requirements

  • Python 3.11 or newer
  • uv — the package manager used by this project

Install

git clone https://github.com/tongqqiu/filedge.git
cd filedge
uv sync

The filedge command is now available in the project's virtual environment:

uv run filedge --help

To use it without uv run, activate the environment:

source .venv/bin/activate
filedge --help

Step 1: Inspect your file

Start by pointing filedge inspect at your data file. It samples the first 1,000 rows and produces a ready-to-paste columns: block for pipeline.yaml.

filedge inspect data.csv

Prefer an interactive flow?

filedge author data.csv (install with uv sync --extra authoring) walks you through Steps 1–3 in a terminal UI — schema review, write mode, connector, field encryption, and validation — then writes the pipeline folder for you. See the Author guide.

The columns: block is written to stdout; a per-column confidence summary is written to stderr.

The summary (stderr) flags each column's inferred type and confidence — a leading ! marks anything that is not high confidence:

  order_id                                string, high
! amount                                  float, low, nulls=3
  order_date                              date, high
  customer_name                           string, high

The columns: block (stdout) is ready to paste into pipeline.yaml:

# Generated by filedge inspect
# source: data.csv
# sample_rows: 1000
# timestamp: 2026-05-31 09:00:00
columns:
  - source: order_id
    dest: order_id
    type: string
    required: true
  - source: amount
    dest: amount
    type: float
    required: false
  - source: order_date
    dest: order_date
    type: date
    required: true
  - source: customer_name
    dest: customer_name
    type: string
    required: true

required: is inferred from the sample: a column with no nulls becomes required: true, one with nulls becomes required: false. Review columns marked low or ambiguous before using them in production. See the Inspect guide for details.

To write the output directly to a file:

filedge inspect data.csv --output pipeline.yaml

Step 2: Complete the config

filedge inspect produces a columns: block. Wrap it in a full pipeline.yaml:

format: csv
dest_table: orders
write_mode: append

connector:
  type: sqlite
  url: sqlite:///orders.db

columns:
  - source: order_id
    dest: order_id
    type: string
    required: true
  - source: amount
    dest: amount
    type: float
    required: true
  - source: order_date
    dest: order_date
    type: date
    required: false
  - source: customer_name
    dest: customer_name
    type: string
    required: true

See the pipeline.yaml reference for every available option.

Step 3: Validate the config

Before writing any data, dry-run the file against your config:

filedge validate data.csv --config pipeline.yaml

Exit code 0 means clean; exit code 1 means failures were found:

✓ 1000 rows checked, no failures

Or with failures:

✗ row 42  amount  cannot coerce 'n/a' to float
✗ row 87  amount  cannot coerce '' to float (required)
2 failure(s) in 1000 rows checked

Fix the source data (or adjust required: false in the config) until validation is clean. See the Validate guide for more options.

Step 4: Run the pipeline

Place your files in an incoming directory and run:

filedge run --dir ./incoming --config pipeline.yaml --audit-db-url sqlite:///filedge.db
# Committed: 1  Failed: 0  Skipped: 0  New: 1  Reclaimed: 0  Retried: 0

The one-line summary above is printed to stdout. filedge run also emits structured progress logs (one JSON object per line) to stderr — handy in production, noisy at the prompt. Redirect with 2>/dev/null to see only the summary, or pass --log-format text for human-readable logs. See the Run guide.

--audit-db-url can also be set via the FILEDGE_AUDIT_DB_URL environment variable.

Check status any time:

filedge status --audit-db-url sqlite:///filedge.db
# PENDING:    0
# PROCESSING: 0
# COMMITTED:  1
# FAILED:     0

Previewing rows

If validation reports a bad row, jump straight to it without opening the file in an editor:

filedge preview data.csv --start-row 42 --rows 5

See the Preview guide for details.

Other file formats

This walkthrough used CSV, but Filedge handles NDJSON, Parquet, Excel, and fixed-width files the same way — every read command detects the format from the file extension. Parquet and Excel need a one-line optional extra. See File formats for the full matrix and gotchas.

Next steps