Ledger ingest pipeline

A batch ingest service that takes fixed-width and CSV settlement files from a handful of upstream providers and lands them in Postgres as a single normalised ledger. The brief was unglamorous: the existing importer was a cron job over a shared NFS mount that failed silently and left partial days in the database.

What it does

Files arrive in object storage. A worker claims one with an advisory lock, parses it into typed rows, and writes them inside a single transaction per file, so a malformed line at row 40,000 rolls the whole file back rather than leaving half a day behind. Every file carries a content hash, and re-uploading the same file is a no-op rather than a duplicate.

• Rust, with tokio for the worker pool and sqlx for queries checked at compile time
• Postgres as the queue as well as the store: SELECT ... FOR UPDATE SKIP LOCKED, no separate broker
• An axum admin endpoint for replaying a date range
• Structured logs and per-file timings exported to the metrics stack that was already there

Throughput was never the hard part. Roughly two million rows a day fits comfortably in a single writer. The hard part was making failure boring: each run records what it read, what it rejected, and why, so a mismatched total can be traced to a line number without opening a shell on the box.

The reconciliation query that used to take minutes now runs against a covering index and finishes well inside a second, which is what made the daily close practical to automate.