Checkpoint-based ETL recovery for data engineers

The moment that triggers the search for something better: a pipeline fails three hours into a six-hour sync, and the only option is to restart from zero. The engineer watches the same data get re-pulled, re-transformed, and re-loaded — wasting compute, wasting time, and in some cases causing duplicate records downstream. This happens constantly, and yet every major ETL tool treats it as an edge case.

The gap persists because the ETL vendors' incentive is to make the happy path fast and marketable. Failure recovery is unglamorous, hard to demo, and disproportionately expensive to build into a generic data movement architecture. The buyer — usually a VP of Data or a platform engineering lead — often doesn't feel the pain directly. The data engineer who reruns the job at 2am is the one who suffers, and they don't control the budget.

What existing tools get wrong: they either offer no checkpointing at all (so a failed job at step 47 of 50 restarts at step 1), or they offer coarse-grained retries with no awareness of what data was already successfully written downstream. The complaint 'needs to redo the whole process again' is not a bug report — it's a description of how the architecture works by default.

This is a business and not a feature because the cost is concrete and recurring: failed pipelines mean delayed reporting, stale dashboards, and engineering time spent babysitting reruns rather than building. A team running 50+ pipelines a day will hit this multiple times a week. The need doesn't go away — it compounds as data volume grows, which is exactly when 'too much data, too many processes' becomes the dominant complaint.

What to build

Build a middleware layer that sits between any JDBC or REST-based ETL source and destination, tracks successful batch checkpoints to a lightweight state store, and on failure resumes the job from the last confirmed successful offset rather than re-pulling from the source.

Where to start

Target teams already using a specific orchestrator like Airflow or Prefect, where you can hook into the existing task failure callbacks and offer checkpoint recovery as a drop-in operator — no pipeline rewrites, just a wrapper they install.

The hard part

Proving the checkpoint state is actually consistent with what landed downstream — if the destination write succeeded but the acknowledgment was lost, replaying from that checkpoint causes duplicates, which is arguably worse than the original failure.

How it makes money

Monthly subscription priced per number of monitored pipeline runs, with a free tier capped at 5 pipelines to drive adoption among individual engineers who then advocate upward.