Event-driven pipeline triggers without an orchestrator

The moment: a data engineer realizes their ETL tool only fires jobs on a cron schedule, but the business wants data loaded within minutes of an upstream event — a file landing in S3, a webhook firing, a database row changing. They either bolt on a full orchestration layer like Airflow to handle what should be a simple trigger, or they accept that their 'real-time' pipeline is actually running every 5 minutes and hoping for the best.

This gap persists because ETL tools are historically built around the batch paradigm — scheduled pulls at fixed intervals — and event-driven triggering is architecturally different enough that it requires a separate runtime model. Vendors add it late and poorly because their core scheduling engine isn't designed for it. The complaint is direct: 'it only supports time-based flows (scheduled/cron)' and 'limited control over delivery timing, affecting real-time delivery systems.' The people who feel this most are analytics engineers at companies where the data team is small and cannot justify standing up a full orchestration stack just to get sub-minute pipeline triggers.

What makes this worse: when teams do bolt on an orchestrator to fill the gap, they now have two systems to debug when something fails. The orchestrator fires the trigger, the ETL tool runs the job, something goes wrong in the middle, and neither system has the full picture of what happened.

This is a business and not a feature because the trigger logic itself is simple, but maintaining reliable delivery across cloud events, webhooks, database change streams, and file system watchers — and surfacing failure state clearly when a trigger fires but the downstream job doesn't complete — requires ongoing work that no small data team wants to own themselves. The need grows with every new data source the company adds.

What to build

Build a hosted trigger router that listens to S3 events, webhooks, Postgres logical replication streams, and SNS/SQS messages, maps each event type to a configured Airbyte sync or NiFi flow via their existing APIs, and maintains a per-trigger audit log showing whether the downstream job completed, failed, or was skipped due to a duplicate event.

Where to start

Launch with S3-to-Airbyte triggers only, targeting teams who already use Airbyte Cloud and have complained on the Airbyte forum about the lack of event-based syncs — a public, searchable, pre-qualified audience with the exact problem.

The hard part

Exactly-once delivery is genuinely hard — if the trigger fires but the job acknowledgment is delayed, you risk double-triggering, and the edge cases multiply fast across different event sources, making the reliability guarantee the entire product's credibility.

How it makes money

Usage-based pricing per trigger event processed, with a free tier of 10,000 events per month to cover small teams, expanding to volume tiers that grow naturally as the customer's data pipeline activity grows.