In-flight transformation layer for small and mid-size streaming pipelines

Engineers running Kafka or Kinesis repeatedly describe the same frustration: by the time data reaches the destination system — a data warehouse, a database, an API — it's in the raw format the producer sent. Any enrichment, filtering, type coercion, or field renaming has to happen either at the producer (which means coupling application code to downstream concerns) or in a separate batch job after ingestion (which defeats the point of streaming).

The core complaint is direct: 'there is not much transformation I am able to perform before I store the data in destination systems.' Flink's DataStream API can technically do this, but the documentation for anything beyond basic operations is sparse and the examples for broadcast streams or process functions are nearly absent — meaning teams spend weeks figuring out something that should take days.

The reason this isn't fixed inside the existing tools is structural. Kafka is a durable log; transformation is intentionally out of scope. Flink handles complex transformations but assumes you have engineers who can write Java or Scala stream processing jobs — and its documentation gap is a known community complaint, not something the project maintainers have prioritized closing. There is no middle ground: either you write full Flink jobs or you push untransformed data to your destination and deal with it there.

What's missing is a configuration-driven transformation layer — not a visual drag-and-drop toy for analysts, but a YAML or JSON-defined set of per-topic rules (filter by field value, rename fields, parse nested JSON, enrich from a small reference lookup) that runs as a managed process between your Kafka topic and your destination. Small and mid-size teams don't need Flink's full power; they need 80% of the use cases handled without writing streaming jobs.

This is a business because every new data source a company adds creates the same transformation problem again. It's not a one-time setup; it's a recurring operational cost that compounds as pipelines multiply.

What to build

Build a lightweight process that reads from Kafka or Kinesis topics, applies per-topic transformation rules defined in YAML (field filtering, renaming, type casting, simple lookups against a Redis or Postgres reference table), and writes the transformed output to a destination topic or sink — deployable as a single Docker container with no JVM dependency.

Where to start

Target teams migrating off a managed ETL service who already have Kafka and need transformation logic that doesn't require rewriting their pipeline from scratch — they have an immediate, time-boxed need and a concrete comparison point for cost.

The hard part

The transformation rule language needs to be expressive enough to cover real use cases but simple enough that it doesn't become a second programming language to maintain — getting that balance wrong in either direction kills adoption.

How it makes money

Flat monthly fee per pipeline (defined as a source topic plus destination), with pricing tiers based on message volume — starts cheap enough for small pipelines so teams adopt it broadly, then expands as volume grows.