Persistent message store for safety-critical and compliance-bound event pipelines

Several of the complaints cluster around a specific failure mode that stream processors handle badly by design: messages disappear. Kafka deletes messages after a retention window. Kinesis data expires after 24 hours by default (up to 7 days with extended retention). If a consumer goes down during that window and doesn't recover in time, the data is gone. One user described exactly this: 'data will vanish if not consumed for a long time, making it unsuitable for safety-critical systems like medical systems.' Another flagged 'inability to have persistent messages in case of breakage of communication between the services.'

This is a structural problem, not a bug. Kafka and Kinesis are designed for high-throughput ephemeral consumption — infinite retention is architecturally expensive and not their value proposition. The existing workaround is to dump everything to S3 or a database in parallel, but that adds operational complexity, doesn't replay in order, and requires the consuming service to implement its own recovery logic.

The gap is sharpest in regulated industries — medical device telemetry, financial transaction events, industrial sensor data — where losing a message isn't just an inconvenience, it's a compliance or safety incident. These are exactly the buyers who will pay for reliability guarantees rather than negotiate on price. But they're also exactly the buyers that general-purpose stream processors don't design for, because the typical stream processing buyer doesn't have those constraints.

What's needed is a durable message store that sits between Kafka or Kinesis and downstream consumers, acknowledges messages only when the consumer has confirmed processing, holds undelivered messages indefinitely with configurable replay, and produces an audit log of every delivery attempt and acknowledgment. This is a narrow, boring-sounding product — and that's exactly why it's a real business. The regulatory buyer has a compliance requirement that forces them to solve this problem regardless of cost.

What to build

Build a message broker proxy that subscribes to Kafka or Kinesis topics, stores each message in a durable append-only log with indefinite retention, tracks per-consumer delivery and acknowledgment state, and exposes a replay API that lets a recovering consumer reprocess from any point in the sequence — with a per-message audit trail exportable as JSON or CSV.

Where to start

Start with medical device companies that already use Kafka for telemetry but have been told by their compliance team they cannot rely on Kafka retention for audit purposes — they have a documented gap and budget pressure to close it before an audit.

The hard part

Selling into medical or financial compliance requires the product itself to meet regulatory standards (SOC 2, HIPAA BAA, possibly FDA 21 CFR Part 11 for audit trails) before the first enterprise customer will sign — that certification process is expensive and slow relative to a typical early-stage product timeline.

How it makes money

Annual contract priced per number of topics covered plus per-gigabyte retention, with a mandatory onboarding fee that covers compliance documentation — this filters out buyers who aren't serious and funds the certification work.