Automatic data tiering for analyst-owned datasets

The moment this becomes a real problem: an analyst has built six months of reports on a dataset that lives in their BI tool's import cache, and one morning the data is just gone — or truncated — because the tool hit a storage limit or applied a retention policy they didn't set and didn't know existed. One complaint here is exactly this: 'some data disappears after a set time, which prevents long-term analyses.' Another: '1 GB storage space by default is too low for big data usage.' These aren't edge cases — they're what happens when analysts accumulate data over time inside tools that were never designed for long-term storage.

The structural reason this gap exists: BI tools treat data storage as a secondary concern, something the data team should have solved upstream. But in practice, many analysts work with data they've imported, transformed, or assembled themselves, outside of a formal data pipeline. Nobody owns this layer. Data engineering owns the warehouse; the analyst owns the dashboard; and the middle layer — how data is cached, retained, and tiered — belongs to no one.

Existing workarounds are painful: manually exporting to S3 and reimporting, paying for more SPICE capacity as a band-aid, or rebuilding reports when data disappears. None of these are systematic, and all of them pull analysts away from analysis.

This is a business because data volumes grow in one direction. Every analyst who works with time-series data or accumulating event data hits this ceiling eventually, and hits it again every time their dataset grows. The cost of not solving it is either recurring engineering time or lost institutional knowledge when historical data quietly disappears.

What to build

Build a lightweight data retention manager that sits between an analyst's BI tool and their cloud storage, automatically tiering imported datasets to cheap object storage when they exceed configurable size thresholds, and transparently rehydrating them on query — with a UI that shows exactly what data is stored where and when it will expire.

Where to start

Start with QuickSight users specifically, where SPICE storage costs are explicit and quantified on AWS bills — making the pain financially legible and the ROI of tiering immediately calculable, which shortens the sales conversation.

The hard part

The hardest early challenge is that 'transparent rehydration on query' is easy to promise and hard to make fast enough that users don't notice the latency — the moment someone opens a dashboard and waits 30 seconds for tiered data to reload, they'll blame the product for the exact problem it was supposed to solve.

How it makes money

Usage-based pricing on data moved and stored, with a small fixed monthly minimum — mirrors how the buyer already thinks about cloud storage costs and makes the invoice self-justifying when it saves more in SPICE fees than it charges.