Response caching layer for knowledge base AI chat

Support teams using AI chat on top of static or slow-moving knowledge bases — documentation, product FAQs, policy documents — watch their customers wait 30–180 seconds for answers to questions that were asked and answered identically yesterday. The complaint 'it can be slow, although understandable as it does reference a few informational inputs' captures the resignation perfectly: users have normalized the wait because nobody has given them a reason not to.

The structural reason this hasn't been built is that AI chat vendors bundle retrieval and generation together as a single product experience, and separating them architecturally would require them to admit that most answers to common questions don't need a fresh LLM call every time. Caching semantically similar queries is awkward to retrofit into a product designed to feel 'intelligent' on every response. There's also a real technical challenge: exact-match caching is useless for natural language, and semantic caching requires embedding queries and matching against a cache index — which is a non-trivial infrastructure problem vendors haven't prioritized because it makes their latency stats look better without adding billable AI calls.

What's missing is a transparent caching proxy that intercepts queries before they hit the LLM, embeds the query, checks against a semantic cache of prior responses with a configurable similarity threshold, and returns the cached answer in under 200ms when the match is strong enough. The cache invalidates when the underlying knowledge base is updated. Teams get sub-second responses for the 60–80% of queries that repeat, and real LLM calls only for genuinely novel questions.

For a company running self-service chat at scale, this directly reduces LLM API costs and cuts the 'more than 3 minutes' complaints to near-zero for common queries. The need recurs every time the query volume grows or a new AI chat product is adopted.

What to build

Build a reverse-proxy service that intercepts outbound queries to any LLM-backed chat API, performs semantic similarity matching against a maintained cache of prior query-response pairs using configurable cosine similarity thresholds, returns cached responses instantly for near-duplicate queries, and auto-invalidates cache entries tied to updated knowledge base sections via webhook.

Where to start

Target teams already paying significant monthly LLM API bills who can see the cost reduction immediately in their billing dashboard — cost savings justify the integration before the latency improvement even needs to be argued.

The hard part

Setting the semantic similarity threshold is genuinely hard — too tight and the cache rarely hits, too loose and customers get wrong answers for slightly different questions, which is worse than slowness; getting this tunable default right for the first few customers requires enough real query data to calibrate against.

How it makes money

Usage-based pricing on cache hits served — free up to a threshold (e.g., 10,000 cached responses/month), then a per-thousand-hit fee above that, so the product pays for itself from LLM cost savings before the customer pays a dollar.