CAELUM
← Blog

Caching LLM answers by similarity returns wrong answers. Here is the fix.

By Issouf KANAZOE

If your app calls an LLM, you are probably paying twice for the same answer. The same support question from different customers. The same document summarized again. The same prompt in every CI run. Each call is billed per token, and most of them are repeats.

The obvious fix is a cache. The non-obvious part is that the most popular way to do it — semantic caching — quietly returns wrong answers if you build it the naive way. I learned this by measuring it. Here are the numbers, and what we ended up doing instead in CAELUM, an open-source Python library.

Level 1: the exact cache

Start simple. If the exact same request comes in — same messages, same model, same parameters — return the stored answer. No model call, no cost.

from caelum import OpenAI   # instead of: from openai import OpenAI

client = OpenAI(cache=True)

On a benchmark with 30% repeated requests, this removed 30% of the API calls, with under 1 ms of overhead at p95 on a miss. It is safe because "same input, same output" is hard to argue with. But it only catches exact repeats. "Do you deliver to Bobo-Dioulasso?" and "do you deliver to Bobo?" are different strings, so the exact cache misses them.

Level 2: the semantic cache, and the trap

The tempting next step: embed each question into a vector, and if a new question is close enough to a stored one (high cosine similarity), reuse that answer. This is what most "semantic cache" tutorials show.

It sounds great and it is dangerous. Embeddings measure topic similarity, not same meaning. Two questions can be neighbors in vector space and still have opposite answers.

We built a test set of 138 real question pairs: 69 paraphrases that should share an answer, and 69 look-alikes that must not — questions that are close in wording but different in meaning, like:

Then we tuned the similarity threshold and measured what a pure-embeddings cache would do:

Similarity threshold Paraphrases reused Wrong answers (false positives)
0.97 (strict) ~0% 0%
0.90 some 3%
0.85 more 19%

There is no good setting. Raise the threshold and the cache never fires, so you save nothing. Lower it and it starts confidently serving the Ouaga answer to the Bobo customer. A 19% wrong-answer rate is not a cache; it is a liability.

Level 3: add a cheap verifier

The fix is to stop trusting the embedding as the final word. Use it only as a fast filter to find candidates, then ask a small, cheap model one yes/no question: do these two questions mean the same thing? Only reuse the answer if it says yes.

With a small judge model (gpt-4o-mini) confirming each match, on the same 138 pairs:

That is the whole point: you keep most of the savings and you stop serving wrong answers. The test set is replayed offline on every CI run, and any regression fails the build.

The part the tutorials skip: the verifier costs money too

Calling a judge model on every candidate is not free. If your answers come from an expensive model, the judge is a rounding error and you win big. If your answers come from a cheap model, the judge can eat most of the savings.

So the honest number is not "hit rate" — it is net savings: money avoided, minus the judge and embedding costs. CAELUM's journal reports exactly that, per request, so you can see whether the cache actually pays for itself for your traffic instead of assuming it does.

And some things should never be cached

A cache that reuses answers needs guardrails, or it will reuse something personal or time-bound. By default, a question is never cached (and its text is kept nowhere) if it contains an order or tracking number, a phone number, an email, an amount of money, a date or time ("tomorrow", "before 5pm"), or a secret like an API key — and follow-up questions inside a conversation are never cached at all. You keep control of the rules per use case.

Putting it together

from caelum import OpenAI

client = OpenAI(
    cache=True,          # exact cache
    semantic=True,       # similarity + verifier
    retry=True,          # automatic retries
)

# Same OpenAI objects, same static types. Your keys, your bill.
# CAELUM never resells tokens — your app calls the provider directly.

CAELUM is open source (Apache 2.0) and works with the OpenAI, Anthropic, Google Gemini and Mistral SDKs. If you want the shared-cache, per-customer cost and budget-alert parts for a team, there is an optional hosted plan, but the library above runs fully on its own.

pip install "caelum-ai[openai]"

The code, the 138-pair benchmark and the full numbers are on GitHub. If you try it, tell me where it breaks — I read every issue.