Ask an LLM the same question twice, worded two different ways, and it will happily do the exact same expensive work twice. No memory, no shortcut, just another full inference pass — because as far as the model's concerned, it's never seen this exact string before.
That's the itch that led me to build Vahini: a local semantic caching layer that sits in front of LLM inference and asks a different question than the usual cache. Not "have I seen this exact string?" but "have I seen something close enough in meaning that I can safely reuse the answer?"
It worked. And then it worked a little too well, in a way that taught me more than the part that actually succeeded.
The Win
The first real test was almost boring in how well it went.
I sent: "What is the capital of Maharashtra?" — cache miss, full inference pipeline, 7575 ms.
Then I sent a differently worded version of the same question: "Tell me what the capital city of Maharashtra is." — cache hit, similarity 0.9486, 27.56 ms.
That's roughly a 275× speedup, and the interesting part wasn't even the number — it was that the cache correctly understood these were the same request despite sharing almost no exact wording. Semantic similarity was doing exactly what I built it to do.
For about a day, I thought the hard part was done.
The Part That Broke It
Then I tried two requests that looked, structurally, exactly like a cache-hit pair:
"India population?"
"India population in 2000?"
Measured similarity: 0.8553. My threshold at the time was 0.85. So the cache did what it was designed to do — it matched them — and returned the cached answer for current population to a question that was explicitly asking about the year 2000.
It was confidently, silently wrong. And it passed every test I'd been running up to that point, because every test I'd been running was checking "does the cache recognize similar phrasing?" — not "does the cache recognize when similar phrasing still requires a different answer?"
Those turned out to be completely different questions.
Why This Actually Matters
Here's the uncomfortable generalization: semantic similarity measures closeness of meaning, not factual equivalence. A cosine similarity score has no concept of "this one word changes everything." A year, a unit, a location, a qualifier like "latest" or "in USD" — any of these can flip the correct answer entirely while barely moving the similarity score at all.
I bumped the threshold from 0.85 to 0.90, and that specific case got fixed — the 2000 query correctly triggered fresh inference instead of serving the stale match. But I want to be honest about what that fix actually was: it wasn't a solution, it was a boundary shift. A higher threshold doesn't teach the cache what a temporal constraint is. It just makes the cache pickier about matching in general, which trades away some of the hit rate I'd built the whole thing to gain. There will always be some pair of requests that reads as similar to the model but isn't safe to treat as identical — no single number fixes that.
The Reframe
This is the part I didn't expect going in: I'd been thinking about this as one problem — "make the cache good." It's actually two, and they don't move together.
Hit rate is a performance problem. Correctness is a trust problem. A cache that's blazing fast and wrong a meaningful fraction of the time isn't a fast cache — it's a fast way to erode trust in the system serving it. Optimizing one doesn't automatically improve the other, and in this case, they were actively in tension: the threshold that gave me the best hit rate was also the threshold that let a wrong answer through.
That's a more useful lesson than the 275× number, honestly. It's easy to build something that's fast. It's a different, harder problem to build something that's fast and knows the difference between "similar enough to reuse" and "similar enough to fool you."
Where It's Headed
The fix isn't a better threshold — it's giving the cache something the raw embedding doesn't have: awareness of the specific words that change an answer. That means detecting entities and constraints explicitly (dates, units, "latest," "today"), treating time-sensitive queries as a separate class from reusable ones, and layering keyword or metadata checks on top of the similarity score instead of trusting it alone.
The goal was never to maximize cache hits. It's to maximize the ones that are actually safe to serve.
Code's here if you want to see how it's put together: github.com/Hedau0404/Vahini