Overview
What happens when an LLM receives the same — or semantically similar — question repeatedly?
The model performs inference again, even when the answer could have been reused.
That was the problem behind Vahini.
Vahini is an experiment in building a local semantic caching layer for LLM inference. Instead of relying on exact string matches, it uses semantic similarity to decide whether an incoming request is close enough to a previous one to reuse its response.
The goal was simple: reduce unnecessary inference without changing the underlying model.
The Problem
Repeated LLM inference has two obvious costs:
- Latency — every request waits for a full model inference pass.
- Compute — the model does work even when the same intent has already been handled.
The first inference tests gave me a baseline:
- First inference: 5483ms
- Second inference: 3289ms
- Third inference: 2957ms
The request pipeline itself worked fine:
Client → FastAPI → Ollama → Qwen → Response
But repeated inference was the next bottleneck to solve.
Why Semantic Caching?
Before choosing semantic caching, a few approaches were ruled out.
Exact cache — simple, but too strict. Two prompts can have identical intent while using completely different wording.
Normalization — better, but still limited. It cannot reliably capture meaning.
LLM judge — could work, but it means another inference call, which partly defeats the purpose.
No cache — simplest of all. Every request just repeats the same work.
Semantic similarity felt like the middle ground.
Instead of asking:
"Are these strings identical?"
the cache asks:
"Are these requests semantically similar enough to reuse the result?"
Architecture
The initial architecture kept the layers deliberately separate:
Client → FastAPI → Ollama → Qwen → Response
- FastAPI handles the API layer.
- Ollama handles local model serving.
- Qwen handles generation.
That separation makes it possible to add caching, routing, and monitoring in front of inference without touching the model itself.
The current architecture extends that idea:
Client → FastAPI → Semantic Cache → Ollama → Qwen → Response
On a cache hit, the flow is simpler:
Client → FastAPI → Semantic Cache → Cached Response
Semantic Cache Flow
Results
The first meaningful semantic-cache test showed a significant reduction in latency.
Cache Miss
Query
"What is the capital of Maharashtra?"
Result: MISS — 7575.43ms
The request went through the full inference pipeline.
Cache Hit
A semantically equivalent query was then sent:
"Tell me what the capital city of Maharashtra is"
Result: HIT — similarity 0.9486 — 27.56ms
The cached response was returned without another LLM inference.
Latency Comparison
| Request | Result | Similarity | Latency | | ----------------------------------------------- | ------ | ---------: | --------: | | What is the capital of Maharashtra? | MISS | — | 7575.43ms | | Tell me what the capital city of Maharashtra is | HIT | 0.9486 | 27.56ms |
This resulted in roughly a 275× reduction in latency for this cache-hit case.
The important result was not just the speed. The cache correctly recognized two differently worded questions as the same underlying request.
Similarity Threshold
The initial similarity threshold was 0.85.
The trade-off was:
- Lower threshold → more cache hits, but greater risk of wrong matches.
- Higher threshold → safer matches, but potentially fewer cache hits.
Rather than assuming there was a universally correct value, the threshold was tested against real queries.
Issues & Failure Cases
The biggest discovery was:
A cache hit does not necessarily mean a correct answer.
Consider these two requests:
"India population?"
and:
"India population in 2000?"
The second query needs a different answer because it contains a temporal constraint that the first query does not.
However, the measured similarity between them was 0.8553.
With the threshold at 0.85, the second query was considered similar enough to receive the cached response.
This exposed a fundamental problem:
Semantic similarity measures closeness of meaning, not factual equivalence.
A small detail — such as a year, location, unit, or other constraint — can completely change the correct answer while the overall similarity remains high.
Threshold Experiment
To reduce this type of false cache hit, the threshold was increased:
0.85 → 0.90
After increasing the threshold, the query:
"India population in 2000?"
was no longer served from the cached response and instead triggered fresh LLM inference.
This fixed the specific case.
However, it also revealed another important lesson:
A higher similarity threshold does not guarantee correctness.
It only changes the boundary for accepting a semantic match.
There will still be edge cases where two requests appear highly similar but require different answers.
Lessons Learned
Semantic Similarity ≠ Semantic Equivalence
A high similarity score does not prove that two requests can safely share a response.
Similarity should be treated as a signal, not a correctness guarantee.
Small Constraints Matter
Words and phrases such as:
- "in 2000"
- "latest"
- "today"
- "for India"
- "in USD"
can completely change the expected answer.
A caching system needs to account for these constraints rather than relying only on overall similarity.
Performance and Correctness Are Different Problems
The Maharashtra test showed a significant latency improvement:
7575.43ms → 27.56ms
But the India population test showed that an aggressive cache can return a confidently wrong answer.
The system therefore needs to optimize both:
Cache hit rate + Answer correctness
A cache that is extremely fast but frequently wrong is not useful.
Threshold Tuning Is a Trade-off
There is no universally correct similarity threshold.
The appropriate value depends on the actual workload and the acceptable balance between cache hits and correctness.
Current Direction
Semantic caching clearly reduces repeated-inference latency, but safe caching requires more than a single similarity threshold.
The next direction is to make cache decisions more context-aware.
Potential improvements include:
- Detecting important entities and constraints.
- Separating time-sensitive queries from reusable ones.
- Combining semantic similarity with keyword or metadata checks.
- Adding cache invalidation for changing information.
- Measuring false-hit and false-miss rates.
- Evaluating cache hit rate across a larger query dataset.
- Adding monitoring around similarity scores and cache decisions.
The broader goal is not to maximize cache hits.
It is to maximize useful cache hits.
Key Features
- Semantic caching — Matches requests by meaning rather than exact text.
- Low-latency cache hits — Tested hits return in tens of milliseconds.
- Local inference — Uses Ollama and Qwen locally.
- Privacy-first architecture — Requests and cached data can remain within the local environment.
- OpenAI-compatible API — Exposes
/v1/chat/completions. - Model-independent caching layer — Caching sits before inference rather than modifying the model.
- Telegraph Protocol support — Includes a
miner.yamlmanifest for protocol integration.
Tech Stack
- Python 3.10+
- FastAPI — API layer
- Uvicorn — ASGI server
- Ollama — Local model runtime
- Qwen — Local language model
- Vector similarity search — Semantic cache
- Telegraph Protocol — Node specification and manifest
API Compatibility
Vahini exposes an OpenAI-compatible endpoint:
POST /v1/chat/completions
This allows Vahini to sit between existing LLM clients and the local inference layer.
The Idea
Vahini started with a simple question:
What if an LLM gets the same, or semantically similar, question repeatedly?
The experiment showed two things.
First, semantic caching can produce a dramatic latency improvement when the match is correct.
Second, semantic similarity alone is not enough to guarantee that a cached answer is safe to reuse.
That tension is the core engineering problem Vahini is exploring:
How can we avoid unnecessary inference while preserving answer correctness?
Repository
The complete implementation is available on GitHub: