Building Kshipra: Why LLM Inference Belongs on Average Consumer Hardware

September 21, 2026

Most modern LLM inference guides start with an implicit assumption: "First, spin up an NVIDIA A100, or rent a cloud cluster."

That assumption quietly excludes a huge chunk of the people actually trying to learn this stuff — students, hobbyist engineers, and developers running the kind of hardware most of us actually own: an 8GB RAM laptop with no dedicated VRAM. High-end accelerators genuinely matter for training and massive-scale serving, but defaulting to them creates a barrier that has nothing to do with whether someone understands transformers.

Cloud APIs solve the access problem, but they solve it by hiding the mechanics. When inference happens behind an endpoint, you never have to think about how auto-regressive decoding actually works, what a KV-cache is doing under the hood, or why your process's memory footprint just spiked. You get an answer, and the machinery disappears with it.

I built Kshipra (क्षिप्रा — Sanskrit for "swift") to push back on both problems at once. It's a modular, CPU-optimized causal LLM inference engine, written from scratch in Python and PyTorch, with two goals:

  1. Democratize local AI serving — build a lightweight, self-sustaining inference runtime that works strictly within consumer hardware limits: 8GB of RAM, no GPU, no CUDA.
  2. Learn inference at a systems level — move past black-box wrappers like vLLM or Ollama, and manually manipulate attention states, position embeddings, and memory allocators myself.

The Philosophy: Efficiency Over Unlimited VRAM

When you have 80GB of High Bandwidth Memory, inefficient code gets quietly absorbed by raw hardware throughput — you can afford to be sloppy and never notice. When you have 8GB of system RAM that your OS and everything else running on the laptop is also fighting over, every tensor allocation, every matmul, every footprint spike matters, and shows up immediately.

That constraint forces you to actually answer questions most GPU-first tutorials skip past entirely:

  • How much memory do the model weights actually consume, versus the runtime intermediate states — activations, KV-cache, attention scores?
  • Where does the real latency bottleneck live during token generation: is it compute-bound, or memory-bandwidth-bound?
  • How do you maximize arithmetic intensity when you don't have CUDA kernels to lean on?

Kshipra exists to answer those questions empirically, one optimization technique at a time, instead of assuming the answers the way a lot of tutorials do.

Project Structure & Architecture

To isolate each optimization and measure its impact cleanly, I structured Kshipra as a clean engine with an automated benchmarking harness sitting on top of it:

Kshipra/
├── engine/
│   ├── __init__.py
│   ├── baseline.py           # Pure O(N^2) autoregressive generation
│   ├── kv_cache.py           # O(1) step-wise KV-cache persistence
│   ├── batching.py           # Static batched execution with positional fixes
│   ├── dynamic_batching.py   # Asynchronous request queueing
│   ├── streaming.py          # Generator-based token streaming
│   ├── early_exit.py         # Confidence-gated layer exit
│   ├── speculative.py        # Draft/Target speculative decoding
│   └── quantization.py       # INT8 dynamic layer quantization
├── benchmarks/
│   └── run_bench.py          # Unified benchmarking harness (TTFT, RSS RAM, throughput)
├── latency_comparison.png    # Automatically generated benchmark visual
├── requirements.txt
└── README.md

Each file under engine/ is a self-contained implementation of one optimization technique, so I can benchmark them against each other directly instead of guessing at relative impact. This post covers baseline.py — the starting point everything else in the series improves on.

The Starting Point: Baseline Autoregressive Generation

engine/baseline.py implements standard autoregressive generation with zero caching and zero batching tricks — as close to "the textbook definition" as I could make it, on purpose.

Autoregressive models generate text token-by-token. To produce token $N+1$, the model takes all previous tokens $1 \dots N$, runs a full forward pass through every transformer layer, and reads the logits off the final position via greedy (argmax) sampling.

import time
import psutil
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

class BaselineGenerator:
    def __init__(self, model_name: str = "gpt2"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name).to("cpu")
        self.model.eval()

    @torch.no_grad()
    def generate(self, prompt: str, max_new_tokens: int = 20) -> str:
        input_ids = self.tokenizer(prompt, return_tensors="pt")["input_ids"]
        generated_ids = input_ids.clone()

        for step in range(max_new_tokens):
            # The bottleneck: re-running the entire sequence through the model
            outputs = self.model(input_ids=generated_ids)
            next_token_logits = outputs.logits[:, -1, :]
            next_token = torch.argmax(next_token_logits, dim=-1, keepdim=True)

            # Append the new token to the context window
            generated_ids = torch.cat([generated_ids, next_token], dim=-1)

        return self.tokenizer.decode(generated_ids[0], skip_special_tokens=True)

The $O(N^2)$ Bottleneck

Look at what happens by, say, step 15 of generation: generated_ids now holds the prompt plus 14 generated tokens. On this step, the model recomputes the Key and Value projections for all 14 historical tokens from scratch — even though their outputs are mathematically identical to what they were on the previous step. A past token's activations never change once it's been processed; recomputing them is pure waste.

Benchmarked on a 117M-parameter GPT-2 model, CPU-only, this baseline produced:

  • Time-To-First-Token (TTFT): 0.1263 s
  • System Throughput: 5.52 tokens/sec
  • Peak RAM (RSS): 799.28 MB

As the number of generation steps grows, per-token latency grows right along with it — linearly, because the forward pass gets more expensive on every single step. Past a few hundred tokens, this approach becomes unusable for anything resembling a real conversation.

Fixing that isn't a smarter-model problem — it's a rethink of how memory interacts with the attention layers in the first place.

What's Next

In the next post, I replace this $O(N^2)$ loop with Key-Value (KV) caching and static batching — and along the way, hit a subtle position-embedding bug that quietly broke batched generation, plus a genuinely counterintuitive RAM result that took a while to explain.