Design Your LLM Prompts for KV Cache Reuse: Prompt Architecture Is Infrastructure Architecture
Prefix caching means prompt layout is an infrastructure decision, not a style choice. How we structure Digital Twin prompts for AntFabric.ai — stable layers first, timestamps last — so vLLM can reuse 20K+ tokens of KV cache instead of recomputing them on every request.
Lessons we're learning while building AntFabric.ai and Digital Twin (Humans)
When we tune an LLM inference server, the checklist is predictable: GPU memory, quantization, context length, batching, concurrency, model selection.
But while building AntFabric.ai, we keep learning that some of the most valuable optimizations don't live at the GPU or model-serving layer at all.
They start much earlier — with how the prompt itself is assembled.
At AntFabric.ai we're building toward an AI-native workspace powered by Digital Twin (Humans) — intelligent digital counterparts that understand organizational knowledge, context, responsibilities, conversations, workflows, and tools. The goal is a workplace where AI doesn't simply answer questions, but understands how people and organizations actually work.
As we build it, we keep running into things that look like small implementation details and turn out to be architecture. One of them:
Prompt architecture is infrastructure architecture.
KV prefix caching is what makes that true.
KV Cache in Sixty Seconds
During transformer inference, the model computes attention keys and values for every token it processes. Recomputing them for the whole sequence at every generation step would be catastrophically wasteful, so inference engines keep them in GPU memory — the KV cache.
PREFILL — all prompt tokens processed in one pass
─────────────────────────────────────────────────
[t₁][t₂][t₃] ─────────────────────────► [t₃₀₀₀₀]
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────────────────────────────────────┐
│ Transformer forward pass │
└───────────────────────┬──────────────────────┘
▼
┌───────────────────────┐
│ K/V states in GPU │ ◄── the KV cache
└───────────┬───────────┘
│ reused every step
DECODE ─────────────────┘
──────
token 30,001 ─► read cache ─► compute 1 new K/V ─► append ─► repeatPrefill is compute-bound and scales with prompt length. Decode is memory-bound and scales with cache size. That split is the whole game, and I've written about the memory side of it in detail in The GPU KV Cache: Why Your LLM's Memory Matters More Than You Think.
This post is about the other side: not recomputing prefill you've already paid for.
Prefix Caching: The Part That Changes How You Write Prompts
Imagine a Digital Twin serving two requests a few seconds apart.
Request A Request B
───────── ─────────
┌───────────────────────────────┐ ┌───────────────────────────────┐
│ SYSTEM PROMPT 4K │ │ SYSTEM PROMPT 4K │
│ DIGITAL TWIN RULES 3K │ = │ DIGITAL TWIN RULES 3K │
│ TOOL DEFINITIONS 6K │ │ TOOL DEFINITIONS 6K │
│ ORGANIZATION POLICIES 7K │ │ ORGANIZATION POLICIES 7K │
└───────────────────────────────┘ └───────────────────────────────┘
└──────────── 20,000 identical tokens ───────────┘
│
┌───────────┴───────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ memory + RAG │ │ memory + RAG │
│ question A │ │ question B │
│ +10K │ │ +12K │
└─────────────────┘ └─────────────────┘
prefill 10K, not 30K prefill 12K, not 32KWithout prefix caching, each request pays full prefill for all 30K or 32K tokens. With prefix caching, an engine like vLLM hashes the prompt in fixed-size token blocks and reuses the KV blocks it has already computed for an identical prefix. The shared 20K is computed once and looked up thereafter.
At the scale of one chatbot, this is a nice-to-have. At the scale of an organization full of Digital Twins, it's a line item.
Why Digital Twins Make This Urgent
A Digital Twin is not a chatbot with a longer system prompt. The context requirement is structurally different.
Chatbot Digital Twin (Human)
─────── ────────────────────
┌───────────────────┐ ┌──────────────────────────────┐
│ System prompt │ │ Identity, role, duties │
│ Conversation │ │ Org policies & permissions │
│ User question │ │ Team structure │
└───────────────────┘ │ Tool + schema definitions │
~2K tokens │ Long-term knowledge │
│ Memory │
│ Projects, recent threads │
│ Current workspace state │
│ Retrieved documents │
│ Current request │
└──────────────────────────────┘
30K – 60K tokensTens of thousands of tokens per request. But here's the observation that matters:
Those tokens do not change at the same rate.
Some of them are effectively frozen. Some change weekly. Some change on every keystroke. Treating them as one undifferentiated blob throws away the difference — and the difference is exactly what the cache is keyed on.
Design Prompts Stable → Dynamic
Order the prompt by rate of change, slowest first.
┌──────────────────────────────────────┬───────────────┐
│ LAYER │ CHANGE RATE │
├──────────────────────────────────────┼───────────────┤
│ Global system instructions │ ~never │ ▲
│ Organization policies │ ~never │ │
│ Agent / Digital Twin definition │ per release │ │ CACHEABLE
│ Tool + schema definitions │ per deploy │ │ PREFIX
│ Role & responsibilities │ monthly │ │
│ Stable user profile │ monthly │ ▼
├──────────────────────────────────────┼───────────────┤
│ Retrieved memory │ per session │ ▲
│ Knowledge / RAG chunks │ per request │ │
│ Current workspace state │ per request │ │ RECOMPUTED
│ Timestamp, request ID, trace ID │ per request │ │ EVERY TIME
│ Conversation │ per turn │ │
│ Current request │ unique │ ▼
└──────────────────────────────────────┴───────────────┘The rule is one line:
MOST STABLE ──► MOST DYNAMICThis isn't tidiness. The boundary between those two halves is the cache boundary. Push it down and you buy reuse; push it up and you pay prefill.
One Timestamp Can Destroy Twenty Thousand Tokens of Reuse
This is the mistake we see most often, and it looks completely harmless.
Current time: 10:21:32
Request ID: 84f82d
You are this person's Digital Twin.
Organization policies...
Agent instructions...
Tool definitions...
Role information...Five seconds later:
Current time: 10:21:37
Request ID: 91ac14
You are this person's Digital Twin.
Organization policies...
Agent instructions...
Tool definitions...
Role information...To a human reader these are the same prompt. To a block-hash cache they share nothing.
✗ Volatile fields first
┌────────┬─────────────────────────────────────────────┐
│ TIME │████████████ stable 20K ████████████████████ │
└────────┴─────────────────────────────────────────────┘
▲
└─ block 1 hash differs ─► every later block invalid ─► prefill 22K
✓ Volatile fields after the stable layers
┌─────────────────────────────────────────┬───────┬─────┐
│████████████ stable 20K ████████████████ │ TIME │ ask │
└─────────────────────────────────────────┴───────┴─────┘
└──────── 20K cache hit ────────┘ ▲
└─ prefill ≈ 2KThe reordered version carries exactly the same information:
SYSTEM
ORGANIZATION
DIGITAL TWIN
TOOLS
ROLE
STABLE PROFILE
──────────── cache boundary ────────────
DYNAMIC CONTEXT
MEMORY
KNOWLEDGE
CURRENT TIME
CURRENT STATE
CONVERSATION
CURRENT REQUESTSame tokens. Same semantics. Roughly a 10× difference in prefill work.
Prefix Caching Is Not Arbitrary Chunk Caching
It's worth being precise about why the timestamp is so destructive, because "the cache stores chunks" is the wrong mental model.
Engines hash prompts in fixed-size blocks, and each block hash is chained to the one before it:
hash(blockₙ) = f( hash(blockₙ₋₁), tokens(blockₙ) )The hash of a block depends on everything that came before it. So identical text in the middle of two prompts produces different hashes if anything ahead of it differed.
Prefix hit Prefix miss
────────── ───────────
A: [A][B][C][D][X] A: [A][X][B][C][D]
B: [A][B][C][D][Y] B: [A][Y][B][C][D]
└── reused ──┘ ✓ └┘ diverged at block 2 ✗
B, C and D are byte-identical
and completely unreachable.There is no partial credit and no re-alignment. The cache matches a prefix, not a set.
If you've built HTTP caching, this will feel familiar — it's the same discipline as keeping cache keys stable so a varying header doesn't shatter your hit rate, which I went through in How to Build Vercel-Like Caching With Nginx for Your Backend. Same law, different layer of the stack.
Think in Cache Layers, Not in "a 64K Prompt"
Once the ordering is right, the prefix stops being one blob and becomes a tree. Different requests share different depths of it.
┌──────────────────────────────┐
│ GLOBAL INSTRUCTIONS 4K │ every request, every tenant
└───────────────┬──────────────┘
┌───────────────┴──────────────┐
│ ORGANIZATION POLICIES 7K │ every request in one org
└───────────────┬──────────────┘
┌───────────────┴──────────────┐
│ AGENT DEFINITION 3K │ every twin of this type
└───────────────┬──────────────┘
┌───────────────┴──────────────┐
│ TOOL SCHEMAS 6K │ every twin with this toolset
└───────────────┬──────────────┘
┌───────────────┴──────────────┐
│ ROLE + STABLE PROFILE 3K │ every request from one person
└───────────────┬──────────────┘
═════════════ cache boundary ═════════════ ◄── 23K reusable
┌───────────────┴──────────────┐
│ MEMORY / RAG / STATE │
│ CONVERSATION │ recomputed per request
│ CURRENT REQUEST │
└──────────────────────────────┘Which reframes the question you ask about a prompt. Not:
"We have a 64K prompt."
But:
"How much of that 64K is actually unique?"
The first is a capacity question. The second is a cost question, and it's the one that shows up on the bill.
Deterministic Prompt Construction Matters
Here's the subtler failure: the same information is not automatically the same prompt.
Tools: Tools:
- calendar - CRM
- email vs - search
- CRM - calendar
- search - emailSemantically identical. Token-wise, two different sequences and two different hashes. Prefix caching operates on tokens, not meaning.
Anything non-deterministic in your prompt builder is a silent cache leak:
| Source of non-determinism | What it looks like | Fix |
|---|---|---|
| Unordered tool lists | Registry iteration order, Set or dict ordering | Sort by a stable key before rendering |
| JSON serialization | Key order, spacing, unicode escaping | One serializer, sorted keys, fixed separators |
| Policy / document assembly | Results returned in retrieval-score order | Sort the stable subset; keep scored results below the boundary |
| Schema generation | Auto-generated tool schemas regenerated per call | Generate once at startup, cache the string |
| Whitespace and templating | Trailing newlines, conditional blank lines | Normalize the rendered block, snapshot-test it |
| Floats and IDs | 0.7000000001, trace IDs inside the system block | Format explicitly; move IDs below the boundary |
This matters most for agentic systems, where tool schemas alone can run to thousands of tokens. If those schemas re-serialize in a different key order on each pod restart, half your fleet is running a different prefix from the other half — and neither half can use the other's cache. It's the kind of coordination problem that shows up everywhere once agents multiply, which is the broader argument in How AI Is Changing the World — And Why Your Agents Need a Company, Not Just a Prompt.
A useful test: render the same logical prompt twice in a unit test and assert byte equality. It catches this class of bug immediately.
`max-num-batched-tokens` Is a Different Knob
We're also experimenting with vLLM scheduling for workloads where most prompts land in the 8K–64K range while still allowing much larger contexts when needed:
--max-model-len 262144
--max-num-batched-tokens 8192
--enable-prefix-cachingIt's easy to conflate these two settings. They're unrelated.
--max-num-batched-tokens controls how much token work the scheduler can process in a single iteration. With chunked prefill, a long prompt is split across iterations:
--max-num-batched-tokens 8192 (scheduler budget, not a cache)
64K prompt
├─ iteration 1 │████████│ 8K prefill
├─ iteration 2 │████████│ 8K
├─ iteration 3 │████████│ 8K
├─ ... │ .... │
└─ iteration 8 │████████│ 8K ─► first token emitted
Other users' decode steps slot into the same iterations.
smaller budget ─► fairer latency under concurrency
larger budget ─► higher prefill throughput
either way ─► identical cache hit rateThe range worth benchmarking for our workload:
| Budget | Character |
|---|---|
| 4K | Highly granular — best interactive fairness, lowest prefill throughput |
| 8K | Balanced default |
| 16K | Throughput-leaning |
| 32K | Aggressive prefill — long prompts finish fast, short requests wait |
Two things to keep straight. First, on recent vLLM versions prefix caching is enabled by default — check your version instead of assuming the flag is doing the work. Second, and more important: choosing 4K over 8K does not improve prefix reuse. The scheduler budget changes how prefill work is paced. Only prompt structure changes how much prefill work exists.
What It's Actually Worth
Illustrative arithmetic, not a benchmark — but the shape is what matters.
| Metric | Per twin / day | 1,000 twins | 10,000 twins |
|---|---|---|---|
| Requests | 50 | 50,000 | 500,000 |
| Stable prefix tokens | 1.15M | 1.15B | 11.5B |
| Dynamic tokens | 0.6M | 0.6B | 6B |
| Prefill avoidable by reuse | ~66% | ~66% | ~66% |
Assuming a 23K stable prefix and ~12K of dynamic context per request. At ten thousand Digital Twins, that's roughly 11.5 billion prompt tokens per day that are byte-identical to something the GPU computed minutes ago.
Recomputing them isn't an implementation detail. It's a capital expenditure.
One Caution: Don't Share Prefixes Across Trust Boundaries Carelessly
Prefix reuse is a performance feature with a security surface. Two things to keep in mind before you push a tenant's policies into a widely shared prefix:
- Cache-hit timing is observable. The latency difference between a hit and a miss can reveal whether a particular prefix has been seen before. If the prefix content is itself sensitive, isolate the cache per tenant rather than relying on hashing for privacy.
- Shared prefixes must contain only shared truths. Global instructions and public policy are safe to share. Per-user permissions, entitlements, and PII belong below the boundary, scoped to that user's own prefix chain — and permission checks belong in the tool layer, enforced at call time, never in prompt text alone.
Design the layers so the widely-shared ones are the ones you'd be comfortable printing.
AI-Native Workspaces Need a Different Architecture
This lesson goes past inference tuning.
Traditional workplace software is application-centric. Each app owns its own data, its own workflow, and its own idea of what you're working on:
APPLICATION-CENTRIC TWIN-CENTRIC
─────────────────── ────────────
┌─────┐ ┌─────┐ ┌─────┐ ┌──────────────┐
│Email│ │ Cal │ │ CRM │ │ DIGITAL TWIN │
└──┬──┘ └──┬──┘ └──┬──┘ └──────┬───────┘
┌──┴──┐ ┌──┴──┐ ┌──┴──┐ ┌───────────┼───────────┐
│Docs │ │ HR │ │Chat │ ▼ ▼ ▼
└──┬──┘ └──┬──┘ └──┬──┘ MEMORY KNOWLEDGE CONTEXT
│ │ │ └───────────┼───────────┘
└───────┼───────┘ ▼
▼ AI ORCHESTRATION
┌─────────────┐ │
│ HUMAN │ ┌─────┬───────┼───────┬─────┐
│ moves the │ ▼ ▼ ▼ ▼ ▼
│ information │ Email Cal CRM Docs HR
└─────────────┘In the left model, the human is the integration layer — copying context between tools all day. In the right model, AI becomes the layer that connects knowledge, context, people, and actions, and the Digital Twin is the contextual representation of the person inside that workspace.
It should be able to answer, without being told:
Who am I?
What am I responsible for?
What am I working on right now?
What does my organization already know?
What happened before this?
What decisions were made, and by whom?
What needs my attention?
Which tools can I use, and what am I allowed to do with them?
What should happen next?That is a different product from an LLM chat box bolted onto existing workplace software — which is the distinction I've argued elsewhere in Why AI-Native Organizations Will Outperform AI-Assisted Ones. Prefix caching turns out to be one of the places where that architectural difference becomes measurable rather than philosophical.
The Infrastructure Has to Support the Vision
Once you think this way, the layers stop being independent.
Digital Twin
│
┌─────────────────┼─────────────────┐
│ │ │
Memory Knowledge Identity
│ │ │
└─────────────────┼─────────────────┘
▼
Agent Runtime
▼
AI API Gateway
▼
Model Serving
▼
vLLM
▼
GPUEvery decision propagates:
| Decision at this layer | Shows up here |
|---|---|
| Prompt structure | Prefix cache hit rate |
| Memory architecture | Context size per request |
| Tool architecture | Prompt size and schema churn |
| RAG architecture | Size of the dynamic tail |
| Agent design | Concurrency and request fan-out |
| Model selection | KV cache footprint per token |
| Context length | GPU capacity and max batch size |
Which is the actual lesson: prompt engineering is systems engineering. The gateway and routing layer is where most of this gets enforced in practice, and I've written about how that layer is consolidating in The Strategic Evolution of Enterprise AI Infrastructure. Once a router can send the easy majority of work to open models and reserve frontier models for the hard cases — the split I described in The 90/10 AI Stack — cache-friendly prompt layout has to hold across every backend in the pool, not just one.
A Practical Checklist
What we're standardizing on internally:
- Order every prompt by rate of change. Slowest-changing content first, unique content last. No exceptions for convenience.
- Put a literal boundary marker in the template. A comment or delimiter that says "nothing below this line is cacheable" makes violations obvious in code review.
- Move timestamps, request IDs, trace IDs and session IDs below the boundary. They almost never need to be at the top, and they are the single most common cache killer.
- Make the builder deterministic. Sorted tool lists, one canonical JSON serializer, schemas generated once at startup.
- Snapshot-test the stable prefix. Render it twice, assert byte equality; store the golden output so an accidental reorder fails CI.
- Measure the prefix hit rate, not just latency. If you can't see the hit rate, you can't tell whether a refactor destroyed it.
- Keep the widely-shared layers free of anything user-specific. Performance and trust boundaries have to agree.
- Benchmark the scheduler budget separately. Tune
max-num-batched-tokensfor fairness and throughput — never expect it to fix cache reuse.
Most of these cost nothing at build time. They just have to be decided before the prompt builder has fifty call sites.
Our Current Mental Model
We increasingly think about an AI request as two halves with a boundary between them:
AI REQUEST
│
┌──────────────┴──────────────┐
▼ ▼
REUSABLE CONTEXT DYNAMIC CONTEXT
──────────────── ───────────────
System prompt Memory
Organization RAG results
Agent rules Workspace state
Tool schemas Conversation
Stable profile Current request
│ │
maximize this keep this relevant
└──────────────┬──────────────┘
▼
MODELThe goal is to maximize genuinely reusable context while keeping the dynamic half small, fresh and relevant. Getting the second half right is its own discipline — how much specificity a model actually needs before it stops guessing is roughly the problem I framed as The 3D Prompting Framework.
The Bigger Lesson
As context windows stretch to 32K, 64K, 128K and 256K, "how many tokens can the model handle?" stops being the interesting question.
The better one:
How many of those tokens actually need to be recomputed?
Request = 50K tokens
25K stable ─► computed once, reused thousands of times
+ 25K dynamic ─► the real per-request costIf the first 25K is shared across thousands of requests, it isn't really 50K of work. It's 25K of work and a lookup.
Sometimes the answer isn't:
Buy another GPU.Sometimes it's:
Design the prompt better.That's one of many things we're learning as we build AntFabric.ai and figure out what an AI-native workspace powered by Digital Twin (Humans) should actually be. The deeper we go, the clearer it gets that intelligent workplace software isn't an LLM problem. It's a combination of AI, knowledge, memory, context, agents, tools, infrastructure — and people.
And sometimes a decision as small as where you put a timestamp in a prompt reaches all the way down to GPU utilization.
That's what makes building AI-native systems so interesting.
We're not just learning how to use AI. We're learning how software itself needs to be redesigned around it.