Below the Ice — Prompt Caching: Why Claude Reads Almost Nothing Twice
98.5% of tokens in a tracked Claude Code session were spent re-reading conversation history. That one number explains why prompt caching isn't a feature — it's the load-bearing beam of the whole product. Tonight we go under the headline.
This is the print twin of tonight's Below the Ice — our evening deep-dive, one topic told properly. Prefer it in your ears while you wind down? Listen to tonight's episode.
There is a number buried in the Claude Code engineering record that I want to start with tonight. A developer tracked one real session — not a stress test, just a normal working session — and found that 98.5% of the tokens the model consumed were spent re-reading conversation history. Not thinking. Not reasoning. Re-reading.
That one measurement reframes almost everything about how you think about AI coding agents. It is also the reason the Anthropic team spent serious engineering effort on a mechanism called prompt caching — and why tonight's dive is worth your time if you build anything that puts a language model in a loop.
What it is
Prompt caching is the mechanism that lets Claude skip re-reading the parts of a conversation that have not changed since the last message.
Without it, every single turn in a multi-message session would require the model to process the entire conversation from the beginning — your system instructions, every message, every file snippet, every tool response — before it could think about your new question. The context window is not a memory. The model has no state between calls. On every turn, it starts from a blank slate.
Prompt caching is how Anthropic works around that constraint. It stores the computed result of processing a stable block of text — the prefix — so the next turn can pick up at the end of that block instead of re-processing it. What looks like a conversation with a model that "remembers" is, under the hood, a carefully managed series of cache hits.
How it actually works
The analogy I find clearest is a professor who has memorized a textbook.
The first time the professor teaches from it, they have to read the whole thing — cover to cover. That is expensive. But afterward, when a student asks a question, the professor does not re-read the textbook. They answer from their internalized version. A new student joining a week later also benefits from that same internalized state.
In Claude's architecture, the "textbook" is the stable part of your context. The first time it processes that block, it does a full compute pass. But the result — the internal state the model builds up from reading it — can be saved. On the next turn, if that prefix has not changed, Claude picks up from the saved state instead of starting over.
There are three layers in the cache, stacked from most-stable to least:
| Layer | What it holds | What breaks it |
|---|---|---|
| System | The static system prompt and built-in tool definitions | Adding or removing an MCP server mid-session; switching models (caches are per-model, so a model swap voids everything) |
| Project | Your CLAUDE.md and project memory files | Adding a new skill mid-session |
| Conversation | The growing message history | Every new turn extends it by design; editing an earlier message voids everything below it |
Breaking the cache at a higher layer voids all lower layers. This is not a quirk — it is an architectural constraint. The cache is byte-for-byte prefix matching: the stored computation only stays valid if the prefix it covers is identical character-for-character.
The pricing asymmetry makes this concrete. According to Anthropic's pricing docs, a cache write (the first pass where the model processes and stores the prefix) costs roughly 1.25× a normal input token — slightly more expensive than uncached. But a cache read on all subsequent turns costs approximately 0.1× — one tenth the price. A long, warm, cache-hitting session is roughly ten times cheaper per turn than a series of fresh contexts.
Nate Herk, who teaches Claude Code extensively and has built large-scale agentic workflows with it, captured his token dashboard on a single heavy working day and found 91 million tokens saved from cache hits. Over a week, north of 300 million.
Why it matters now
That efficiency number is interesting. The quote that makes it load-bearing is from Thariq Shihipar, a member of technical staff on the Claude Code team at Anthropic.
Writing in February 2026, Thariq described how the Claude Code engineering team thinks about prompt caching internally:
"We build our entire harness around prompt caching. A high prompt cache hit rate decreases costs and helps us create more generous rate limits for our subscription plans, so we run alerts on our prompt cache hit rate and declare SEVs if they're too low."
A SEV — severity incident — is what an engineering team declares when something is broken badly enough to warrant immediate response. Treating a low cache hit rate as an incident puts it in the same category as the server being down. Because in practice, it has similar effects: Claude Code feels slower, subscription limits feel less generous, and Anthropic's serving cost per user goes up.
The cache is not a nice-to-have optimization layered on top of the product. It is the mechanism that makes the subscription economics work.
For you as a builder, this has three immediate implications.
First: pauses are expensive. The cache TTL for a Claude Code subscription session is one hour. If you walk away from a session for longer than that, the cached prefix expires. The next message re-processes everything from scratch — at the 1.25× write cost — before it can answer you. If you are near a limit with hours left, stepping away and letting the cache expire is one of the more expensive things you can quietly do.
Second: sub-agents start cold. Sub-agents operate at the API-tier cache TTL, not the subscription TTL. That means five minutes. If you spawn a sub-agent and it takes longer than five minutes to return results, its reply comes back after the cache has expired. Every sub-agent in a fan-out is essentially working from a cold start.
Third: context hygiene is now a financial discipline, not just a speed one. A developer tracked the same work across two months with identical output quality. The difference — $345 per month versus $42,000 per month — was entirely context-management drift. The work was the same. The cache hit rate was not.
What is overhyped
Prompt caching does not eliminate the "dumb zone."
The dumb zone is the practitioner name for the quality degradation that sets in long before the context window fills. Experienced Claude Code users report the model feeling noticeably less sharp past around 100,000 tokens in a conversation — misremembering recent decisions, failing to apply skills it used ten turns earlier, producing slightly off-key responses. Nate and Cole (of the 1,000 Hours with Claude podcast) put the onset point at around 250,000 tokens for Opus.
Prompt caching makes the session cheaper. It does not fix the attention problem. The lost-in-the-middle effect — where the model reliably recalls what is at the very front and the very end of its context, but loses track of the middle — persists regardless of cache state. A warm, cheap session is still a degraded session if it has gone on too long.
The right mental model: caching keeps your session from bleeding money while you work. Context hygiene (compacting at 60%, starting fresh when you switch tasks, keeping CLAUDE.md tight) is what keeps quality from degrading. These are complementary disciplines, not substitutes.
The other overstated claim is that the cache is set-and-forget. It is not. Because it operates on prefix matching, any mutation high in the stack — switching models mid-session, adding a new tool, editing an early message — silently voids everything below it. The cache does not warn you. You just pay full price on the next turn and continue without noticing.
What to watch
The TTL ladder is going to keep evolving. Right now the defaults are stark: one hour for Claude Code subscription sessions, five minutes for the API and sub-agents. The practical consequence is that sophisticated agentic pipelines — where multiple sub-agents collaborate over a long task — pay cold-start costs repeatedly. Watch for Anthropic to experiment with configurable or context-aware TTLs as agentic use cases become dominant. There is too much engineering effort invested in the cache mechanism for the TTL story to stay this binary.
Cache hit rate as a developer metric. Right now, most developers do not know their cache hit rate. They feel the effects — fast sessions, slow sessions, expensive sprints — without having a clean number to optimize against. The Claude Code team monitors it as an internal SLA. It is only a matter of time before it surfaces as a first-class metric in the Claude Code interface or API response metadata. When it does, expect a wave of tooling built around optimizing it the same way teams optimize database query plans today.
The boundary between session and memory. The current caching model is ephemeral — it lives for the TTL window and is gone. What builders are building toward is a world where useful computed context persists across sessions: where the prefix that covers your entire project's history can be cached semi-permanently and loaded on demand. Anthropic is not the only lab working on this shape of problem. The convergence of long-context models, persistent caches, and cross-session memory is probably the most interesting infrastructure frontier in applied AI right now.
The 98.5% number is not just a technical curiosity. It is a description of where AI infrastructure is today: enormously powerful systems spending most of their budget re-reading what they already know. Prompt caching is the first serious engineering answer to that. The question of what comes next — sessions that remember, agents that share computed state, architectures that never re-read anything — is still open.
That is a good reason to stay curious.
Sources: Thariq Shihipar — Lessons from building Claude Code: prompt caching is everything · Anthropic — Prompt Caching (docs) · Nate Herk — The One Habit That Doubles Your Claude Code Session Limit · Simon Willison on Thariq's article
Below the Ice publishes every evening at 7 PM MTY — one topic, told properly. Find the audio episode and all our shows at penguinalley.com.