Comment by tchavali
Hi HN, I built Prefixwise because prompt caching is where most of an agent's token bill is won or lost, and the providers only tell you how many tokens were cached, never why the rest weren't.An agent resends its whole conversation every turn. If one byte near the top changes (tools serialized in another order, a timestamp in the system prompt, an earlier message rewritten), everything after it is billed at full price again. In my own runs the common causes were boring and invisible: a tool list built from a set, a date in the system prompt, a missing cache_control marker.
How it works: it sits between the agent and the model (a proxy, a one-flag vLLM middleware, or an SDK wrapper), fingerprints each request block by block, and compares it with the previous turn of the same conversation. Each miss gets one of 17 causes, the first block that changed, and the cost. It stores hashes and token counts, never prompt text. On vLLM and SGLang it also reads the engine's KV-cache events, so an eviction is confirmed rather than guessed.
It also shows where the rest of the bill goes. On a well-cached session the largest part of the cost is often re-reading context the cache already holds, which no hit-rate fix lowers, so the report says so instead of framing everything as waste. A gateway engineer on Reddit asked for that; on their DeepSeek session it was 62% of the bill.
Some numbers from live runs, each with a record in the repo:
Agrees with Anthropic's own miss diagnostics: 32 of 32.
Names injected causes: 120 of 120 on vLLM, LMCache and SGLang.
Sorting a drifting tool list: 88% to 97% cached on a vLLM replay, task success flat.
Reasoning templates that drop earlier thinking re-prefill the last answer: 57% of prompt tokens on an R1 distill.
Limits: the fixes were proven on scripted sessions and small models, not months of production traffic, and Ollama and most llama.cpp builds don't report cached tokens, so there it can name prompt changes but can't tell a hit from a miss.
I'd most like to hear from people running agents in production: what cache problems do you hit that this wouldn't catch?