CacheKit
A behaviour-checked workflow for diagnosing and repairing LLM prompt caching.
Python · vLLM · Modal · Next.js
View sourceThe problem
A changing timestamp, customer identity or retrieved passage near the start of a prompt can invalidate a long shared prefix. The inference server then repeats work for unchanged input. Moving those fields may improve cache reuse, but it can also alter model behaviour, so the source change needs more than a faster replay.
The approach
- Diagnose recorded requests offline: group call sites, align stable and changing content, and identify the tokenizer positions where shared prefixes break.
- Turn redacted findings into a constrained coding-agent task with allowed source files, a reuse target and verification requirements; an explicitly authorized Devin session can open a fix PR.
- Check behaviour before merging, then replay matching BEFORE/AFTER traffic on a pinned vLLM server and retain the measured evidence for human review.
Inside the system
Diagnosis and simulation
The Python CLI consumes local JSONL requests, maps prefix breaks and simulates a capacity-limited vLLM prompt-prefix cache. Diagnosis runs deterministically and offline; it does not call a model or start a GPU.
Scoped source fixes
A local API and Next.js dashboard expose saved findings. CacheKit exports a redacted task for Devin, which can inspect the repository, make a branch and open a capped, explicitly authorized PR. Private request logs stay local.
Quality guard and measured replay
Verification checks preserved content, tools and roles, source scope, reuse targets, project tests and required golden-case evidence. Explicit collectors replay actual model requests; the dashboard only inspects the saved results.
Decisions and trade-offs
Separate estimates from measurements
Simulated prefix reuse is not a latency result. The before/after comparison uses actual vLLM replay evidence and reports cache reuse and time to first token separately.
Keep behaviour and review in the loop
The guard exposed a missing authorization commitment during the refactor. The final candidate includes declared calendar and confirmation-behaviour repairs, followed by review. Changed outputs require human approval; the workflow does not claim that the coding agent repaired and verified everything autonomously.
State the measurement boundary
The saved results use Qwen3-4B on vLLM and one L4, with sequential fictional helpdesk traffic. The comparison is the repaired candidate against the original, not an isolated causal estimate of one prompt movement or a concurrency benchmark. Passing the recorded behaviour cases does not establish universal equivalence.
What it produces
- On Qwen3-4B (vLLM, L4), the measured cache-hit rate rose from 0.3% to 80.7%.
- Median time to first token fell from 2.0s to 1.4s on matching before/after traffic.
- Released saved request replays, source-fix evidence and behaviour-guard verification so the measured change can be reviewed.