Reload or Recompute? A Calibrated Rule for KV-Cache Reuse Across Commodity GPUs

Summary

When a conversation resumes, a server can reload its saved working state or calculate it again. This study tests a rule for choosing the faster option across T4, L4, A10, A10G, and H100 GPUs. The rule uses about a minute of calibration before making its predictions. Extending those predictions to much longer conversations is a known failure case.

Key findings

• Frozen predictions from about a minute of calibration correctly classified 574/636 resolved decisions on L4, 192/206 on A10, and 398/426 on A10G.
• Break-even bandwidth rose from 100–400 MB/s on T4 to 800–1,600 MB/s on L4/A10 for Qwen2.5-1.5B with a 1,024-token prefix.
• Long-context extrapolation is a known failure case; the data is available in v0.1.0-data.

Cite

@misc{mohith2026reloadorrecompute,
  author = {M, Mohith},
  title = {Reload or Recompute? A Calibrated Rule for KV-Cache Reuse Across Commodity GPUs},
  year = {2026},
  version = {v0.1.0-data},
  url = {https://github.com/mohi-devhub/reload-or-recompute/releases/tag/v0.1.0-data},
  note = {In preparation; data release}
}