Leaderboard Noise: Greedy Variation Across Batches and Runs, One-Item Leads, and Their Mitigation

Summary

A benchmark ranking can change even when the models and questions stay the same. This study compares answers across repeated runs, batch settings, and A10 and L4 GPUs. A decoding method designed to keep answers consistent across batch settings removed all observed changes, but reduced throughput. The released evalnoise command-line tool supports the analysis.

Key findings

• 70,800 outputs from 6 open models on A10 and L4 GPUs; default greedy decoding changed 2.0–4.6% of answers and a near-tied ranking reversed between GPUs.
• Batch-invariant decoding removed all observed changes at a 32–60% throughput cost, measured on A10.
• Released the evalnoise CLI with the v0.1.0-paper release.

Cite

@misc{mohith2026leaderboardnoise,
  author = {M, Mohith},
  title = {Leaderboard Noise: Greedy Variation Across Batches and Runs, One-Item Leads, and Their Mitigation},
  year = {2026},
  version = {v0.1.0-paper},
  url = {https://github.com/mohi-devhub/leaderboard-noise/releases/tag/v0.1.0-paper}
}