Grammar vs Bits: Format Contracts and Weight Precision in Small-Model Tool Calling

Summary

Small language models can be stored using less memory, but this can make them worse at calling tools. This study asks whether forcing answers to follow a strict grammar restores that accuracy. It finds no clear recovery specific to the reduced precision. In a separate, non-randomized comparison, changing the required answer format from JSON to Python-style calls produced large gains on parallel calls.

Key findings

• No clear quantization-specific recovery from grammar decoding.
• Changing JSON to Pythonic took Llama-3B Q4 from 0% to 88.3% and Llama-1B from 1.0% to 62.3% on parallel calls, in a non-randomized sensitivity analysis.
• All inputs, responses, and code are released.

Cite

@misc{mohith2026grammarvsbits,
  author = {M, Mohith},
  title = {Grammar vs Bits: Format Contracts and Weight Precision in Small-Model Tool Calling},
  year = {2026},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.23926882},
  version = {v0.1.3},
  url = {https://doi.org/10.5281/zenodo.23926882}
}