Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics

• URL: http://arxiv.org/abs/2606.03982v1
• Date: Tue, 02 Jun 2026 17:58:02 GMT
• Title: Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics
• Abstract summary: Quantities with measurement units, such as 110 cm and 1.2 m, require language models (LMs) to combine a numeral with a symbolic unit scale.<n>We find that accuracy degrades near the comparison boundary, where small changes in value determine the correct answer.<n>The results suggest that LMs compare quantities through a bag of numerals over numerals and units, rather than first converting both expressions to an exact shared-scale representation.
• Score: 32.21373620078722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantities with measurement units, such as 110 cm and 1.2 m, require language models (LMs) to combine a numeral with a symbolic unit scale. Here, we study how LMs compare such quantities in controlled settings spanning several unit systems. We find that accuracy degrades near the comparison boundary, where small changes in value determine the correct answer. The resulting errors are systematic: linear surrogate models predict LM preferences from numerical-difference and unit-scale-difference cues, and causal interventions on subspaces aligned with these variables shift model's output. The results suggest that LMs compare quantities through a bag of heuristics over numerals and units, rather than first converting both expressions to an exact shared-scale representation.

Related papers

• Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs [0.8129517423858409]
  Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication.<n>We introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation.<n>We also define arithmetic load, C, as the product of the total and non-zero digit count.
  arXiv  Detail & Related papers  (2026-04-20T12:52:07Z)
• Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models [13.5805504750573]
  We show that large language models (LLMs) encode real-world numerical correlations but tend to systematically amplify them.<n> irrelevant context induces consistent shifts in magnitude representations, with downstream effects that vary by model size.<n>These findings lay the groundwork for fairer, representation-aware control under multi-attribute entanglement.
  arXiv  Detail & Related papers  (2025-11-06T04:47:08Z)
• Unravelling the Mechanisms of Manipulating Numbers in Language Models [9.583581545538479]
  We explore how language models manipulate numbers and quantify the lower bounds of accuracy of these mechanisms.<n>We find that despite surfacing errors, different language models learn interchangeable representations of numbers that are systematic, highly accurate and universal.<n>Our results lay a fundamental understanding of how pre-trained LLMs manipulate numbers and outline the potential of more accurate probing techniques.
  arXiv  Detail & Related papers  (2025-10-30T09:08:50Z)
• Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification [9.397157329808254]
  MUSE is a simple information-theoretic method to identify and aggregate well-calibrated subsets of large language models.<n> Experiments on binary prediction tasks demonstrate improved calibration and predictive performance compared to single-model and na"ive ensemble baselines.
  arXiv  Detail & Related papers  (2025-07-09T19:13:25Z)
• NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models [37.15662878141497]
  We analyze existing Large Language Models (LLMs) on processing of numerals and units of measurement. We first anatomize the reasoning of math word problems to different sub-procedures like numeral conversions from language to numbers and measurement conversions based on units. We further annotate math word problems from ancient Chinese arithmetic works which are challenging in numerals and units of measurement.
  arXiv  Detail & Related papers  (2024-06-05T02:26:14Z)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models [63.118592279833656]
  Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs)<n>We propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise.<n> Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths.
  arXiv  Detail & Related papers  (2024-05-23T16:21:48Z)
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch [72.97553348776425]
  We introduce DARE to set most delta parameters without affecting the abilities of Supervised Fine-Tuning (SFT) LMs. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT models and merge them into a single model. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open Leaderboard.
  arXiv  Detail & Related papers  (2023-11-06T13:43:07Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• Examining Scaling and Transfer of Language Model Architectures for Machine Translation [51.69212730675345]
  Language models (LMs) process sequences in a single stack of layers, and encoder-decoder models (EncDec) utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs.
  arXiv  Detail & Related papers  (2022-02-01T16:20:15Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.