The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It

• URL: http://arxiv.org/abs/2605.03258v1
• Date: Tue, 05 May 2026 01:13:06 GMT
• Title: The Right Answer, the Wrong Direction: Why Transformers Fail at Counting and How to Fix It
• Abstract summary: Large language models often fail at simple counting tasks, even when the items to count are explicitly present in the prompt.<n>We investigate whether this failure occurs because transformers do not represent counts internally, or because they cannot convert those representations into the correct output tokens.<n>We find strong evidence for the second explanation. Linear probes recover the correct count from intermediate layers with near-perfect accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models often fail at simple counting tasks, even when the items to count are explicitly present in the prompt. We investigate whether this failure occurs because transformers do not represent counts internally, or because they cannot convert those representations into the correct output tokens. Across three model families, Pythia, Qwen3, and Mistral, ranging from 0.4B to 14B parameters, we find strong evidence for the second explanation. Linear probes recover the correct count from intermediate layers with near-perfect accuracy ($R^2>0.99$), showing that the information is present. However, the internal directions that encode counts are nearly orthogonal to the output-head rows for digit tokens ($|\cos|\leq0.032$). In other words, the model stores the count in a form that the digit logits do not naturally read out. We localize this failure with two interventions. Updating only the digit rows of the output head (36,864 parameters) substantially improves constrained next-token digit prediction (60.7 to 100.0% across four tasks), but it does not fix autoregressive generation. By contrast, a small LoRA intervention on attention Q/V weights (7.67M parameters) improves upstream routing and achieves 83.1% +/- 7.2% in true greedy autoregressive generation. Logit-lens measurements confirm the mechanism: the correct digit's vocabulary rank drops from 55,980 to 1, a 50,000x improvement. Additional norm, logit-lens, and cross-task analyses show that the bottleneck generalizes across character counting, addition, and list length, while remaining absent from broader multi-step reasoning benchmarks, including MMLU, GSM8K, and DROP. These results identify counting failure as a geometric readout bottleneck rather than a failure of internal representation: the model knows the count but the output pathway is geometrically misaligned with the tokens needed to express it.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution [0.0]
  We train transformers on a synthetic language where recency and rarity are exactly coextensive.<n>We separate them with a minimal causal edit that inverts one cue while holding the truth, token count and answer position fixed.
  arXiv  Detail & Related papers  (2026-08-25T12:09:19Z)
• When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs [0.0]
  Self-consistency via majority vote reduces per-problem accuracy on most GPQA Diamond problems for small instruction-tuned models.<n>This version reports that the most natural repair also fails, and separates three signal failures that v1 treated as one.
  arXiv  Detail & Related papers  (2026-08-11T20:08:51Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs [0.0]
  Large language models fail at counting repeated tokens despite strong performance on broader reasoning benchmarks.<n>We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer.
  arXiv  Detail & Related papers  (2026-05-10T00:45:19Z)
• Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts [51.84894623128418]
  Modern reasoning language models generate dense, sequential chain-of-thought traces implicitly assuming that every token contributes and that steps must be consumed in order.<n>We challenge both assumptions through a systematic intervention pipeline--removal, masking, shuffling, and noise injection--applied to model-generated reasoning chains.<n>We find that answer extraction operates on a sparse, order-insensitive, and structurally robust informational substrate.
  arXiv  Detail & Related papers  (2026-05-08T06:15:50Z)
• Measuring Representation Robustness in Large Language Models for Geometry [7.743292557234699]
  In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms.<n>Existing benchmarks report accuracy on fixed formats.<n>We propose GeoRepEval, a representation-aware evaluation framework.
  arXiv  Detail & Related papers  (2026-04-03T11:36:49Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models [22.525318796588568]
  Think-at-Hard (TaH) is a dynamic latent thinking method that iterates deeper only at hard tokens.<n>TaH boosts reasoning performance across five challenging benchmarks.
  arXiv  Detail & Related papers  (2025-11-11T18:57:02Z)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference [31.2331188304598]
  Changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses.<n>We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision.<n>Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32.
  arXiv  Detail & Related papers  (2025-06-11T08:23:53Z)
• VeriThinker: Learning to Verify Makes Reasoning Model Efficient [52.74493506816969]
  Large Reasoning Models excel at complex tasks using Chain-of-Thought (CoT) reasoning.<n>Their tendency to overthinking leads to unnecessarily lengthy reasoning chains.<n>We introduce VeriThinker, a novel approach for CoT compression.
  arXiv  Detail & Related papers  (2025-05-23T14:17:56Z)
• Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic [21.014229380679975]
  Large language models (LLMs) achieve impressive results on advanced mathematics benchmarks but sometimes fail on basic arithmetic tasks.<n>We investigate whether they have truly grasped fundamental arithmetic rules or are merely relying on pattern matching.<n>Our evaluation of 12 leading LLMs reveals a stark disconnect: while models achieve high numeric accuracy, they systematically fail these diagnostics.
  arXiv  Detail & Related papers  (2025-04-07T16:57:10Z)
• Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving [0.0]
  This study evalu-ates 10 large language models (LLMs) with 7 to 8 billion parameters using the MATH dataset.<n>The focus is on their ability to generate executable Python code as a step in their reasoning process, involving over 9,450 code executions.
  arXiv  Detail & Related papers  (2025-01-28T17:11:36Z)
• Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing [55.97957664897004]
  An effective recipe for building seq2seq, non-autoregressive, task-orienteds to map utterances to semantic frames proceeds in three steps. These models are typically bottlenecked by length prediction. In our work, we propose non-autoregressives which shift the decoding task from text generation to span prediction.
  arXiv  Detail & Related papers  (2021-04-15T07:02:35Z)
• Cryptanalytic Extraction of Neural Network Models [56.738871473622865]
  We introduce a differential attack that can efficiently steal the parameters of the remote model up to floating point precision. Our attack relies on the fact that ReLU neural networks are piecewise linear functions. We extract models that are 220 times more precise and require 100x fewer queries than prior work.
  arXiv  Detail & Related papers  (2020-03-10T17:57:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.