Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs

• URL: http://arxiv.org/abs/2603.26828v1
• Date: Fri, 27 Mar 2026 03:40:27 GMT
• Title: Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs
• Abstract summary: We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails.<n>We provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.

Related papers

• Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores [31.709549159768727]
  We observe a consistent readout gap across diverse reasoning benchmarks.<n> hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses.<n>Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic.
  arXiv  Detail & Related papers  (2026-08-31T16:43:55Z)
• RePair: Turning Retrieval Failures into Counterfactual Hard Pairs [15.148342372814662]
  RePair mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective.<n>On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples.
  arXiv  Detail & Related papers  (2026-08-30T06:47:38Z)
• Does Super-Resolution Preserve Defect Evidence? A Low-False-Call Benchmark for Semiconductor Inspection [4.906746849209501]
  Super-resolution can make inspection images appear sharper without preserving the evidence needed to detect a defect.<n>We study this failure mode with a benchmark that separates reconstruction from detection and evaluates both at a predeclared low false-positive rate.
  arXiv  Detail & Related papers  (2026-07-19T20:22:55Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuning [6.908972852063454]
  Uncertainty reliability can degrade earlier and more sharply than top-1 performance.<n>We study this empirically by measuring conformal coverage and calibration error on sequentially fine-tuned models.<n>We propose calibration replay, a lightweight post-hoc procedure that maintains a task-specific buffer and refits a task-specific conformal threshold.
  arXiv  Detail & Related papers  (2026-04-27T03:03:38Z)
• Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions [0.4086920237000588]
  We study how arithmetic answers are finalized after cross-token routing becomes causally irrelevant.<n>We find that Causal residual patching and cumulative attention ablations localize a sharp boundary near layer17.<n>In this regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately map inside a shared low-rank subspace.
  arXiv  Detail & Related papers  (2026-02-22T09:49:36Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• From Flows to Words: Can Zero-/Few-Shot LLMs Detect Network Intrusions? A Grammar-Constrained, Calibrated Evaluation on UNSW-NB15 [0.41998444721319217]
  Large Language Models (LLMs) can reason over natural-language inputs, but their role in intrusion detection without fine-tuning remains uncertain.<n>This study evaluates a promptonly approach by converting each network flow to a compact textual record and augmenting it with lightweight, domain-inspired flags.<n>We compare zero-shot, instruction-guided, and fewshot prompting to strong neural baselines under identical splits, reporting accuracy, precision, recall, F1, and macro scores.
  arXiv  Detail & Related papers  (2025-10-18T02:11:50Z)
• Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularization [85.50560211492898]
  Test-time adaptation (TTA) may fail to improve or even harm the model performance when test data have mixed distribution shifts.<n>This is often a key obstacle preventing existing TTA methods from being deployed in the real world.<n>We propose a sharpness-aware and reliable entropy minimization method, called SAR, for stabilizing TTA from two aspects.
  arXiv  Detail & Related papers  (2025-09-05T10:03:00Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning [59.44422468242455]
  We propose a novel method dubbed ShrinkMatch to learn uncertain samples. For each uncertain sample, it adaptively seeks a shrunk class space, which merely contains the original top-1 class. We then impose a consistency regularization between a pair of strongly and weakly augmented samples in the shrunk space to strive for discriminative representations.
  arXiv  Detail & Related papers  (2023-08-13T14:05:24Z)
• Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign? [12.557493236305211]
  We study benign overfitting in two-layer ReLU networks trained using gradient descent and hinge loss on noisy data for binary classification. We identify conditions on the margin of the clean data that give rise to three distinct training outcomes: benign overfitting, in which zero loss is achieved and with high probability test data is classified correctly; overfitting, in which zero loss is achieved but test data is misclassified with probability lower bounded by a constant; and non-overfitting, in which clean points, but not corrupt points, achieve zero loss and again with high probability test data is classified correctly.
  arXiv  Detail & Related papers  (2023-06-16T16:40:04Z)
• Beyond calibration: estimating the grouping loss of modern neural networks [68.8204255655161]
  Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
  arXiv  Detail & Related papers  (2022-10-28T07:04:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.