Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions

• URL: http://arxiv.org/abs/2602.19109v1
• Date: Sun, 22 Feb 2026 09:49:36 GMT
• Title: Post-Routing Arithmetic in Llama-3: Last-Token Result Writing and Rotation-Structured Digit Directions
• Abstract summary: We study how arithmetic answers are finalized after cross-token routing becomes causally irrelevant.<n>We find that Causal residual patching and cumulative attention ablations localize a sharp boundary near layer17.<n>In this regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately map inside a shared low-rank subspace.
• Score: 0.4086920237000588
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study three-digit addition in Meta-Llama-3-8B (base) under a one-token readout to characterize how arithmetic answers are finalized after cross-token routing becomes causally irrelevant. Causal residual patching and cumulative attention ablations localize a sharp boundary near layer~17: beyond it, the decoded sum is controlled almost entirely by the last input token and late-layer self-attention is largely dispensable. In this post-routing regime, digit(-sum) direction dictionaries vary with a next-higher-digit context but are well-related by an approximately orthogonal map inside a shared low-rank subspace (low-rank Procrustes alignment). Causal digit editing matches this geometry: naive cross-context transfer fails, while rotating directions through the learned map restores strict counterfactual edits; negative controls do not recover.

Related papers

• Behaviorally Effective LoRA Writes Are Sparse and Structured [0.0]
  Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained write actually carry behavior.<n>We show that behaviorally effective LoRA writes are sparse, structured, and far more concentrated than the raw low-rank parameterization suggests.
  arXiv  Detail & Related papers  (2026-09-01T15:09:43Z)
• Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models [0.0]
  We study the hidden-state inversion problem as a continuous embedding-space optimisation.<n>We analyse which tokens break the reconstructions and find a sharp categorical asymmetry.<n>Last-layer hidden states of GPT-2 are as sensitive as original text.
  arXiv  Detail & Related papers  (2026-07-01T12:18:12Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs [0.0]
  Large language models fail at counting repeated tokens despite strong performance on broader reasoning benchmarks.<n>We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer.
  arXiv  Detail & Related papers  (2026-05-10T00:45:19Z)
• Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs [0.0]
  We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails.<n>We provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.
  arXiv  Detail & Related papers  (2026-03-27T03:40:27Z)
• Are you going to finish that? A Practical Study of the Partial Token Problem [85.49816027251013]
  Language models (LMs) are trained over sequences of tokens, whereas users interact with LMs via text.<n>This mismatch gives rise to the partial token problem, which occurs when a user ends their prompt in the middle of the expected next-token.<n>In this work, we identify three domains where token and "word" boundaries often do not line up.
  arXiv  Detail & Related papers  (2026-01-30T17:47:16Z)
• Liquid Reasoning Transformers: A Sudoku-Based Prototype for Chess-Scale Algorithmic Tasks [0.15469452301122175]
  Liquid Reasoning Transformer (LRT) is a transformer architecture designed for inference with adaptive depths using iterative changes, discard-based correction, and a learned stopping mechanism.<n>We evaluate the LRT on Sudoku as a controlled testbed for structured reasoning and show that it achieves strong performance, reaching 98.68% digit accuracy and 36.30% full-puzzle accuracy without using symbolic rules or search.
  arXiv  Detail & Related papers  (2025-12-14T18:20:05Z)
• Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs [0.4086920237000588]
  We analyze the internal arithmetic processes in LLaMA-3-8B-Instruct.<n>Inspired by the step-by-step manner in which humans perform addition, we propose and analyze a coherent four-stage trajectory.<n>Near the output, the model organizes and generates the final content, with the correct token reliably occupying the top rank.
  arXiv  Detail & Related papers  (2025-06-09T14:48:43Z)
• Fully-Geometric Cross-Attention for Point Cloud Registration [51.865371511201765]
  Point cloud registration approaches often fail when the overlap between point clouds is low due to noisy point correspondences.<n>This work introduces a novel cross-attention mechanism tailored for Transformer-based architectures that tackles this problem.<n>We integrate the Gromov-Wasserstein distance into the cross-attention formulation to jointly compute distances between points across different point clouds.<n>At the point level, we also devise a self-attention mechanism that aggregates the local geometric structure information into point features for fine matching.
  arXiv  Detail & Related papers  (2025-02-12T10:44:36Z)
• Rethinking PGD Attack: Is Sign Function Necessary? [131.6894310945647]
  We present a theoretical analysis of how such sign-based update algorithm influences step-wise attack performance. We propose a new raw gradient descent (RGD) algorithm that eliminates the use of sign. The effectiveness of the proposed RGD algorithm has been demonstrated extensively in experiments.
  arXiv  Detail & Related papers  (2023-12-03T02:26:58Z)
• A Strong Baseline for Point Cloud Registration via Direct Superpoints Matching [7.308509114539376]
  We propose a simple and effective baseline to find correspondences of superpoints in a global matching manner. Our simple yet effective baseline shows comparable or even better results than state-of-the-art methods on three datasets.
  arXiv  Detail & Related papers  (2023-07-03T21:33:40Z)
• REGTR: End-to-end Point Cloud Correspondences with Transformers [79.52112840465558]
  We conjecture that attention mechanisms can replace the role of explicit feature matching and RANSAC. We propose an end-to-end framework to directly predict the final set of correspondences. Our approach achieves state-of-the-art performance on 3DMatch and ModelNet benchmarks.
  arXiv  Detail & Related papers  (2022-03-28T06:01:00Z)
• Geometric Transformer for Fast and Robust Point Cloud Registration [53.10568889775553]
  We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios. We propose Geometric Transformer to learn geometric feature for robust superpoint matching.
  arXiv  Detail & Related papers  (2022-02-14T13:26:09Z)
• Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation [109.46348908829697]
  We propose a novel Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a target n-gram with all n-grams in the generated sequence. We conduct experiments on three tasks: machine translation with noisy target sequences, unsupervised text style transfer, and non-autoregressive machine translation.
  arXiv  Detail & Related papers  (2021-06-29T03:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.