The Hard Decision Layer: Evidence for Committed Inference in Transformers

• URL: http://arxiv.org/abs/2607.21613v1
• Date: Mon, 25 May 2026 03:54:43 GMT
• Title: The Hard Decision Layer: Evidence for Committed Inference in Transformers
• Abstract summary: We identify the _Hard Decision Layer_ (HDL), a natural architectural property where answer option rankings stabilize abruptly during inference.<n>Our results reveal striking accuracy improvements at the HDL: up to +0.61 (Qwen on CommonsenseQA)
• Score: 2.5782420501870296
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate where and how transformer-based language models commit to predictions in multiple-choice question answering. We identify the _Hard Decision Layer_ (HDL), a natural architectural property where answer option rankings stabilize abruptly during inference. Empirical validation across four language models (Qwen, Llama, Granite, Mistral) and four benchmark datasets demonstrates consistent HDL emergence without learned routing policies. We also show that the HDL is invariant to fine-tuning. Our results reveal striking accuracy improvements at the HDL: up to +0.61 (Qwen on CommonsenseQA), after which performance stabilizes. Systematic ablations on label formats and problem complexity confirm the phenomenon is fundamental to model architecture. These findings offer mechanistic insights into transformer inference and suggest opportunities for efficient reasoning and model steering. All code and results required to reproduce this work are available in https://github.com/Mystic-Slice/hard-decision-layer

Related papers

• Sparse Visual Thought Circuits in Vision-Language Models [2.5754366051855837]
  We develop a causal pipeline to localize and test sparse visual thought circuits in Qwen3-VL-8B.<n>We train SAEs at this layer, construct task-selective sets via an explicit rule, and perform inference time scaling and ablation while quantifying accuracy and drift.
  arXiv  Detail & Related papers  (2026-03-26T06:24:36Z)
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding [59.60915947702282]
  Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs)<n>Existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability.<n>We propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region.
  arXiv  Detail & Related papers  (2025-09-08T17:36:21Z)
• Learning-Order Autoregressive Models with Application to Molecular Graph Generation [52.44913282062524]
  We introduce a variant of ARM that generates high-dimensional data using a probabilistic ordering that is sequentially inferred from data.<n>We demonstrate experimentally that our method can learn meaningful autoregressive orderings in image and graph generation.
  arXiv  Detail & Related papers  (2025-03-07T23:24:24Z)
• Fourier Head: Helping Large Language Models Learn Complex Probability Distributions [7.074506869260538]
  We introduce a neural network layer, constructed using Fourier series, which we can easily substitute for any linear layer if we want the outputs to have a more continuous structure.<n>We perform extensive analysis on synthetic datasets, as well as on large-scale decision making and time series forecasting tasks.<n>All of our results support the effectiveness of our proposed Fourier head in scenarios where the underlying data distribution has a natural continuous structure.
  arXiv  Detail & Related papers  (2024-10-29T17:27:58Z)
• When should we prefer Decision Transformers for Offline Reinforcement Learning? [29.107029606830015]
  Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT) We study this question empirically by exploring the performance of these algorithms across the commonly used D4RL and Robomimicity benchmarks. We find that scaling the amount of data for DT by 5x gives a 2.5x average score improvement on Atari.
  arXiv  Detail & Related papers  (2023-05-23T22:19:14Z)
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference [72.61732440246954]
  Large pre-trained language models often lack logical consistency across test inputs. We propose a framework, ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models. We show that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models.
  arXiv  Detail & Related papers  (2022-11-21T21:58:30Z)
• Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probes [8.591839265985412]
  We present four confusion probes to test for problems such as prior bias and choice paralysis. We show that the model exhibits significant prior bias and to a lesser, but still highly significant degree, choice paralysis, in addition to other problems. Our results suggest that stronger testing protocols and additional benchmarks may be necessary before the language models are used in front-facing systems or decision making with real world consequences.
  arXiv  Detail & Related papers  (2022-10-03T22:36:44Z)
• IOT: Instance-wise Layer Reordering for Transformer Structures [173.39918590438245]
  We break the assumption of the fixed layer order in the Transformer and introduce instance-wise layer reordering into the model structure. Our method can also be applied to other architectures beyond Transformer.
  arXiv  Detail & Related papers  (2021-03-05T03:44:42Z)
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [80.82194311274694]
  We examine the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness.
  arXiv  Detail & Related papers  (2020-12-02T03:53:13Z)
• Query Training: Learning a Worse Model to Infer Better Marginals in Undirected Graphical Models with Hidden Variables [11.985433487639403]
  Probabilistic graphical models (PGMs) provide a compact representation of knowledge that can be queried in a flexible way. We introduce query training (QT), a mechanism to learn a PGM that is optimized for the approximate inference algorithm that will be paired with it. We demonstrate experimentally that QT can be used to learn a challenging 8-connected grid Markov random field with hidden variables.
  arXiv  Detail & Related papers  (2020-06-11T20:34:32Z)
• Robust Question Answering Through Sub-part Alignment [53.94003466761305]
  We model question answering as an alignment problem. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets.
  arXiv  Detail & Related papers  (2020-04-30T09:10:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.