Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

• URL: http://arxiv.org/abs/2606.21906v1
• Date: Sat, 20 Jun 2026 07:03:26 GMT
• Title: Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding
• Abstract summary: We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer.<n> Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks.
• Score: 92.1521161575198
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Autoregressive generation in large language models (LLMs) conventionally decodes from the final layer, assuming that deeper representations yield more reliable next-token predictions. We revisit this assumption by revealing a recurring Guess-Refine-Perturb dynamic: early layers form coarse guesses, intermediate layers refine reasoning-relevant semantics, and final layers can perturb these refined predictions toward generic or alignment-preferred tokens. We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer through entropy-guided conservative backward search. We further provide a theoretical formulation of layer selection as an optimal stopping problem, showing that under bounded projection noise and dominant late-stage alignment perturbation, our search rule filters perturbation while bounding the loss relative to the oracle refinement layer. Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks, including GPQA-Diamond, Omni-MATH, and HLE, with zero memory overhead and less than 2% latency increase. These results suggest dynamically bypassing final-layer perturbations can unlock stronger reasoning behavior from aligned LLMs.

Related papers

• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• BLADE: Boundary-Expanded and Layer-Adaptive Dynamic Exit for Efficient LLM Reasoning [12.482949673236563]
  BLADE terminates reasoning by estimating whether the generated prefix is sufficient for correct answering.<n> BLADE constructs multi-granular checkpoints from sentence, self-doubt, and paragraph boundaries.<n>Experiments show that BLADE preserves near-baseline accuracy while reducing generated tokens by 24.8% on Qwen3-8B and 15.8% on Qwen3-4B.
  arXiv  Detail & Related papers  (2026-07-31T02:36:55Z)
• Depth Exploration for LLM Decoding [0.920033202445524]
  LLM decoding evaluates every generated token through the full layer stack, even though many tokens become predictable at intermediate depths.<n>We propose Depth Exploration Decoding (DEX) to replace single-depth selection with parallel exploration over multiple candidate depths.<n>DEX outperforms representative depth-selection baselines and achieves competitive end-to-end throughput against speculative and distributed decoding methods.
  arXiv  Detail & Related papers  (2026-06-28T06:22:09Z)
• SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration [13.454534256560558]
  Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models.<n>We propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision.<n>We show that our method achieves up to 2.33x wall-time speedup over standard autoregressive decoding.
  arXiv  Detail & Related papers  (2026-04-14T03:47:04Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• When Less Is More? Diagnosing ASR Predictions in Sardinian via Layer-Wise Decoding [39.94212430129149]
  intermediate layers in multilingual speech models often encode more phonetically accurate representations than the final output layer.<n>We show that truncating upper transformer layers leads to improved Phoneme Error Rates (PER), with the best performance achieved not at the final layer, but two layers earlier.
  arXiv  Detail & Related papers  (2026-02-10T22:45:03Z)
• PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendations [52.67948063133533]
  Generative Recommendation has emerged as a promising paradigm, reformulating recommendation as a sequence-to-sequence generation task over hierarchical Semantic IDs.<n>Existing methods suffer from a critical issue we term Semantic Drift, where errors in early, high-level tokens irreversibly divert the generation trajectory into irrelevant semantic subspaces.<n>We propose Promise, a novel framework that integrates dense, step-by-step verification into generative models.
  arXiv  Detail & Related papers  (2026-01-08T07:38:46Z)
• Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations [15.542741121573203]
  LAGER is a framework for improving the alignment of "LLM-as-a-Judge" evaluations with human scores by leveraging internal representations.<n>We evaluate our method on the standard alignment benchmarks Flask, HelpSteer, and BIGGen using Spearman correlation, and find that LAGER achieves improvements of up to 7.5% over the best baseline.
  arXiv  Detail & Related papers  (2025-08-05T15:18:36Z)
• FIRP: Faster LLM inference via future intermediate representation prediction [54.897493351694195]
  FIRP generates multiple tokens instead of one at each decoding step. We conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets.
  arXiv  Detail & Related papers  (2024-10-27T15:53:49Z)
• Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models [55.45444773200529]
  Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination. Recent work has focused on decoding techniques to improve factuality during inference.
  arXiv  Detail & Related papers  (2024-04-14T19:45:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.