Length Penalties Make Chain-of-Thought Less Monitorable

• URL: http://arxiv.org/abs/2607.09786v1
• Date: Wed, 08 Jul 2026 14:18:26 GMT
• Title: Length Penalties Make Chain-of-Thought Less Monitorable
• Abstract summary: Training with length penalties does not stop misleading hints from steering models.<n>We train Qwen3-4B and Qwen3-14B variants with different target chain lengths.<n>We evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model's answer. In our experiments, training with length penalties does not stop misleading hints from steering models, even though the models' chains of thought mention the hint much less often. A token-accuracy evaluation would count these runs as successful because they use fewer reasoning tokens with little accuracy loss; it would miss whether the remaining trace still shows what drove the answer. We train Qwen3-4B and Qwen3-14B variants with different target chain lengths, then evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks. Compression sharply cuts reasoning tokens, preserves most multiple-choice accuracy, and leaves hint influence near baseline. At the strongest target, lower-bound faithfulness falls to 63.1% of baseline for Qwen3-14B and 69.4% for Qwen3-4B; the raw rate at which a monitor catches hint use falls from 69% to 49% and from 60% to 48%. To separate length from content, we randomly delete sentences from uncompressed baseline chains until the remaining text matches the compressed length. Even after this length matching, compressed chains disclose the hint 7-35 percentage points less often than baseline chains that we shorten at random, for both Qwen3 sizes and all five evaluation distributions. Compression therefore does more than shorten reasoning, preferentially removing the cues a monitor needs to see what influenced the answer. Together, these results reveal a compression-monitorability frontier in which cheaper reasoning can preserve answers while making the influences behind them harder to detect.

Related papers

• Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression [18.435649248662944]
  We show that when retained text contains an answer but deleted text defines the entity needed to interpret it, we call the result referential dangling.<n>We train a compact to rank omitted sentences by whether they are needed to interpret retained text and reinsert the top-ranked candidates without support annotations at inference.
  arXiv  Detail & Related papers  (2026-08-05T07:58:53Z)
• Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters [10.230437172157474]
  Chain-of-thought prompting improves LLM reasoning, but the source is contested.<n>We repeatedly sample each model on the same question and pair a shorter with a longer of its own natural generations that follow the same reasoning plan.<n>We find that the extra tokens leave accuracy essentially unchanged for every independently-trained reasoner.<n>We ask whether two traces expressing the same semantic content produce different outcomes when one is more verbose.
  arXiv  Detail & Related papers  (2026-06-29T11:06:46Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces [56.497263592610295]
  Language models often generate long chain-of-thought traces, but it remains unclear how much of this reasoning is necessary for preserving the final prediction.<n>We study this through the lens of overcomplete reasoning traces.<n>We define the minimal core as the smallest subset of steps that preserves either the final answer or predictive distribution.
  arXiv  Detail & Related papers  (2026-05-14T04:35:45Z)
• The Last Word Often Wins: A Format Confound in Chain-of-Thought Corruption Studies [0.0]
  Corruption studies infer which steps are computationally important'' from accuracy loss when steps are corrupted.<n>We show that when benchmark chains end with an explicit terminal answer line, these tests largely measure emphanswer placement rather than where intermediate computation is carried out.
  arXiv  Detail & Related papers  (2026-05-11T16:26:50Z)
• Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts [51.84894623128418]
  Modern reasoning language models generate dense, sequential chain-of-thought traces implicitly assuming that every token contributes and that steps must be consumed in order.<n>We challenge both assumptions through a systematic intervention pipeline--removal, masking, shuffling, and noise injection--applied to model-generated reasoning chains.<n>We find that answer extraction operates on a sparse, order-insensitive, and structurally robust informational substrate.
  arXiv  Detail & Related papers  (2026-05-08T06:15:50Z)
• Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning [66.22060690012512]
  Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy.<n>We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution.
  arXiv  Detail & Related papers  (2026-02-27T20:23:59Z)
• Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization [27.13234369037689]
  Chain-of-thought reasoning in large language models can trigger an "overthinking trap"<n>We propose mastery-gated, sample-level, soft reinforcement learning compression.<n>It penalizes long rollouts only when the model already solves the problem and has produced a shorter rollout.
  arXiv  Detail & Related papers  (2025-12-19T06:30:54Z)
• VeriThinker: Learning to Verify Makes Reasoning Model Efficient [52.74493506816969]
  Large Reasoning Models excel at complex tasks using Chain-of-Thought (CoT) reasoning.<n>Their tendency to overthinking leads to unnecessarily lengthy reasoning chains.<n>We introduce VeriThinker, a novel approach for CoT compression.
  arXiv  Detail & Related papers  (2025-05-23T14:17:56Z)
• CoT-Valve: Length-Compressible Chain-of-Thought Tuning [50.196317781229496]
  We introduce a new tuning and inference strategy named CoT-Valve, designed to allow models to generate reasoning chains of varying lengths.<n>We show that CoT-Valve successfully enables controllability and compressibility of the chain and shows better performance than the prompt-based control.
  arXiv  Detail & Related papers  (2025-02-13T18:52:36Z)
• Faithful Chain-of-Thought Reasoning [51.21714389639417]
  Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of reasoning tasks. We propose Faithful CoT, a reasoning framework involving two stages: Translation and Problem Solving. This guarantees that the reasoning chain provides a faithful explanation of the final answer.
  arXiv  Detail & Related papers  (2023-01-31T03:04:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.