What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis

• URL: http://arxiv.org/abs/2606.20075v1
• Date: Thu, 18 Jun 2026 10:51:53 GMT
• Title: What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis
• Abstract summary: We analyze Latent Chain-of-Thought (CoT) from an information-theoretic perspective and identify this failure as a dual collapse.<n>To measure these effects, we introduce the Unified Latent Probe (ULP), which quantifies the mutual information between latent trajectories and explicit reasoning steps.
• Score: 24.026935801991744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent Chain-of-Thought (CoT) internalizes reasoning within continuous hidden states, offering a promising alternative to verbose discrete reasoning traces. However, robust latent reasoning remains difficult because outcome supervision provides weak learning signals and leaves latent trajectories prone to semantic drift. In this work, we analyze Latent CoT from an information-theoretic perspective and identify this failure as a dual collapse: gradient attenuation along the optimization path and representational drift in the latent space. We further decompose process supervision into two complementary dimensions: Trajectory Supervision, which injects dense stepwise reasoning signals, and Space Supervision, which preserves the semantic structure of the latent manifold. Our analysis shows that rigid geometric compression can collapse the reasoning space, whereas generative reconstruction provides a more flexible semantic anchor that better preserves information capacity. To measure these effects, we introduce the Unified Latent Probe (ULP), which quantifies the mutual information between latent trajectories and explicit reasoning steps. Experiments reveal a clear Information-Performance Binding: reasoning accuracy depends on the information fidelity preserved in the latent chain. These findings provide a principled framework for latent reasoning supervision and suggest shifting from geometric imitation toward mutual information maximization. Our code is available at \href{https://github.com/EIT-NLP/Supervision-in-Latent-CoT}{this repository}.

Related papers

• Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning [3.216436532980238]
  Chain-of-thought (CoT) reasoning improves the reasoning ability of large language models by introducing intermediate computation.<n>However, explicit rationales increase decoding length, latency, and context cost.<n>Implicit CoT offers a more efficient alternative by moving intermediate reasoning into continuous latent states.
  arXiv  Detail & Related papers  (2026-09-07T12:18:51Z)
• Interpreting Latent CoT Reasoning as Dynamical Systems [2.2448294058653455]
  We model latent token sequences as trajectories in representation space.<n>We show that latent CoT exhibits structured, non-random dynamics with two distinct stability classes.<n>This framework advances the interpretability of latent CoT reasoning dynamics.
  arXiv  Detail & Related papers  (2026-06-20T16:02:55Z)
• Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Intervention [33.40772052822545]
  This paper bridges the gap between mechanistic interpretability and actionable control.<n>We first present a systematic analysis using structural, causal, and geometric probes.<n>We then operationalize these interpretability insights into a suite of training-free, decode-time interventions.
  arXiv  Detail & Related papers  (2026-05-31T13:52:24Z)
• Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning [10.87972575497941]
  Chain-of-thought (CoT) reasoning is useful for monitoring language models.<n>Models can rely on prompt-to-answer shortcuts that bypass the CoT.<n>We study CoT faithfulness through a structural information-flow perspective.
  arXiv  Detail & Related papers  (2026-05-22T23:37:29Z)
• Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model [56.21523258053447]
  SCOLAR (Self-COnsistent LAtent Reasoning) introduces a lightweight detransformer that generates auxiliary visual tokens in a single shot.<n>SCOLAR extends acceptable latent CoT length by over $30times$, achieves state-of-the-art among open-source models on real-world reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T14:13:08Z)
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs [54.16324124242172]
  Continuous latent-space reasoning offers a compact alternative to textual chain-of-thought for multimodal models.<n>We identify a previously overlooked optimization pathology in existing latent visual reasoning methods.<n>We show that inference-time latent optimization, without any parameter updates, effectively unleashes the suppressed reasoning capacity of visual latents.
  arXiv  Detail & Related papers  (2026-05-04T15:36:12Z)
• GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler [54.10960908347221]
  We model latent thought exploration as conditional sampling from learnable densities and instantiate this idea as a Gaussian Thought Sampler (GTS)<n>GTS predicts context-dependent perturbation distributions over continuous reasoning states and is trained with GRPO-style policy optimization while keeping the backbone frozen.
  arXiv  Detail & Related papers  (2026-02-15T09:57:47Z)
• No Global Plan in Chain-of-Thought: Uncover the Latent Planning Horizon of LLMs [65.783709850324]
  This work stems from prior complementary observations on the dynamics of Chain-of-Thought (CoT): Large Language Models (LLMs)<n>LLMs are shown latent planning of subsequent reasoning prior to CoT emergence, thereby diminishing the significance of explicit CoT.<n>We investigate the latent planning strength of LLMs, through our probing method, Tele-Lens, applying to hidden states across diverse task domains.
  arXiv  Detail & Related papers  (2026-02-02T13:46:56Z)
• Beyond Dense States: Elevating Sparse Transcoders to Active Operators for Latent Reasoning [25.054823575096332]
  We propose LSTR (Latent Sparse Transcoder Reasoning), a latent reasoning framework that elevates sparse transcoders into active reasoning operators.<n>At its core, LSTR employs a Latent Transition Transcoder (LTT) with a residual skip architecture that decouples linear manifold transport from sparse semantic updates.<n>Extensive experiments show that LSTR preserves reasoning accuracy and compression efficiency while substantially improving interpretability.
  arXiv  Detail & Related papers  (2026-02-02T06:08:35Z)
• Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought [64.43689151961054]
  We theoretically analyze the training dynamics of a simplified two-layer transformer on the directed graph reachability problem.<n>Our analysis reveals that during training using continuous thought, the index-matching logit will first increase and then remain bounded under mild assumptions.
  arXiv  Detail & Related papers  (2025-09-27T15:23:46Z)
• A Survey on Latent Reasoning [100.54120559169735]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities.<n>CoT reasoning that verbalizes intermediate steps limits the model's expressive bandwidth.<n>Latent reasoning tackles this bottleneck by performing multi-step inference entirely in the model's continuous hidden state.
  arXiv  Detail & Related papers  (2025-07-08T17:29:07Z)
• Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
  We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
  arXiv  Detail & Related papers  (2023-11-03T03:18:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.