Invariant Reasoning Directions in Latent Trajectories of Language Models

• URL: http://arxiv.org/abs/2606.29164v1
• Date: Sun, 28 Jun 2026 02:59:08 GMT
• Title: Invariant Reasoning Directions in Latent Trajectories of Language Models
• Abstract summary: We show that latent reasoning trajectories contain stable invariant directions mixed with unstable instance-specific variation.<n>We introduce TILR, a training-free intervention framework for identifying and manipulating stable reasoning directions in latent space.<n>We find that TILR improves answer consistency under paraphrase by 10% and reduces latent trajectory variance by up to $50%$ while preserving reasoning accuracy.
• Score: 23.0736392766448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Latent reasoning models perform multi-step inference directly in hidden-state space, yet the structure of these latent reasoning trajectories remains poorly understood. We show that contrastive refinement signals between stronger and weaker reasoning trajectories exhibit a highly concentrated low-rank structure, while unconstrained latent updates remain sensitive to paraphrases, checkpoint choice, and trajectory perturbations. These observations suggest that latent reasoning trajectories contain stable invariant directions mixed with unstable instance-specific variation. We introduce \textbf{Trajectory-Invariant Latent Refinement (TILR)}, a training-free intervention framework for identifying and manipulating stable reasoning directions in latent space. TILR first learns a low-rank invariant subspace from contrastive trajectory differences across inputs, then constrains latent interventions to this subspace while suppressing poorly aligned updates through an adaptive alignment gate. Across six reasoning benchmarks, we find that a small number of latent directions explain most variation between strong and weak reasoning trajectories. Interventions on these directions causally improve reasoning consistency and reduce trajectory instability under paraphrases and perturbations. TILR improves answer consistency under paraphrase by ~10% and reduces latent trajectory variance by up to $50\%$ while preserving reasoning accuracy. These results support a geometric view of latent reasoning in which transferable reasoning behavior emerges from stable low-dimensional structure within hidden-state trajectories.

Related papers

• Learning to Refine Hidden States for Reliable LLM Reasoning [0.21485350418225244]
  Large language models show strong reasoning ability, but their internal reasoning process can remain unstable in complex multi-step settings.<n>We propose ReLAR, a reinforcement-guided latent refinement framework that iteratively updates hidden representations before decoding.
  arXiv  Detail & Related papers  (2026-06-16T05:03:27Z)
• Geometric Latent Reasoning Induces Shorter Generations in LLMs [24.29594681334716]
  We formulate latent reasoning as a geometric path-approximation problem within the model's pretrained token-embedding space.<n>We introduce Geometric Latent Reasoning (GLR), which uses a lightweight transition head to predict iterative direction updates in embedding space.<n> Evaluations on mathematical reasoning benchmarks using Qwen3 models reveal an emergent phenomenon: geometric latent reasoning induces substantially shorter generations without an explicit length objective.
  arXiv  Detail & Related papers  (2026-06-01T13:40:55Z)
• ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMs [37.068201407788884]
  We propose ProSR, a process-shaping optimization framework for spatial reasoning.<n>ProSR extends the optimization objective from single answer correctness to two process-level dimensions: visual dependence and trajectory stability.<n> Experiments show that ProSR improves answer accuracy while generating reasoning trajectories that are more stable and more dependent on visual evidence.
  arXiv  Detail & Related papers  (2026-05-25T07:27:28Z)
• GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler [54.10960908347221]
  We model latent thought exploration as conditional sampling from learnable densities and instantiate this idea as a Gaussian Thought Sampler (GTS)<n>GTS predicts context-dependent perturbation distributions over continuous reasoning states and is trained with GRPO-style policy optimization while keeping the backbone frozen.
  arXiv  Detail & Related papers  (2026-02-15T09:57:47Z)
• ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces [39.09794443825156]
  latent reasoning improves reasoning efficiency by replacing explicit reasoning trajectories with continuous representations in a latent space.<n>We show that thinking trajectories ending in incorrect answers contain fewer low-confidence steps than those ending in correct answers.<n>We propose Think, an inference-time confidence-aware routing mechanism, to avoid high confidence and noise for efficient reasoning.
  arXiv  Detail & Related papers  (2026-02-12T08:01:01Z)
• Latent Chain-of-Thought as Planning: Decoupling Reasoning from Verbalization [9.193078163792427]
  Chain-of-Thought (CoT) empowers Large Language Models (LLMs) to tackle complex problems.<n>Recent latent reasoning approaches attempt to optimize efficiency by performing reasoning within continuous hidden states.<n>We introduce PLaT, a framework that reformulates latent reasoning as planning by fundamentally decouple reasoning from verbalization.
  arXiv  Detail & Related papers  (2026-01-29T07:38:18Z)
• Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thought [33.267497114389734]
  Large Language Models (LLMs) have demonstrated strong reasoning capabilities through emphChain-of-Thought (CoT) prompting.<n>CoT methods rely on discrete token-level reasoning processes prone to error propagation and limited by vocabulary.<n>We propose EBM-CoT, an Energy-Based Chain-of-Thought framework that refines latent thought representations through an energy-based model.
  arXiv  Detail & Related papers  (2025-11-10T14:10:58Z)
• Directional Reasoning Injection for Fine-Tuning MLLMs [51.53222423215055]
  Multimodal large language models (MLLMs) are rapidly advancing, yet their reasoning ability often lags behind that of strong text-only counterparts.<n>Existing methods to bridge this gap rely on supervised fine-tuning over large-scale multimodal reasoning data or reinforcement learning.<n>We propose Directional Reasoning Injection for Fine-Tuning (DRIFT) to solve this problem.
  arXiv  Detail & Related papers  (2025-10-16T18:06:46Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• CTRLS: Chain-of-Thought Reasoning via Latent State-Transition [57.51370433303236]
  Chain-of-thought (CoT) reasoning enables large language models to break down complex problems into interpretable intermediate steps.<n>We introduce groundingS, a framework that formulates CoT reasoning as a Markov decision process (MDP) with latent state transitions.<n>We show improvements in reasoning accuracy, diversity, and exploration efficiency across benchmark reasoning tasks.
  arXiv  Detail & Related papers  (2025-07-10T21:32:18Z)
• A Survey on Latent Reasoning [100.54120559169735]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities.<n>CoT reasoning that verbalizes intermediate steps limits the model's expressive bandwidth.<n>Latent reasoning tackles this bottleneck by performing multi-step inference entirely in the model's continuous hidden state.
  arXiv  Detail & Related papers  (2025-07-08T17:29:07Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.