Reasoning Fails Where Step Flow Breaks

• URL: http://arxiv.org/abs/2604.06695v1
• Date: Wed, 08 Apr 2026 05:21:13 GMT
• Title: Reasoning Fails Where Step Flow Breaks
• Abstract summary: We introduce Step-Saliency, which pools attention--gradient scores into step-to-step maps along the question--thinking--summary trajectory.<n>Across several models, Step-Saliency reveals two recurring information-flow failures: Shallow Lock-in and Deep Decay.<n>We propose StepFlow, a saliency-inspired test-time intervention that adjusts shallow saliency patterns measured by Step-Saliency via Odds-Equal Bridge and adds a small step-level residual in deep layers via Step Momentum Injection.
• Score: 9.62455367512369
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large reasoning models (LRMs) that generate long chains of thought now perform well on multi-step math, science, and coding tasks. However, their behavior is still unstable and hard to interpret, and existing analysis tools struggle with such long, structured reasoning traces. We introduce Step-Saliency, which pools attention--gradient scores into step-to-step maps along the question--thinking--summary trajectory. Across several models, Step-Saliency reveals two recurring information-flow failures: Shallow Lock-in, where shallow layers over-focus on the current step and barely use earlier context, and Deep Decay, where deep layers gradually lose saliency on the thinking segment and the summary increasingly attends to itself and the last few steps. Motivated by these patterns, we propose StepFlow, a saliency-inspired test-time intervention that adjusts shallow saliency patterns measured by Step-Saliency via Odds-Equal Bridge and adds a small step-level residual in deep layers via Step Momentum Injection. StepFlow improves accuracy on math, science, and coding tasks across multiple LRMs without retraining, indicating that repairing information flow can recover part of their missing reasoning performance.

Related papers

• Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models [86.02826269126308]
  We propose Trend-aware Pruning, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem.<n>This enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues.
  arXiv  Detail & Related papers  (2026-07-30T15:07:00Z)
• Offline-Online Curriculum RL for Multimodal Reasoning [28.712061130761654]
  Multimodal large language models exhibit capabilities on reasoning tasks, yet often produce flawed intermediate steps while yielding correct final answers.<n>We propose $O2$-CritiCuRL, a novel curriculum reinforcement learning framework that introduces critical-step awareness.<n>In the offline stage, $O2$-CritiCuRL conducts multi-rollout analysis over step-annotated trajectories to estimate step-level importance.<n>In the online stage, we employ a progressive step-level reinforcement learning strategy, where truncated chains guide the model to infer missing steps and refine its reasoning.
  arXiv  Detail & Related papers  (2026-07-26T14:47:47Z)
• Do Agents Think Deeper? A Mechanistic Investigation of Layer-Wise Dynamics in Sequential Planning [6.5668779922726666]
  Large language models (LLMs) may utilize their depth inefficiently in standard single-turn tasks.<n>We show that agentic reasoning exhibits a distinct depth profile from static tasks.
  arXiv  Detail & Related papers  (2026-05-27T04:08:08Z)
• Step-GRPO: Internalizing Dynamic Early Exit for Efficient Reasoning [39.067556802856444]
  Step-GRPO internalizes dynamic early-exit capabilities directly into the model.<n>Our method reduces token consumption by 32.0% compared to the vanilla model.
  arXiv  Detail & Related papers  (2026-04-18T07:48:53Z)
• DenoiseFlow: Uncertainty-Aware Denoising for Reliable LLM Agentic Workflows [20.319113495948294]
  We formalize the multi-step reasoning process as a Noisy MDP.<n>We propose DenoiseFlow, a closed-loop framework that performs progressive denoising through three coordinated stages.
  arXiv  Detail & Related papers  (2026-02-28T08:11:38Z)
• Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregation [32.86351316550696]
  We analyze raw long CoTs and uncover a reasoning hierarchy consisting of planning and execution steps.<n>Motivated by this observation, we propose Multi-Path Plan Aggregation (MPPA), a framework that augments single-pass reasoning with plan exploration and aggregation.<n>To overcome this, we introduce online Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision.
  arXiv  Detail & Related papers  (2025-10-13T17:02:41Z)
• Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation [82.2288581878096]
  We present a framework for difficulty-aware reasoning that teaches models to dynamically adjust reasoning depth based on problem complexity.<n>We show that models can be endowed with such dynamic inference pathways without any architectural modifications.
  arXiv  Detail & Related papers  (2025-09-05T16:40:13Z)
• StepWiser: Stepwise Generative Judges for Wiser Reasoning [52.32416311990343]
  Process reward models address this by providing step-by-step feedback.<n>Inspired by recent advances, we reframe stepwise reward modeling from a classification task to a reasoning task itself.<n>We show it provides (i) better judgment accuracy on intermediate steps than existing methods; (ii) can be used to improve the policy model at training time; and (iii) improves inference-time search.
  arXiv  Detail & Related papers  (2025-08-26T17:45:05Z)
• Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs [52.663816303997194]
  A key factor influencing answer quality is the length of the thinking stage.<n>This paper explores and exploits the mechanisms by which LLMs understand and regulate the length of their reasoning.<n>Our results demonstrate that this "overclocking" method mitigates overthinking, improves answer accuracy, and reduces inference latency.
  arXiv  Detail & Related papers  (2025-06-08T17:54:33Z)
• DISC: Dynamic Decomposition Improves LLM Inference Scaling [83.5708537758088]
  Inference scaling methods for LLMs often rely on decomposing problems into steps, followed by sampling and selecting the best next steps.<n>We propose dynamic decomposition, a method that adaptively and automatically partitions solution and reasoning traces into manageable steps during inference.
  arXiv  Detail & Related papers  (2025-02-23T20:37:32Z)
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [54.20763128054692]
  We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We prove that an interesting "task allocation" phenomenon emerges during the gradient flow dynamics.
  arXiv  Detail & Related papers  (2024-02-29T18:43:52Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• Low-memory stochastic backpropagation with multi-channel randomized trace estimation [6.985273194899884]
  We propose to approximate the gradient of convolutional layers in neural networks with a multi-channel randomized trace estimation technique. Compared to other methods, this approach is simple, amenable to analyses, and leads to a greatly reduced memory footprint. We discuss the performance of networks trained with backpropagation and how the error can be controlled while maximizing memory usage and minimizing computational overhead.
  arXiv  Detail & Related papers  (2021-06-13T13:54:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.