Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach

• URL: http://arxiv.org/abs/2603.20899v1
• Date: Sat, 21 Mar 2026 18:17:17 GMT
• Title: Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach
• Abstract summary: Shortcut-Aware Reasoning Training (SART) is a gradient-aware framework that detects and mitigates shortcut-promoting samples.<n>Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration.
• Score: 31.233459232809608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models exhibit strong reasoning capabilities, yet often rely on shortcuts such as surface pattern matching and answer memorization rather than genuine logical inference. We propose Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that detects and mitigates shortcut-promoting samples via ShortcutScore and gradient surgery. Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration, and modifies training dynamics accordingly. Experiments on controlled reasoning benchmarks show that SART achieves +16.5% accuracy and +40.2% robustness over the strongest baseline, significantly improving generalization under distribution shifts. Code is available at: https://github.com/fuyanjie/short-cut-aware-data-centric-reasoning.

Related papers

• Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space [78.05046281165885]
  Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form.<n>GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes.
  arXiv  Detail & Related papers  (2026-08-31T14:54:30Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Suppressing Forgery-Specific Shortcuts for Generalizable Deepfake Detection [59.056076147081775]
  Deepfake detection suffers from poor generalization across forgery methods.<n>Recent approaches attempt to improve generalization, but lack explicit mechanism to identify and suppress such shortcuts.<n>We propose Shortcut Subspace Suppression (S3) framework that explicitly characterizes and suppresses method-specific shortcuts.
  arXiv  Detail & Related papers  (2026-06-01T07:54:58Z)
• Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning [10.87972575497941]
  Chain-of-thought (CoT) reasoning is useful for monitoring language models.<n>Models can rely on prompt-to-answer shortcuts that bypass the CoT.<n>We study CoT faithfulness through a structural information-flow perspective.
  arXiv  Detail & Related papers  (2026-05-22T23:37:29Z)
• Models Know Their Shortcuts: Deployment-Time Shortcut Mitigation [5.949467977832433]
  Shortcut Guardrail mitigates token-level shortcuts without access to the original training data or shortcut annotations.<n>Shortcut Guardrail improves overall accuracy and worst-group accuracy over the unmitigated model under distribution shifts.
  arXiv  Detail & Related papers  (2026-04-14T04:43:29Z)
• SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation [21.945468909808923]
  Large language models often default to step-by-step computation even when efficient numerical shortcuts are available.<n>This raises a basic question: do they exhibit number sense in a human-like behavioral sense?<n>We introduce SenseMath, a benchmark for evaluating structure-sensitive numerical reasoning in LLMs.
  arXiv  Detail & Related papers  (2026-04-02T12:50:42Z)
• Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models [59.6715047267181]
  Small reasoning models (SRMs) are prone to hallucinations, especially in intermediate reasoning steps.<n>Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained chain-of-thought evaluation.<n>We propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model.
  arXiv  Detail & Related papers  (2026-02-05T17:15:12Z)
• Shortcut Invariance: Targeted Jacobian Regularization in Disentangled Latent Space [7.8904984750896885]
  Deep neural networks are prone to learning shortcuts, spurious and easily learned correlations.<n>We present a simple and effective training method that renders the classifier functionally invariant to shortcut signals.<n>We analyze this as targeted Jacobian regularization, which forces the classifier to ignore spurious features and rely on more complex, core semantic signals.
  arXiv  Detail & Related papers  (2025-11-24T07:09:08Z)
• Diagnosing Shortcut-Induced Rigidity in Continual Learning: The Einstellung Rigidity Index (ERI) [7.587193411022608]
  Shortcut features undermine robustness and reduce reliability under distribution shifts.<n>In continual learning (CL), the consequences of shortcut exploitation can persist and intensify.<n>In CL, shortcut-induced rigidity throttles the acquisition of new ones.
  arXiv  Detail & Related papers  (2025-10-01T03:52:40Z)
• Fast Thinking for Large Language Models [67.7238685892317]
  We introduce Latent Codebooks for Fast Thinking, a framework that uses concise CoT sketches only during training to learn a codebook of discrete strategy priors.<n>At inference, the model conditions on a handful of continuous thinking switches distilled from the codebook in a single pass, enabling strategy-level guidance without producing explicit reasoning tokens.
  arXiv  Detail & Related papers  (2025-09-28T04:19:48Z)
• Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching [64.74765550805024]
  Chain-of-Thought prompting elicits step-by-step problem solving, but often at the cost of excessive verbosity in intermediate outputs.<n>We propose Sketch-of-Thought (SoT), a prompting framework that integrates cognitively inspired reasoning paradigms with linguistic constraints.<n>SoT achieves token reductions of up to 84% with minimal accuracy loss across 18 reasoning datasets.
  arXiv  Detail & Related papers  (2025-03-07T06:57:17Z)
• Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback [94.25162866972077]
  Step-KTO is a training framework that combines process-level and outcome-level binary feedback.<n>Our experiments show that Step-KTO significantly improves both final answer accuracy and the quality of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-01-18T15:38:03Z)
• Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropy [22.69591517487717]
  We show that default-ERM's preference for maximizing the margin leads to models that depend more on the shortcut than the stable feature. We develop loss functions that encourage uniform-margin solutions, called margin control (MARG-CTRL)
  arXiv  Detail & Related papers  (2023-08-24T04:39:25Z)
• Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers [44.97660597940641]
  We show that generative models alone are not sufficient to prevent shortcut learning. In particular, we propose Chroma-VAE, a two-pronged approach where a VAE is initially trained to isolate the shortcut in a small latent subspace. In addition to demonstrating the efficacy of Chroma-VAE on benchmark and real-world shortcut learning tasks, our work highlights the potential for manipulating the latent space of generative classifiers to isolate or interpret specific correlations.
  arXiv  Detail & Related papers  (2022-11-28T11:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.