Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning

• URL: http://arxiv.org/abs/2511.19942v1
• Date: Tue, 25 Nov 2025 05:28:55 GMT
• Title: Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
• Abstract summary: It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to textitdiversity collapse, where outputs lack variety.<n>We introduce a principled method -- textitdifferential smoothing -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based languages.
• Score: 23.202550214006237
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to \textit{diversity collapse}, where outputs lack variety. Prior work has proposed a range of heuristics to counteract this effect, but these methods are ad hoc: they frequently trade off correctness for diversity, their effectiveness varies across tasks, and in some cases they even contradict one another. In this work, we place these observations on a rigorous foundation. We first provide a formal proof of why RL fine-tuning exhibits diversity collapse via a selection and reinforcement bias. Next, we make a key observation that any reward modification to address diversity collapse only needs to be applied on the correct trajectories. Building directly on this analysis, we introduce a principled method -- \textit{differential smoothing} -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based heuristics. Our theory precisely characterizes when existing heuristics help and why they fail, while showing that differential smoothing is universally superior. Extensive experiments with models from 1B to 7B parameters, across domains including CountDown and real-world mathematical reasoning, demonstrate consistent gains. Differential smoothing improves both Pass@1 and Pass@k, with up to 6.7\% improvements on AIME24 dataset.

Related papers

• Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design [59.05127237532803]
  Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.<n>We study this sensitivity through a principled analysis of generalization error and identify three critical factors.<n>We propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling.
  arXiv  Detail & Related papers  (2026-06-13T14:41:55Z)
• RAGEN-2: Reasoning Collapse in Agentic RL [123.40730283427136]
  In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic.<n>We decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI)<n>Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality.
  arXiv  Detail & Related papers  (2026-04-07T04:29:41Z)
• Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting [44.23640219583819]
  Reinforced Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting.<n>We propose Entropy-Adaptive Fine-Tuning (EAFT) to solve this problem.<n>EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.
  arXiv  Detail & Related papers  (2026-01-05T14:28:17Z)
• Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs [85.37131922131657]
  We reveal a critical yet underexplored flaw in Large Vision-Language Models (LVLMs)<n>Even when these models know the correct answer, they frequently arrive there through incorrect reasoning paths.<n>We propose PSO (Path-Select Optimization), a two-stage post-training framework designed to enhance both the reasoning performance and stability of existing LVLMs.
  arXiv  Detail & Related papers  (2025-12-06T03:02:55Z)
• Directional Reasoning Injection for Fine-Tuning MLLMs [51.53222423215055]
  Multimodal large language models (MLLMs) are rapidly advancing, yet their reasoning ability often lags behind that of strong text-only counterparts.<n>Existing methods to bridge this gap rely on supervised fine-tuning over large-scale multimodal reasoning data or reinforcement learning.<n>We propose Directional Reasoning Injection for Fine-Tuning (DRIFT) to solve this problem.
  arXiv  Detail & Related papers  (2025-10-16T18:06:46Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• On the Effect of Sampling Diversity in Scaling LLM Inference [57.31028064284527]
  Large language model (LLM) scaling inference is key to unlocking greater performance.<n>Motivated by the observed relationship between solution accuracy and meaningful response diversity, we systematically study the effect of prompt diversity in scaling inference.<n>We show that responses generated from meaningful diverse prompts after Best-of-$N$ selection exhibit significantly lower error rates than those produced from stationary prompts.
  arXiv  Detail & Related papers  (2025-02-16T07:37:58Z)
• Improving the Natural Language Inference robustness to hard dataset by data augmentation and preprocessing [1.7487745673871375]
  Natural Language Inference (NLI) is the task of inferring whether the hypothesis can be justified by the given premise.<n>We propose the data augmentation and preprocessing methods to solve the word overlap, numerical reasoning and length mismatch problems.
  arXiv  Detail & Related papers  (2024-12-10T01:49:23Z)
• Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing [55.791818510796645]
  We aim to develop models that generalize well to any diverse test distribution, even if the latter deviates significantly from the training data. Various approaches like domain adaptation, domain generalization, and robust optimization attempt to address the out-of-distribution challenge. We adopt a more conservative perspective by accounting for the worst-case error across all sufficiently diverse test distributions within a known domain.
  arXiv  Detail & Related papers  (2024-10-08T12:26:48Z)
• There is no Double-Descent in Random Forests [6.967385165474138]
  Random Forests (RFs) are among the state-of-the-art in machine learning and offer excellent performance with nearly zero parameter tuning. Recently, a broadly received study argued that a RF exhibits a so-called double-descent curve: First, the model overfits the data in a u-shaped curve and then, once a certain model complexity is reached, it suddenly improves its performance again. In this paper, we challenge the notion that model capacity is the correct tool to explain the success of RF and argue that the algorithm which trains the model plays a more important role than previously thought
  arXiv  Detail & Related papers  (2021-11-08T12:25:32Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control [42.960199987696306]
  Reinforcement learning experiments have notoriously high variance. Minor details can have disproportionately large effects on measured outcomes. We show that one cause for early variance is numerical instability.
  arXiv  Detail & Related papers  (2021-10-21T15:59:37Z)
• What causes the test error? Going beyond bias-variance via ANOVA [21.359033212191218]
  Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. Recent work aimed to understand in greater depth why overparametrization is helpful for generalization. We propose using the analysis of variance (ANOVA) to decompose the variance in the test error in a symmetric way.
  arXiv  Detail & Related papers  (2020-10-11T05:21:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.