Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuning

• URL: http://arxiv.org/abs/2605.04431v1
• Date: Wed, 06 May 2026 02:50:00 GMT
• Title: Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuning
• Abstract summary: We propose an automatic failure management framework for reinforcement fine-tuning.<n>RFT-FaultBench is the first benchmark for fine-grained failures in reinforcement fine-tuning.<n>RFT-FM shows strong capability in detecting, diagnosing, and mitigating RFT failures.
• Score: 44.75827047453428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement fine-tuning (RFT) has become a core paradigm for post-training large language models, yet its training process remains highly fragile. Existing efforts mainly improve reliability at the system level or address specific issues in individual subproblems by modifying RFT algorithms. Despite their effectiveness, they largely overlook the problem of failure management at the training-process level. When training goes wrong, practitioners still rely heavily on expert-driven manual inspection and correction, and automatic failure management for RFT remains largely unexplored. In this paper, we take a first step toward systematic failure management for reinforcement fine-tuning. To understand the empirical structure of RFT failures, we first construct RFT-FaultBench, the first benchmark for fine-grained failures in reinforcement fine-tuning, covering 5 fault families, 16 fault types, 779 training runs, 22,549 train-step records, and 1,457,288 trajectory-level records. Based on this benchmark, we conduct a comprehensive empirical study showing that RFT failures are both observable from training dynamics and distinguishable through their empirical fault fingerprints. Building on these findings, we propose RFT-FM, an automatic failure management framework for reinforcement fine-tuning that unifies anomaly detection, failure diagnosis, and auto remediation in a closed loop. Experimental results show that RFT-FaultBench is neither trivial nor saturated: it exhibits clear anomaly structure while still posing substantial challenges, especially under subtle fault settings. Moreover, RFT-FM shows strong capability in detecting, diagnosing, and mitigating RFT failures.

Related papers

• FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents [49.519184792774304]
  FailForge is an agentic framework that converts failed rollouts into training signal.<n>FailForge recovers over 26% of previously failed instances at marginal additional cost.
  arXiv  Detail & Related papers  (2026-08-09T08:22:57Z)
• Tracing Agentic Failure from the Flow of Success [7.071269758592172]
  We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data.<n>We propose OAT, which casts this problem as one-class learning with neural controlled differential equations.<n>With training on only 100 successful trajectories, OAT is 200--5000 $times$ faster than prompting-based baselines.
  arXiv  Detail & Related papers  (2026-07-14T13:16:14Z)
• DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training [12.82389963785782]
  DRIFT is an online self-evolution policy optimization framework for large language models.<n>It regulates the model's self-improvement process through the joint use of Difficulty Routing and Rhythm Gating.<n> evaluated across five benchmarks and three model scales, DRIFT surpasses the peak performance of both GRPO and SDPO.
  arXiv  Detail & Related papers  (2026-06-29T14:20:47Z)
• Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis [68.28714988482703]
  Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs)<n>We present a study revealing that general-domain PRMs struggle to supervise data analysis agents.<n>We introduce DataPRM, a novel environment-aware generative process reward model.
  arXiv  Detail & Related papers  (2026-04-27T09:00:30Z)
• Trust-Region Adaptive Policy Optimization [82.09255251747818]
  Post-training methods play an important role in improving large language models' (LLMs) complex reasoning abilities.<n>We introduce TRAPO, a framework that interleavesSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) within each training instance.<n>Experiments on five mathematical reasoning benchmarks show that TRAPO consistently surpasses standard SFT, RL, and SFT-then-RL pipelines.
  arXiv  Detail & Related papers  (2025-12-19T14:37:07Z)
• Anchored Supervised Fine-Tuning [26.17356786243252]
  Post-training of large language models involves a trade-off between supervised fine-tuning and reinforcement learning.<n> Dynamic Fine-Tuning (DFT) recently emerged as a promising middle ground, reweighting SFT objectives with token probabilities.<n>We propose Anchored Supervised Fine-Tuning (ASFT) to augment DFT's reweighting with lightweight KL regularization to preserve tightness while ensuring stability.
  arXiv  Detail & Related papers  (2025-09-28T08:58:12Z)
• Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training [36.69514399442043]
  This paper presents a comparative analysis of two core post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT)<n>Our experiments are conducted on a benchmark comprising seven diverse multimodal tasks.
  arXiv  Detail & Related papers  (2025-07-07T18:17:06Z)
• Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective [98.45690529036848]
  Post-training algorithms such as Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) are widely used to adapt multimodal large language models to downstream tasks.<n>While effective at task adaptation, their impact on prior knowledge remains unclear.
  arXiv  Detail & Related papers  (2025-06-30T04:15:01Z)
• Exploring Expert Failures Improves LLM Agent Tuning [74.0772570556016]
  We propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories.<n>EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance.
  arXiv  Detail & Related papers  (2025-04-17T17:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.