ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training

• URL: http://arxiv.org/abs/2609.16639v1
• Date: Tue, 15 Sep 2026 04:59:03 GMT
• Title: ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training
• Abstract summary: Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
• Score: 66.72708893922605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual post-training of large multimodal models should add new capabilities while preserving those from pre-training, and the two goals pull in opposite directions. SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting; on-policy methods such as RLVR and self-distillation preserve policy proximity yet supply little signal when the policy cannot yet solve the task. We introduce ReDraft (Reference-Driven Revision and Fine-Tuning), which obtains both from the model's own failures: using an expert response only as a reference, it has the model revise its own incorrect rollout, keeps the revision only if a verifier accepts it, and fine-tunes on what survives. Each retained target is therefore explicit, yet still close to the current policy. Across Counting, Clock Reading, and Jigsaw on Qwen2.5-VL-3B/7B, two of them with near zero accuracy, ReDraft gains 56.9 points on the target task against SFT's 52.9 while cutting prior-task loss from 16.6 to 1.5 points (11.3x less forgetting), and improves on OPSD along both axes (19.3 gain, 6.2 loss). Data- and parameter-space analyses match the design: revised targets are more probable under the base model, and the updates they induce stay compact and follow SFT's direction more closely than OPSD's. Repairing the model's own output, rather than replacing it with an expert's, is what lets one objective do both.

Related papers

• REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation [64.84437332310516]
  We introduce textbfREER-PT, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data.<n>REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation.<n>This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training.
  arXiv  Detail & Related papers  (2026-08-31T11:34:19Z)
• PROOF-Gen: From Optimized Data to Better Distillation [6.158612515104146]
  Post-training pipelines that drive tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle.<n>On 2-bench, 57% of teacher trials fail, two-thirds of them near-misses.<n>We introduce PROOF-Gen, which recovers golden trajectories from these failures via per-scenario prompt optimization.
  arXiv  Detail & Related papers  (2026-08-24T23:33:56Z)
• Invocation-Level Reliability of Tool-Using Agents [0.7874708385247353]
  We measure a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context.<n>By depth 6, roughly 70% of a model's own clean-context capability is lost to its own earlier mistakes.
  arXiv  Detail & Related papers  (2026-08-23T09:04:59Z)
• UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists [10.23869027501654]
  We present Upgrade, a decision-driven longitudinal benchmark covering four consecutive Qwen releases.<n>Upgrade disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether assets transfer across versions, and what recovery resources are usable.
  arXiv  Detail & Related papers  (2026-08-21T09:37:47Z)
• RL Forgets! Towards Continual Policy Optimization [57.4336338996653]
  Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks.<n>Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting.<n>We propose Continual Policy Optimization (CPO), a replay-free framework grounded in a prior-task behavioral KL objective.
  arXiv  Detail & Related papers  (2026-07-05T15:46:11Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• TASR: Training-Free Adaptive Stopping for Iterative Retrieval [12.177557521540082]
  Iterative retrieval-augmented generation agents commonly overspend by continuing to retrieve after the model has converged on an answer.<n>We propose TASR (Training-Free Adaptive Stopping Rule), a one-line predicate that fires when the model repeats its previous-round normalized answer.
  arXiv  Detail & Related papers  (2026-06-11T18:35:14Z)
• PURGE: Projected Unlearning via Retain-Guided Erasure [5.069332646113575]
  PURGE is a machine unlearning algorithm built on a simple but an under-exploited observation: continual learning (CL) and machine unlearning (MU)<n> CL tries to learn new tasks without forgetting old ones; MU tries to erase specific data without hurting retained performance representing the same underlying tension in opposite directions.<n> PURGE consistently keeps accuracy above 96% while achieving MIA AUROC close to 0.5 (the ideal), outperforming gradient ascent, KL-uniform, and several published baselines on the privacy-utility frontier.
  arXiv  Detail & Related papers  (2026-06-02T15:53:01Z)
• SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Models [48.335262141752715]
  Post-training pipeline for diffusion models has two stages: supervised fine-tuning (SFT) on curated data and reinforcement learning (RL) with reward models.<n>We propose SOAR (Self-Correction for Optimal Alignment and Refinement), a bias-correction post-training method that fills this gap.<n>It is on-policy, reward-free, and provides dense per-time supervision with no credit-assignment problem.
  arXiv  Detail & Related papers  (2026-04-14T11:45:15Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference [45.96968721472664]
  We introduce TDRM, a method for learning smoother and more reliable reward models.<n>Experiments show that TD-trained process reward models (PRMs) improve performance across Best-of-N (up to 6.6%) and tree-search (up to 23.7%) settings.
  arXiv  Detail & Related papers  (2025-09-18T16:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.