ReGuide: From Test-Time Guidance to Self-Improving Diffusion Policies

• URL: http://arxiv.org/abs/2606.28939v1
• Date: Sat, 27 Jun 2026 14:23:21 GMT
• Title: ReGuide: From Test-Time Guidance to Self-Improving Diffusion Policies
• Abstract summary: We introduce ReGuide, a framework that treats guided rollouts as reusable on-policy recovery data.<n>Onmimic Can, Square, Transport, and Tool Hang, ReGuide improves base-policy success by $1.3$--$7.7times$, outperforms LPB in the test-time-only setting.
• Score: 20.452659267550217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Behavior-cloned diffusion policies are expressive but remain vulnerable to covariate shift: small deviations from demonstrated states can compound into task failure. Existing methods address this either by expanding the training distribution through expert corrections or synthetic augmentation, or by steering a frozen policy at test time with guidance from a learned model. The former can be expensive or assumption-dependent, while the latter discards the corrected trajectories after execution. We introduce ReGuide, a self-improving framework that treats guided rollouts as reusable on-policy recovery data. ReGuide first uses Phase-Conditioned Guidance (PCG) to generate corrective rollouts: it constructs phase-specific latent targets, applies guidance only in the drifted-but-recoverable regime, and guides through the estimated clean action to match the dynamics model's training distribution. Successful guided rollouts are then absorbed back into the policy through ReGuide-FT, which fine-tunes the current checkpoint, or ReGuide-FS, which retrains from scratch on the augmented dataset; the two can also be composed and iterated. On Robomimic Can, Square, Transport, and Tool Hang, ReGuide improves base-policy success by $1.3$--$7.7\times$, outperforms LPB in the test-time-only setting, and matched-data ablations show that the gains come from guided recovery data rather than additional rollouts alone.

Related papers

• FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience [18.64380706071327]
  FlowBalance is a verifier-grounded self-improvement method that learns a normalized distribution over complete responses.<n>For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains.<n>Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses.
  arXiv  Detail & Related papers  (2026-09-03T00:47:11Z)
• EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning [25.200068256200293]
  reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update.<n>We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds.<n> EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector.
  arXiv  Detail & Related papers  (2026-08-22T13:16:25Z)
• Adaptive Recurrent Message Passing for Test Time Computing on Graphs [80.56829418752513]
  We show that recurrent graph models can overcome the intrinsic mismatch between graph data and architectural designs.<n>We propose AdaR, an Adaptive Recurrent graph model, empowering flexible test-time computing on various downstream tasks without changing model parameters.
  arXiv  Detail & Related papers  (2026-06-21T12:14:11Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Implicit Drifting Policy: One-Step Action Generation via Conditional Expert Geometry [46.0185525503119]
  Implicit Drifting Policy (IDP) is a one-step imitation learning framework for robot control.<n>IDP brings the training-time correction of Drifting into policy learning without explicit vector field estimation.
  arXiv  Detail & Related papers  (2026-05-31T08:39:57Z)
• Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation [71.16738415436458]
  We introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision.<n>RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook.<n> Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines.
  arXiv  Detail & Related papers  (2026-05-12T20:46:05Z)
• RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models [90.39703013636868]
  RePO-VLA is a recovery-driven policy optimization framework.<n>It assigns distinct roles to success, recovery, and failure trajectories.<n>It raises adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
  arXiv  Detail & Related papers  (2026-05-10T08:24:05Z)
• Interactive Critique-Revision Training for Reliable Structured LLM Generation [18.00222080273147]
  We propose DPA-GRPO, a paired-action training method for a two-player generator--verifier game with structured verifier interventions.<n>We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation.<n>Experiments on TaxCalc TY24 show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL baselines.
  arXiv  Detail & Related papers  (2026-05-08T17:00:38Z)
• PPGuide: Steering Diffusion Policies with Performance Predictive Guidance [28.698103923760172]
  PPGuide steers a pre-trained diffusion policy away from failure modes at inference time.<n>It uses attention-based multiple instance learning to automatically estimate which observation-action chunks from the policy's rollouts are relevant to success or failure.<n>We validated our proposed PPGuide across a diverse set of tasks from the Robomimic and MimicGen benchmarks.
  arXiv  Detail & Related papers  (2026-03-11T17:10:16Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Unifying Stable Optimization and Reference Regularization in RLHF [64.16830602324345]
  This paper introduces a unified regularization approach that balances objectives of preventing reward hacking and maintaining stable policy updates.<n>Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity.
  arXiv  Detail & Related papers  (2026-02-12T03:31:19Z)
• Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behavior [58.751981587234916]
  This paper formalizes the Refinement Provenance Inference (RPI) audit task as Refinement Provenance Inference (RPI)<n>We propose RePro, a logit-based framework that fuses teacher-forced likelihood features with logit-ranking signals.<n>During training, RePro learns a transferable representation via shadow fine-tuning, and uses a lightweight linear head to infer provenance on unseen victims without training-data access.
  arXiv  Detail & Related papers  (2026-01-05T10:16:41Z)
• Policy Gradient Guidance Enables Test Time Control [9.24766442685354]
  We introduce Policy Gradient Guidance (PGG)<n>PGG augments the policy gradient with an unconditional branch and interpolates conditional and unconditional branches.<n>We evaluate PGG on discrete and continuous control benchmarks.
  arXiv  Detail & Related papers  (2025-10-02T16:00:35Z)
• Feedback Guidance of Diffusion Models [14.162420300295365]
  Interval-Free Guidance (CFG) has become standard for improving sample fidelity in conditional diffusion models.<n>We propose FeedBack Guidance (FBG), which uses a state-dependent coefficient to self-regulate guidance amounts based on need.
  arXiv  Detail & Related papers  (2025-06-06T13:46:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.