SPAR: Support-Preserving Action Rectification

• URL: http://arxiv.org/abs/2605.27877v1
• Date: Wed, 27 May 2026 02:53:41 GMT
• Title: SPAR: Support-Preserving Action Rectification
• Abstract summary: We propose Support-Preserving Action Rectification (SPAR), which reframes global learning as a local residual rectification anchored to a frozen pure behavior cloning policy.<n>This framework performs fine-grained fitting and local policy improvement in the residual space, thereby contracting the search space.<n>We further introduce Latent Self-Imitation, utilizing a latent-sampling weighted-regression mechanism to address fitting-improvement gradient conflict in the residual space.
• Score: 21.412581761834996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline policy improvement faces an inherent conflict between maximizing value and fitting the data distribution. While in-sample weighted regression is stable, it suffers from over-conservatism that suppresses high-value actions in the distribution tail; conversely, gradient-based approaches often exhibit a fitting-optimization conflict of gradients, which drives the policy off the data manifold. To address this, we propose Support-Preserving Action Rectification (SPAR), which reframes global learning as a local residual rectification anchored to a frozen pure behavior cloning policy. This framework performs fine-grained fitting and local policy improvement in the residual space, thereby contracting the search space. We further introduce Latent Self-Imitation, utilizing a latent-sampling weighted-regression mechanism to address fitting-improvement gradient conflict in the residual space. Theoretically, we prove this mechanism eliminates the manifold-normal drift of standard value gradients, while extensive D4RL experiments show SPAR extracts significant gains from suboptimal baselines to achieve state-of-the-art performance.

Related papers

• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• DegBins: Degradation-Driven Binning for Depth Super-Resolution [57.61783461543096]
  DegBins is a novel DSR framework that leverages degradation-driven binning to adaptively enhance residual modeling.<n>DegBins consistently outperforms existing state-of-the-art methods in terms of accuracy, robustness, and generalization.
  arXiv  Detail & Related papers  (2026-05-10T16:17:11Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching [42.26405543045843]
  Action-based Policy Compression (APC) is widely recognized as sample-inefficient.<n>We introduce Occupancy-based Policy Compression (OPC), which enhances APC by shifting behavior representation from immediate action-matching to long-horizon state-space coverage.<n>These modifications force the generative model to organize the latent space around true functional similarity, promoting a latent representation that generalizes over a broad spectrum of behaviors.
  arXiv  Detail & Related papers  (2026-03-27T23:16:27Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Score-Guided Proximal Projection: A Unified Geometric Framework for Rectified Flow Editing [1.0312968200748118]
  Rectified Flow models achieve state-of-the-art generation quality, yet controlling them for precise tasks remains a challenge.<n>Current approaches bifurcate into inversion-based guidance, which suffers from "geometric locking"<n>We propose Score-Guided Proximal Projection, a unified framework that bridges the gap between deterministic optimization and contraction sampling.
  arXiv  Detail & Related papers  (2026-03-05T23:44:45Z)
• Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration [42.24031155235307]
  BaryIR is a representation learning framework that aligns multisource degraded features in the Wasserstein barycenter (WB) space.<n>We show that BaryIR performs competitively against state-of-the-art all-in-one methods.
  arXiv  Detail & Related papers  (2026-02-26T16:31:27Z)
• Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification [14.911955979675772]
  We propose Anchored Policy Optimization (APO) to shift the paradigm from global Shape Matching to Support Coverage.<n>APO breaks the accuracy-diversity trade-off, significantly improving Pass@1 while restoring the Pass@K diversity typically lost by standard policy gradient methods.
  arXiv  Detail & Related papers  (2026-02-05T14:41:57Z)
• Less is More: Clustered Cross-Covariance Control for Offline RL [13.198112768636207]
  A fundamental challenge in offline reinforcement learning is distributional shift.<n>We propose partitioned buffer sampling that restricts updates to localized replay partitions.<n>We also introduce an explicit gradient-based corrective penalty that cancels the covariance induced bias within each update.
  arXiv  Detail & Related papers  (2026-01-28T16:55:04Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.