Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting

• URL: http://arxiv.org/abs/2603.16542v1
• Date: Tue, 17 Mar 2026 14:05:35 GMT
• Title: Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting
• Abstract summary: Posterior-Transition Reweighting (PTR) is a reward-free and conservative post-training method.<n>It decides how much each training sample should influence the supervised update.<n>PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation.
• Score: 36.23660195981511
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.

Related papers

• RoboDrop: Curating VLA Post-Training Data via Local Gradient Compatibility [17.469738521824905]
  We introduce textscRoboDrop, a data-curation framework that audits supervision using local gradient compatibility measured along the training trajectory.<n>We evaluate RoboDrop on controlled observation--action corruptions, naturally suboptimal demonstrations in simulation, and real-robot datasets containing non-expert collection errors.
  arXiv  Detail & Related papers  (2026-09-09T10:55:10Z)
• JEPA Policy: Diffusion-Free Imitation Learning via Paired Action and Future Representation Prediction [6.889062419237111]
  JEPA Policy is a diffusion-free framework that uses the action chunk and its observed future representation as paired training targets.<n>Future prediction can therefore shape the representation used to generate actions.<n>Results support paired future-representation supervision as a practical approach to low-latency visuomotor imitation without iterative generative sampling.
  arXiv  Detail & Related papers  (2026-09-09T02:42:20Z)
• ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback [1.711436622393758]
  Online Residual Policy Adaptation (ORPA) is a framework that enables feedback-driven correction of robot actions without modifying the underlying policy parameters.<n>ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space.<n>We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.
  arXiv  Detail & Related papers  (2026-08-18T03:30:12Z)
• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation [73.2142090645987]
  SC3-Eval is a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator.<n>We demonstrate SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison.
  arXiv  Detail & Related papers  (2026-06-17T02:15:46Z)
• Implicit Neural Representations of Individual Behavior [0.0]
  We study policy representation learning from unlabeled multi-policy behavioral data.<n> Behavioral INR represents a policy as a state-action function mapping states to subsequent actions.<n> Behavioral INR most consistently improves policy identifiability in the hardest continuous state-action settings.
  arXiv  Detail & Related papers  (2026-06-10T15:19:41Z)
• Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Corrections [25.961145214027255]
  Diffusion policies provide paired supervision in the form of the robot's undesired actions and the human teacher's corrective actions.<n>We propose Set-Supervised Diffusion Policy (SDP), a novel learning framework that utilizes contrastive action-chunk data to train diffusion policies from human corrections.<n>SDP consistently improves policy performance, with particularly strong gains in robustness to noisy data.
  arXiv  Detail & Related papers  (2026-06-01T08:14:38Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Double Check My Desired Return: Transformer with Target Alignment for Offline Reinforcement Learning [64.6334337560557]
  Reinforcement learning via supervised learning (RvS) frames offline RL as a sequence modeling task.<n>Decision Transformer (DT) struggles to reliably align the actual achieved returns with specified target returns.<n>We propose Doctor, a novel approach that Double Checks the Transformer with target alignment for Offline RL.
  arXiv  Detail & Related papers  (2025-08-22T14:30:53Z)
• Conformal Inference under High-Dimensional Covariate Shifts via Likelihood-Ratio Regularization [35.16750653336608]
  We introduce the likelihood ratio regularized quantile regression algorithm, which combines the pinball loss with a novel choice of regularization.<n>We show that the LR-QR method has coverage at the desired level in the target domain, up to a small error term.<n>Our experiments demonstrate that the LR-QR algorithm outperforms existing methods on high-dimensional prediction tasks.
  arXiv  Detail & Related papers  (2025-02-18T16:46:44Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Adaptive Principal Component Regression with Applications to Panel Data [29.295938927701396]
  We provide the first time-uniform finite sample guarantees for (regularized) Principal component regression. Our results rely on adapting tools from modern martingale concentration to the error-in-variables setting. We show that our method empirically outperforms a baseline which does not leverage error-in-variables regression.
  arXiv  Detail & Related papers  (2023-07-03T21:13:40Z)
• Post Reinforcement Learning Inference [20.521169740409263]
  We study estimation and inference using data collected by reinforcement learning algorithms.<n>We propose a weighted generalized method of moments (GMM) that uses adaptive weights to stabilize this variance.<n>Key applications include dynamic treatment effect estimation and dynamic off-policy evaluation.
  arXiv  Detail & Related papers  (2023-02-17T12:53:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.