Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

• URL: http://arxiv.org/abs/2606.09115v1
• Date: Mon, 08 Jun 2026 07:11:03 GMT
• Title: Counterfactual Transport Flows for Offline Conservative Trajectory Refinement
• Abstract summary: We propose a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback.<n>Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback.
• Score: 9.60771580834445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{counterfactual transport flows}, a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback. Given a low-feedback candidate trajectory, we construct local preference pairs from offline data by retrieving nearby trajectories in latent trajectory space with higher task-specific feedback, and use them as weak supervision for conservative refinement. The framework learns instance-specific refinement directions: at inference time, a refinement strength parameter controls how far the candidate trajectory is transported, enabling a trade-off between preserving the original behavior and applying stronger improvement. Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback, while providing interpretable trajectory-level refinement paths.

Related papers

• Revisiting Complete Reasoning Traces for Post-Training [61.76889440384448]
  Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability.<n>We show that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation.
  arXiv  Detail & Related papers  (2026-09-07T06:46:17Z)
• TRACE: Ergodic Trajectory Optimization for Active Scene Reconstruction [7.636751274394083]
  We study active reconstruction as an ergodic coverage problem.<n>Our approach derives this target distribution online from uncertainty and visibility.<n>We evaluate TRACE on the Replica dataset against the Next-Best-View (NBV) baselines, improving PSNR by 1.5 dB.
  arXiv  Detail & Related papers  (2026-08-03T14:30:20Z)
• In-Context Reinforcement Learning From Suboptimal Historical Data [56.60512975858003]
  Transformer models have achieved remarkable empirical successes, largely due to their in-context learning capabilities.<n>We propose the Decision Importance Transformer framework, which emulates the actor-critic algorithm in an in-context manner.<n>Our results show that DIT achieves superior performance, particularly when the offline dataset contains suboptimal historical data.
  arXiv  Detail & Related papers  (2026-01-27T23:13:06Z)
• ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts [22.46606397400043]
  We propose ASTRO, a data augmentation framework that generates distributionally novel and dynamics-consistent trajectories for offline RL.<n>ASTRO first learns a temporal-distance representation to identify distinct and reachable stitch targets.<n>We then employ a dynamics-guided stitch planner that adaptively generates connecting action sequences via Rollout Deviation Feedback.
  arXiv  Detail & Related papers  (2025-11-28T18:35:37Z)
• Hindsight Preference Learning for Offline Preference-based Reinforcement Learning [22.870967604847458]
  Offline preference-based reinforcement learning (RL) focuses on optimizing policies using human preferences between pairs of trajectory segments selected from an offline dataset. We propose to model human preferences using rewards conditioned on future outcomes of the trajectory segments. Our proposed method, Hindsight Preference Learning (HPL), can facilitate credit assignment by taking full advantage of vast trajectory data available in massive unlabeled datasets.
  arXiv  Detail & Related papers  (2024-07-05T12:05:37Z)
• Offline Reinforcement Learning from Datasets with Structured Non-Stationarity [50.35634234137108]
  Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation.
  arXiv  Detail & Related papers  (2024-05-23T02:41:36Z)
• Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning [24.654686581207343]
  The performance of offline reinforcement learning (RL) is sensitive to the proportion of high-return trajectories in the offline dataset. We propose Contrastive diffuser (CDiffuser) to make full use of low-return trajectories and improve the performance of offline RL algorithms.
  arXiv  Detail & Related papers  (2024-02-05T07:12:02Z)
• Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning [68.02988959934498]
  We propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories.<n> TR enhances learning efficiency by backward sampling of trajectories that optimize the use of subsequent state information.<n>We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL.
  arXiv  Detail & Related papers  (2023-06-27T14:29:44Z)
• Boosting Offline Reinforcement Learning via Data Rebalancing [104.3767045977716]
  offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets. We propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged. We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time.
  arXiv  Detail & Related papers  (2022-10-17T16:34:01Z)
• RORL: Robust Offline Reinforcement Learning via Conservative Smoothing [72.8062448549897]
  offline reinforcement learning can exploit the massive amount of offline data for complex decision-making tasks. Current offline RL algorithms are generally designed to be conservative for value estimation and action selection. We propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique.
  arXiv  Detail & Related papers  (2022-06-06T18:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.