TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2607.21111v1
• Date: Thu, 23 Jul 2026 09:40:40 GMT
• Title: TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning
• Abstract summary: We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR)<n>TOUR combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and privacy auditing.<n>We show that common deletion baselines have environment-dependent privacy-utility behavior.
• Score: 18.411026872661083
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult because a lower membership score can reflect trajectory removal, residual memorization visible to another attack, or policy collapse that destroys useful behavior. We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR), a benchmark that combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Across D4RL locomotion experiments and an exploratory AntMaze extension, TOUR shows that common deletion baselines have environment-dependent privacy-utility behavior. Retraining and fine-tuning often provide stronger retained-utility references than uniform GA+Refit, while TrajDeleter remains a useful comparator but is not uniformly stronger under the same audit. Reference-model, threshold, deviation, equivalence, action-error, representation-based, and query-limited attacks further show that a single likelihood-based membership score can overstate deletion quality. In the evaluated settings, conclusions about offline RL unlearning are therefore not stable under single-score auditing. They depend on matched non-member construction, retraining-relative calibration, attack family, retained utility, and explicit scope for diagnostic architecture or component-level evidence.

Related papers

• FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience [18.64380706071327]
  FlowBalance is a verifier-grounded self-improvement method that learns a normalized distribution over complete responses.<n>For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains.<n>Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses.
  arXiv  Detail & Related papers  (2026-09-03T00:47:11Z)
• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Group-Reflective Self-Distillation for Agentic Reinforcement Learning [8.041052886037766]
  Group-Reflective Self-Distillation (GRSD) derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts.<n>GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
  arXiv  Detail & Related papers  (2026-07-30T11:49:46Z)
• Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It [23.538941671680806]
  Tool use enables large language models to perform complex tasks.<n>Recent agentic reinforcement learning methods show promise for enhancing model capabilities.<n>Some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail.
  arXiv  Detail & Related papers  (2026-06-24T16:55:56Z)
• Erased, but Not Gone: Output Forgetting Is Not True Forgetting [9.095371772236932]
  We show that standard output-level evaluation can systematically overestimate the success of unlearning.<n>Under this stronger lens, current methods often appear forgotten at the output layer while exhibiting a structured mismatch relative to retraining.<n>This structured mismatch is characterized by forget/retain asymmetry, directional mismatch, and concentrated residuals along retraining-related directions.
  arXiv  Detail & Related papers  (2026-06-23T16:23:53Z)
• PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents [57.69188221227875]
  We propose a Privileged trAce Co-Training framework for multi-turn tool-use agents.<n>The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints.<n>We show that PACT consistently improves over strong SFT- and RL-based baselines.
  arXiv  Detail & Related papers  (2026-06-15T04:46:23Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• TraceHiding: Scalable Machine Unlearning for Mobility Data [1.6042394978941517]
  TraceHiding is a scalable, importance-aware machine unlearning framework for mobility trajectory data.<n>It removes specified user trajectories from trained deep models without full retraining.<n>It achieves superior unlearning accuracy, competitive membership inference attack (MIA) resilience, and up to 40times speedup over retraining.
  arXiv  Detail & Related papers  (2025-09-21T21:29:24Z)
• UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning [57.081646768835704]
  User specifications or legal frameworks often require information to be removed from pretrained models, including large language models (LLMs)<n>This requires deleting or "forgetting" a set of data points from an already-trained model, which typically degrades its performance on other data points.<n>We propose UPCORE, a method-agnostic data selection framework for mitigating collateral damage during unlearning.
  arXiv  Detail & Related papers  (2025-02-20T22:51:10Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning [33.125187822259186]
  Offline Goal-Conditioned Reinforcement Learning (GCRL) is tasked with learning to achieve multiple goals in an environment purely from offline datasets using sparse reward functions. We present a novel approach to GCRL under a new lens of mixture-distribution matching, leading to our discriminator-free method: SMORe.
  arXiv  Detail & Related papers  (2023-11-03T16:19:33Z)
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning [14.432131909590824]
  Offline Reinforcement Learning aims to train effective policies using previously collected datasets. Standard off-policy RL algorithms are prone to overestimations of the values of out-of-distribution (less explored) actions. We improve the behavior regularized offline reinforcement learning and propose BRAC+.
  arXiv  Detail & Related papers  (2021-10-02T23:55:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.