Offline Imitation Learning with Variational Counterfactual Reasoning

• URL: http://arxiv.org/abs/2310.04706v4
• Date: Fri, 29 Dec 2023 09:40:55 GMT
• Title: Offline Imitation Learning with Variational Counterfactual Reasoning
• Abstract summary: In offline imitation learning (IL), an agent aims to learn an optimal expert behavior policy without additional online environment interactions. We propose a framework named underlineOffline underlineImitation underlineLearning with underlineCounterfactual data underlineAugmentation (OILCA) by doing counterfactual inference.
• Score: 17.344961438658427
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In offline imitation learning (IL), an agent aims to learn an optimal expert behavior policy without additional online environment interactions. However, in many real-world scenarios, such as robotics manipulation, the offline dataset is collected from suboptimal behaviors without rewards. Due to the scarce expert data, the agents usually suffer from simply memorizing poor trajectories and are vulnerable to variations in the environments, lacking the capability of generalizing to new environments. To automatically generate high-quality expert data and improve the generalization ability of the agent, we propose a framework named \underline{O}ffline \underline{I}mitation \underline{L}earning with \underline{C}ounterfactual data \underline{A}ugmentation (OILCA) by doing counterfactual inference. In particular, we leverage identifiable variational autoencoder to generate \textit{counterfactual} samples for expert data augmentation. We theoretically analyze the influence of the generated expert data and the improvement of generalization. Moreover, we conduct extensive experiments to demonstrate that our approach significantly outperforms various baselines on both \textsc{DeepMind Control Suite} benchmark for in-distribution performance and \textsc{CausalWorld} benchmark for out-of-distribution generalization. Our code is available at \url{https://github.com/ZexuSun/OILCA-NeurIPS23}.

Related papers

• OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis [19.76083673142665]
  OPERA is a multi-agent ensemble framework that addresses expert weight assignment as an offline policy learning problem.<n>We evaluate OPERA on 9 datasets covering fundus photography, chest X-ray, CT, MRI, and multimodal diagnostic benchmarks.
  arXiv  Detail & Related papers  (2026-07-27T22:12:45Z)
• Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use [14.813603277906623]
  We formalize OpenAgent, a problem setting characterized by distributional shifts across query, action, observation, and domain dimensions.<n>Our analysis yields a series of key insights, demonstrating that agents trained via both Supervised Fine-Tuning(SFT) and Reinforcement Learning suffer from varying degrees of performance degradation.<n>We propose Perturbation-Augmented Fine-Tuning, a disturbance-based intervention strategy for SFT that lays the foundation for enhancing agent robustness and utility in realistic environments.
  arXiv  Detail & Related papers  (2026-07-01T15:40:25Z)
• Causal Representation Learning for Generalisable Recommendation [1.8124328823188354]
  Predictive models trained on observational data often fail to generalise to the distributions they encounter when deployed.<n>We propose an information-theoretic disentanglement criterion and prove that its optimum depends only on the causal components of the input.<n>Our headline evaluation is an A/B test with millions of users on Spotify, applied to a production ranker for personalised playlist generation.
  arXiv  Detail & Related papers  (2026-05-26T13:58:36Z)
• SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment [49.86376148975563]
  Large language models (LLMs) have revolutionized natural language processing through their capabilities in understanding and executing diverse tasks.<n> supervised fine-tuning, particularly in Retrieval-Augmented Generation (RAG) scenarios, often leads to catastrophic forgetting.<n>We propose SelfAug, a self-distribution alignment method that aligns input sequence logits to preserve the model's semantic distribution.
  arXiv  Detail & Related papers  (2025-09-04T06:50:47Z)
• Generate to Discriminate: Expert Routing for Continual Learning [59.71853576559306]
  Generate to Discriminate (G2D) is a continual learning method that leverages synthetic data to train a domain-discriminator. We observe that G2D outperforms competitive domain-incremental learning methods on tasks in both vision and language modalities.
  arXiv  Detail & Related papers  (2024-12-22T13:16:28Z)
• AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning [98.26836657967162]
  textbfAgentOhana aggregates agent trajectories from distinct environments, spanning a wide array of scenarios. textbfxLAM-v0.1, a large action model tailored for AI agents, demonstrates exceptional performance across various benchmarks.
  arXiv  Detail & Related papers  (2024-02-23T18:56:26Z)
• Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust? [91.682459306359]
  In continual learning (CL), an AI agent learns from non-stationary data streams under dynamic environments. In this paper, a novel CL framework is proposed to achieve robust generalization to dynamic environments while retaining past knowledge. The generalization and memorization performance of the proposed framework are theoretically analyzed.
  arXiv  Detail & Related papers  (2023-09-18T21:00:01Z)
• Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning [10.00979536266327]
  We present a novel approach to address the challenge of generalization in offline reinforcement learning (RL) Specifically, we aim to improve the agent's ability to generalize to out-of-distribution goals. We learn a new policy offline based on the augmented dataset, with an off-the-shelf offline RL algorithm.
  arXiv  Detail & Related papers  (2023-09-14T10:22:33Z)
• Personalized Federated Learning via Amortized Bayesian Meta-Learning [21.126405589760367]
  We introduce a new perspective on personalized federated learning through Amortized Bayesian Meta-Learning. Specifically, we propose a novel algorithm called emphFedABML, which employs hierarchical variational inference across clients. Our theoretical analysis provides an upper bound on the average generalization error and guarantees the generalization performance on unseen data.
  arXiv  Detail & Related papers  (2023-07-05T11:58:58Z)
• Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations [5.760034336327491]
  We study the problem of offline Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. We introduce an additional discriminator to distinguish expert and non-expert data. Our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.
  arXiv  Detail & Related papers  (2022-07-20T17:29:04Z)
• Adapting to Online Label Shift with Provable Guarantees [137.89382409682233]
  We formulate and investigate the problem of online label shift. The non-stationarity and lack of supervision make the problem challenging to be tackled. Our algorithms enjoy optimal dynamic regret, indicating that performance is competitive with a clairvoyant nature.
  arXiv  Detail & Related papers  (2022-07-05T15:43:14Z)
• Leveraging Expert Guided Adversarial Augmentation For Improving Generalization in Named Entity Recognition [50.85774164546487]
  Named Entity Recognition (NER) systems often demonstrate great performance on in-distribution data, but perform poorly on examples drawn from a shifted distribution. We propose leveraging expert-guideds to change the entity tokens and their surrounding contexts thereby altering their entity types as adversarial attacks. We found that state-of-the-art NER systems trained on CoNLL 2003 training data drop performance dramatically on our challenging set.
  arXiv  Detail & Related papers  (2022-03-21T01:21:12Z)
• Where is the Grass Greener? Revisiting Generalized Policy Iteration for Offline Reinforcement Learning [81.15016852963676]
  We re-implement state-of-the-art baselines in the offline RL regime under a fair, unified, and highly factorized framework. We show that when a given baseline outperforms its competing counterparts on one end of the spectrum, it never does on the other end.
  arXiv  Detail & Related papers  (2021-07-03T11:00:56Z)
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators [19.026312915461553]
  We propose a model-based offline reinforcement learning (RL) approach called PerSim. We first learn a personalized simulator for each agent by collectively using the historical trajectories across all agents prior to learning a policy. This representation suggests a simple, regularized neural network architecture to effectively learn the transition dynamics per agent, even with scarce, offline data.
  arXiv  Detail & Related papers  (2021-02-13T17:16:41Z)
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning [39.477038093585726]
  Deep reinforcement learning (RL) agents often fail to generalize to unseen scenarios. Data augmentation has recently been shown to improve the sample efficiency and generalization of RL agents. We show that our agent learns policies and representations that are more robust to changes in the environment that do not affect the agent.
  arXiv  Detail & Related papers  (2020-06-23T09:50:22Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.