Causal Imitation Learning under Temporally Correlated Noise

• URL: http://arxiv.org/abs/2202.01312v1
• Date: Wed, 2 Feb 2022 22:33:08 GMT
• Title: Causal Imitation Learning under Temporally Correlated Noise
• Abstract summary: We develop algorithms for imitation learning from policy data corrupted by temporally correlated noise in expert actions. In particular, we present two techniques, one of a generative-modeling flavor (DoubIL) that can utilize access to a simulator, and one of a game-theoretic flavor (ResiduIL) that can be run entirely offline.
• Score: 39.4969161422156
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We develop algorithms for imitation learning from policy data that was corrupted by temporally correlated noise in expert actions. When noise affects multiple timesteps of recorded data, it can manifest as spurious correlations between states and actions that a learner might latch on to, leading to poor policy performance. To break up these spurious correlations, we apply modern variants of the instrumental variable regression (IVR) technique of econometrics, enabling us to recover the underlying policy without requiring access to an interactive expert. In particular, we present two techniques, one of a generative-modeling flavor (DoubIL) that can utilize access to a simulator, and one of a game-theoretic flavor (ResiduIL) that can be run entirely offline. We find both of our algorithms compare favorably to behavioral cloning on simulated control tasks.

Related papers

• Difference-Aware Retrieval Policies for Imitation Learning [8.909659724448819]
  Parametric imitation learning via behavior cloning can suffer from poor generalization to out-of-distribution states.<n>We show that reusing the training data during inference via a semi-parametric retrieval-based imitation learning approach can alleviate this challenge.<n>We present Difference-Aware Retrieval Policies for Imitation Learning (DARP), a semi-parametric retrieval-based imitation learning approach.
  arXiv  Detail & Related papers  (2026-06-08T17:18:19Z)
• Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback [59.768119380109084]
  This paper introduces an interactive continual learning paradigm where AI models dynamically learn new skills from real-time human feedback.<n>We propose RiCL, a Reinforced interactive Continual Learning framework leveraging Large Language Models (LLMs)<n>Our RiCL approach substantially outperforms existing combinations of state-of-the-art online continual learning and noisy-label learning methods.
  arXiv  Detail & Related papers  (2025-05-15T03:22:03Z)
• Noise-Tolerant Coreset-Based Class Incremental Continual Learning [0.6486052012623045]
  This work focuses on label noise and instance noise in the context of class-incremental learning (CIL) We derive a new bound for the robustness of a method to uncorrelated instance noise under a general additive noise threat model. We show that existing memory-based CL are not robust whereas the proposed methods exhibit significant improvements in maximizing classification accuracy.
  arXiv  Detail & Related papers  (2025-04-23T14:34:20Z)
• Diverse Controllable Diffusion Policy with Signal Temporal Logic [13.154661571539577]
  We leverage Signal Temporal Logic (STL) and Diffusion Models to learn controllable, diverse, and rule-aware policy. In the closed-loop testing, our approach reaches the highest diversity, rule satisfaction rate, and the least collision rate. A case study on human-robot encounter scenarios shows our approach can generate diverse and closed-to-oracle trajectories.
  arXiv  Detail & Related papers  (2025-03-04T18:59:00Z)
• A Unifying Framework for Causal Imitation Learning with Hidden Confounders [17.34321540867036]
  We propose a general and unifying framework for causal Imitation Learning (IL) Our framework accounts for two types of hidden confounders: (a) those observed by the expert, which thus influence the expert's policy, and (b) confounding noise hidden to both the expert and the IL algorithm. We show that causal IL in our framework can be reduced to a set of Conditional Moment Restrictions (CMRs) by leveraging trajectory histories as instruments to learn a history-dependent policy.
  arXiv  Detail & Related papers  (2025-02-11T15:43:49Z)
• Bidirectional Decoding: Improving Action Chunking via Closed-Loop Resampling [51.38330727868982]
  Bidirectional Decoding (BID) is a test-time inference algorithm that bridges action chunking with closed-loop operations. We show that BID boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• Policy Gradient-Driven Noise Mask [3.69758875412828]
  We propose a novel pretraining pipeline that learns to generate conditional noise masks specifically tailored to improve performance on multi-modal and multi-organ datasets. A key aspect is that the policy network's role is limited to obtaining an intermediate (or heated) model before fine-tuning. Results demonstrate that fine-tuning the intermediate models consistently outperforms conventional training algorithms on both classification and generalization to unseen concept tasks.
  arXiv  Detail & Related papers  (2024-04-29T23:53:42Z)
• Towards Principled Representation Learning from Videos for Reinforcement Learning [23.877731515619868]
  We study pre-training representations for decision-making using video data. We focus on learning the latent state representations of the underlying MDP using video data.
  arXiv  Detail & Related papers  (2024-03-20T17:28:17Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Differentially Private Linear Regression with Linked Data [3.9325957466009203]
  Differential privacy, a mathematical notion from computer science, is a rising tool offering robust privacy guarantees. Recent work focuses on developing differentially private versions of individual statistical and machine learning tasks. We present two differentially private algorithms for linear regression with linked data.
  arXiv  Detail & Related papers  (2023-08-01T21:00:19Z)
• Latent Exploration for Reinforcement Learning [87.42776741119653]
  In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. We propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network.
  arXiv  Detail & Related papers  (2023-05-31T17:40:43Z)
• Perlin Noise Improve Adversarial Robustness [9.084544535198509]
  Adversarial examples are some special input that can perturb the output of a deep neural network. Most of the present methods for generating adversarial examples require gradient information. Procedural noise adversarial examples is a new way of adversarial example generation.
  arXiv  Detail & Related papers  (2021-12-26T15:58:28Z)
• Curriculum Offline Imitation Learning [72.1015201041391]
  offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2021-11-03T08:02:48Z)
• Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning [131.1852444489217]
  This paper presents Object-aware REgularizatiOn (OREO), a technique that regularizes an imitation policy in an object-aware manner. Our main idea is to encourage a policy to uniformly attend to all semantic objects, in order to prevent the policy from exploiting nuisance variables strongly correlated with expert actions.
  arXiv  Detail & Related papers  (2021-10-27T01:56:23Z)
• DYNOTEARS: Structure Learning from Time-Series Data [6.7638850283606855]
  We propose a method that simultaneously estimates contemporaneous (intra-slice) and time-lagged (inter-slice) relationships between variables in a time-series. Compared to state-of-the-art methods for learning dynamic Bayesian networks, our method is both scalable and accurate on real data.
  arXiv  Detail & Related papers  (2020-02-02T21:47:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.