Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets

• URL: http://arxiv.org/abs/2510.01479v1
• Date: Wed, 01 Oct 2025 21:43:04 GMT
• Title: Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets
• Authors: Shriram Karpoora Sundara Pandian, Ali Baheri,
• Abstract summary: This paper introduces Density-Ratio Weighted Behavioral Cloning (Weighted BC)<n>Weighted BC is a robust imitation learning approach that uses a small, verified clean reference set to estimate trajectory-level density ratios via a binary discriminator.<n> Experiments demonstrate that Weighted BC maintains near-optimal performance even at high contamination ratios.
• Score: 2.922743999325622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline reinforcement learning (RL) enables policy optimization from fixed datasets, making it suitable for safety-critical applications where online exploration is infeasible. However, these datasets are often contaminated by adversarial poisoning, system errors, or low-quality samples, leading to degraded policy performance in standard behavioral cloning (BC) and offline RL methods. This paper introduces Density-Ratio Weighted Behavioral Cloning (Weighted BC), a robust imitation learning approach that uses a small, verified clean reference set to estimate trajectory-level density ratios via a binary discriminator. These ratios are clipped and used as weights in the BC objective to prioritize clean expert behavior while down-weighting or discarding corrupted data, without requiring knowledge of the contamination mechanism. We establish theoretical guarantees showing convergence to the clean expert policy with finite-sample bounds that are independent of the contamination rate. A comprehensive evaluation framework is established, which incorporates various poisoning protocols (reward, state, transition, and action) on continuous control benchmarks. Experiments demonstrate that Weighted BC maintains near-optimal performance even at high contamination ratios outperforming baselines such as traditional BC, batch-constrained Q-learning (BCQ) and behavior regularized actor-critic (BRAC).

Related papers

• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Robust Conformal Outlier Detection under Contaminated Reference Data [20.864605211132663]
  Conformal prediction is a flexible framework for calibrating machine learning predictions.<n>In outlier detection, this calibration relies on a reference set of labeled inlier data to control the type-I error rate.<n>This paper analyzes the impact of contamination on the validity of conformal methods.
  arXiv  Detail & Related papers  (2025-02-07T10:23:25Z)
• How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence [23.019102917957152]
  Kernel Divergence Score (KDS) is a novel method that evaluates dataset contamination by computing the divergence between the kernel similarity matrix of sample embeddings.<n>KDS demonstrates a near-perfect correlation with contamination levels and outperforms existing baselines.
  arXiv  Detail & Related papers  (2025-02-02T05:50:39Z)
• Strategically Conservative Q-Learning [89.17906766703763]
  offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions. We propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate.
  arXiv  Detail & Related papers  (2024-06-06T22:09:46Z)
• Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization [59.038366742773164]
  Linears and leaky ReLU trained by gradient flow on logistic loss have an implicit bias towards satisfying the Karush-KuTucker (KKT) conditions. In this work we establish a number of settings where the satisfaction of these conditions implies benign overfitting in linear classifiers and in two-layer leaky ReLU networks.
  arXiv  Detail & Related papers  (2023-03-02T18:24:26Z)
• Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation [38.95482624075353]
  We introduce gradient penalty over the learned value function to tackle the exploding Q-functions. We then relax the closeness constraints towards non-optimal actions with critic weighted constraint relaxation. Experimental results show that the proposed techniques effectively tame the non-optimal trajectories for policy constraint offline RL methods.
  arXiv  Detail & Related papers  (2022-10-19T11:22:36Z)
• ConserWeightive Behavioral Cloning for Reliable Offline Reinforcement Learning [27.322942155582687]
  The goal of offline reinforcement learning (RL) is to learn near-optimal policies from static logged datasets, thus sidestepping expensive online interactions. Behavioral cloning (BC) provides a straightforward solution to offline RL by mimicking offline trajectories via supervised learning. We propose ConserWeightive Behavioral Cloning (CWBC) to improve the performance of conditional BC for offline RL.
  arXiv  Detail & Related papers  (2022-10-11T05:37:22Z)
• Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection [81.07346419422605]
  Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
  arXiv  Detail & Related papers  (2022-07-24T18:49:26Z)
• Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning [125.8224674893018]
  Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Applying off-policy algorithms to offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. We propose Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven offline algorithm without explicit policy constraints.
  arXiv  Detail & Related papers  (2022-02-23T15:27:16Z)
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning [14.432131909590824]
  Offline Reinforcement Learning aims to train effective policies using previously collected datasets. Standard off-policy RL algorithms are prone to overestimations of the values of out-of-distribution (less explored) actions. We improve the behavior regularized offline reinforcement learning and propose BRAC+.
  arXiv  Detail & Related papers  (2021-10-02T23:55:49Z)
• Continuous Doubly Constrained Batch Reinforcement Learning [93.23842221189658]
  We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates.
  arXiv  Detail & Related papers  (2021-02-18T08:54:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.