Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions

• URL: http://arxiv.org/abs/2411.00465v1
• Date: Fri, 01 Nov 2024 09:28:24 GMT
• Title: Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
• Authors: Rui Yang, Jie Wang, Guoping Wu, Bin Li,
• Abstract summary: Real-world offline datasets are often subject to data corruptions due to sensor failures or malicious attacks. Existing methods struggle to learn robust agents under high uncertainty caused by corrupted data. We propose a novel robust variational Bayesian inference for offline RL (TRACER)
• Score: 8.666879925570331
• License:
• Abstract: Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation in clean environments. To tackle this problem, we propose a novel robust variational Bayesian inference for offline RL (TRACER). It introduces Bayesian inference for the first time to capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions as the uncertainty in the action-value function. Then, to capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based on the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demonstrate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.

Related papers

• Pseudo-Probability Unlearning: Towards Efficient and Privacy-Preserving Machine Unlearning [59.29849532966454]
  We propose PseudoProbability Unlearning (PPU), a novel method that enables models to forget data to adhere to privacy-preserving manner. Our method achieves over 20% improvements in forgetting error compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-11-04T21:27:06Z)
• Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation [49.53202761595912]
  Continual Test-Time Adaptation involves adapting a pre-trained source model to continually changing unsupervised target domains. We analyze the challenges of this task: online environment, unsupervised nature, and the risks of error accumulation and catastrophic forgetting. We propose an uncertainty-aware buffering approach to identify and aggregate significant samples with high certainty from the unsupervised, single-pass data stream.
  arXiv  Detail & Related papers  (2024-07-12T15:48:40Z)
• Robust Decision Transformer: Tackling Data Corruption in Offline RL via Sequence Modeling [34.547551367941246]
  Real-world data collected from sensors or humans often contains noise and errors. Traditional offline RL methods based on temporal difference learning tend to underperform Decision Transformer (DT) under data corruption. We propose Robust Decision Transformer (RDT) by incorporating several robust techniques.
  arXiv  Detail & Related papers  (2024-07-05T06:34:32Z)
• Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation [70.36344590967519]
  We show that noisy data and nuisance features can be sufficient to shatter the Accuracy-on-the-line phenomenon. We demonstrate this phenomenon across both synthetic and real datasets with noisy data and nuisance features.
  arXiv  Detail & Related papers  (2024-06-27T09:57:31Z)
• Effective and Robust Adversarial Training against Data and Label Corruptions [35.53386268796071]
  Corruptions due to data perturbations and label noise are prevalent in the datasets from unreliable sources. We develop an Effective and Robust Adversarial Training framework to simultaneously handle two types of corruption.
  arXiv  Detail & Related papers  (2024-05-07T10:53:20Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Causal Deep Reinforcement Learning Using Observational Data [11.790171301328158]
  We propose two deconfounding methods in deep reinforcement learning (DRL) The methods first calculate the importance degree of different samples based on the causal inference technique, and then adjust the impact of different samples on the loss function. We prove the effectiveness of our deconfounding methods and validate them experimentally.
  arXiv  Detail & Related papers  (2022-11-28T14:34:39Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Classification and Uncertainty Quantification of Corrupted Data using Semi-Supervised Autoencoders [11.300365160909879]
  We present a probabilistic approach to classify strongly corrupted data and quantify uncertainty. A semi-supervised autoencoder trained on uncorrupted data is the underlying architecture. We show that the model uncertainty strongly depends on whether the classification is correct or wrong.
  arXiv  Detail & Related papers  (2021-05-27T18:47:55Z)
• Learning Deep Neural Networks under Agnostic Corrupted Supervision [37.441467641123026]
  We present an efficient robust algorithm that achieves strong guarantees without any assumption on the type of corruption. Our algorithm focuses on controlling the collective impact of data points on the average gradient. Experiments on multiple benchmark datasets have demonstrated the robustness of our algorithm under different types of corruption.
  arXiv  Detail & Related papers  (2021-02-12T19:36:04Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.