ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2505.23871v2
• Date: Thu, 05 Jun 2025 03:54:42 GMT
• Title: ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
• Authors: Zeyuan Liu, Zhihe Yang, Jiawei Xu, Rui Yang, Jiafei Lyu, Baoxiang Wang, Yunjian Xu, Xiu Li,
• Abstract summary: Real-world datasets collected from sensors or human inputs are prone to noise and errors.<n>Existing methods have made progress in addressing corrupted actions and rewards, but remain insufficient for handling corruption in high-dimensional state spaces.<n>We propose Ambient Diffusion-Guided dataset Recovery (ADG), a novel approach that pioneers the use of diffusion models to tackle data corruption in offline RL.
• Score: 31.52561546227886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While existing methods have made progress in addressing corrupted actions and rewards, they remain insufficient for handling corruption in high-dimensional state spaces and for cases where multiple elements in the dataset are corrupted simultaneously. Diffusion models, known for their strong denoising capabilities, offer a promising direction for this problem-but their tendency to overfit noisy samples limits their direct applicability. To overcome this, we propose Ambient Diffusion-Guided Dataset Recovery (ADG), a novel approach that pioneers the use of diffusion models to tackle data corruption in offline RL. First, we introduce Ambient Denoising Diffusion Probabilistic Models (DDPM) from approximated distributions, which enable learning on partially corrupted datasets with theoretical guarantees. Second, we use the noise-prediction property of Ambient DDPM to distinguish between clean and corrupted data, and then use the clean subset to train a standard DDPM. Third, we employ the trained standard DDPM to refine the previously identified corrupted data, enhancing data quality for subsequent offline RL training. A notable strength of ADG is its versatility-it can be seamlessly integrated with any offline RL algorithm. Experiments on a range of benchmarks, including MuJoCo, Kitchen, and Adroit, demonstrate that ADG effectively mitigates the impact of corrupted data and improves the robustness of offline RL under various noise settings, achieving state-of-the-art results.

Related papers

• What is Adversarial Training for Diffusion Models? [4.71482540145286]
  We show that adversarial training (AT) for diffusion models (DMs) fundamentally differs from classifiers.<n>AT is a way to enforce smoothness in the diffusion flow, improving to outliers and corrupted data.<n>We rigorously evaluate our approach with proof-of-concept datasets with known distributions in low- and high-dimensional space.
  arXiv  Detail & Related papers  (2025-05-27T20:32:28Z)
• Restoration Score Distillation: From Corrupted Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We propose textitRestoration Score Distillation (RSD), a principled generalization of Denoising Score Distillation (DSD)<n>RSD accommodates a broader range of corruption types, such as blurred, incomplete, or low-resolution images.<n>It consistently surpasses its teacher model across diverse restoration tasks on both natural and scientific datasets.
  arXiv  Detail & Related papers  (2025-05-19T17:21:03Z)
• Denoising Score Distillation: From Noisy Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We introduce denoising score distillation (DSD), a surprisingly effective and novel approach for training high-quality generative models from low-quality data.<n>DSD pretrains a diffusion model exclusively on noisy, corrupted samples and then distills it into a one-step generator capable of producing refined, clean outputs.
  arXiv  Detail & Related papers  (2025-03-10T17:44:46Z)
• Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions [8.666879925570331]
  Real-world offline datasets are often subject to data corruptions due to sensor failures or malicious attacks. Existing methods struggle to learn robust agents under high uncertainty caused by corrupted data. We propose a novel robust variational Bayesian inference for offline RL (TRACER)
  arXiv  Detail & Related papers  (2024-11-01T09:28:24Z)
• Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling [35.2859997591196]
  offline reinforcement learning holds promise for scaling data-driven decision-making.<n>However, real-world data collected from sensors or humans often contains noise and errors.<n>Our study reveals that prior research falls short under data corruption when the dataset is limited.
  arXiv  Detail & Related papers  (2024-07-05T06:34:32Z)
• On the Influence of Data Resampling for Deep Learning-Based Log Anomaly Detection: Insights and Recommendations [10.931620604044486]
  This study provides an in-depth analysis of the impact of diverse data resampling methods on existingAD approaches. We assess the performance of theseAD approaches across four datasets with different levels of class imbalance. We evaluate the effectiveness of the data resampling methods when utilizing optimal resampling ratios of normal to abnormal data.
  arXiv  Detail & Related papers  (2024-05-06T14:01:05Z)
• Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data [74.2507346810066]
  Ambient diffusion is a recently proposed framework for training diffusion models using corrupted data. We present the first framework for training diffusion models that provably sample from the uncorrupted distribution given only noisy training data.
  arXiv  Detail & Related papers  (2024-03-20T14:22:12Z)
• Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances [55.37242480995541]
  We propose to denoise noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.
  arXiv  Detail & Related papers  (2023-10-25T17:23:37Z)
• MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection [76.97324120775475]
  Cross-domain keypoint detection methods always require accessing the source data during adaptation. This paper considers source-free domain adaptive keypoint detection, where only the well-trained source model is provided to the target domain.
  arXiv  Detail & Related papers  (2023-02-09T12:06:08Z)
• Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection [67.49587673594276]
  We introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem. We demonstrate that the diffusion denoising process (DDP) of DMs serves as a novel form of asymmetric, which is well-suited to enhance the input and mitigate the overconfidence problem. Our experiments on CIFAR10, CIFAR100, and ImageNet show that our method outperforms SOTA approaches.
  arXiv  Detail & Related papers  (2022-11-21T08:45:08Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.