Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization

• URL: http://arxiv.org/abs/2511.17568v1
• Date: Fri, 14 Nov 2025 06:11:13 GMT
• Title: Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
• Authors: Le Xu, Jiayu Chen,
• Abstract summary: offline reinforcement learning vulnerable to real-world data corruption.<n>We are first to apply Sharpness-Aware Minimization (SAM) as a general-purpose plug-and-play for offline RL.<n>We integrate SAM into strong baselines for data corruption: IQL, a top-performing offline RL algorithm, and RIQL, an algorithm designed specifically for data-corruption robustness.
• Score: 9.524029391786557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline reinforcement learning (RL) is vulnerable to real-world data corruption, with even robust algorithms failing under challenging observation and mixture corruptions. We posit this failure stems from data corruption creating sharp minima in the loss landscape, leading to poor generalization. To address this, we are the first to apply Sharpness-Aware Minimization (SAM) as a general-purpose, plug-and-play optimizer for offline RL. SAM seeks flatter minima, guiding models to more robust parameter regions. We integrate SAM into strong baselines for data corruption: IQL, a top-performing offline RL algorithm in this setting, and RIQL, an algorithm designed specifically for data-corruption robustness. We evaluate them on D4RL benchmarks with both random and adversarial corruption. Our SAM-enhanced methods consistently and significantly outperform the original baselines. Visualizations of the reward surface confirm that SAM finds smoother solutions, providing strong evidence for its effectiveness in improving the robustness of offline RL agents.

Related papers

• Towards Sample-Efficient and Stable Reinforcement Learning for LLM-based Recommendation [56.92367609590823]
  Long Chain-of-Thought (Long CoT) reasoning has shown promise in Large Language Models (LLMs)<n>We argue that Long CoT is inherently ill-suited for the sequential recommendation domain.<n>We propose RISER, a novel Reinforced Item Space Exploration framework for Recommendation.
  arXiv  Detail & Related papers  (2026-01-31T10:02:43Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Sparse Offline Reinforcement Learning with Corruption Robustness [24.193236728009918]
  We investigate robustness to strong data corruption in offline sparse reinforcement learning (RL)<n>In our setting, an adversary may arbitrarily perturb a fraction of the collected trajectories from a high-dimensional but sparse Markov decision process.<n>Our results provide the first non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage and corruption.
  arXiv  Detail & Related papers  (2025-12-31T10:28:25Z)
• Distributionally Robust Optimization with Adversarial Data Contamination [49.89480853499918]
  We focus on optimizing Wasserstein-1 DRO objectives for generalized linear models with convex Lipschitz loss functions.<n>Our primary contribution lies in a novel modeling framework that integrates robustness against training data contamination with robustness against distributional shifts.<n>This work establishes the first rigorous guarantees, supported by efficient computation, for learning under the dual challenges of data contamination and distributional shifts.
  arXiv  Detail & Related papers  (2025-07-14T18:34:10Z)
• Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization [61.02719787737867]
  Large language models (LLMs) are increasingly deployed and democratized on edge devices.<n>One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM.<n>We conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings.
  arXiv  Detail & Related papers  (2025-02-06T18:59:11Z)
• Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption [60.958746600254884]
  This study tackles the challenges of adversarial corruption in model-based reinforcement learning (RL) We introduce an algorithm called corruption-robust optimistic MLE (CR-OMLE), which leverages total-variation (TV)-based information ratios as uncertainty weights for MLE. We extend our weighting technique to the offline setting, and propose an algorithm named corruption-robust pessimistic MLE (CR-PMLE)
  arXiv  Detail & Related papers  (2024-02-14T07:27:30Z)
• Corruption Robust Offline Reinforcement Learning with Human Feedback [33.33154679893122]
  We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting. We aim to design algorithms that identify a near-optimal policy from the corrupted data, with provable guarantees.
  arXiv  Detail & Related papers  (2024-02-09T19:09:48Z)
• CR-SAM: Curvature Regularized Sharpness-Aware Minimization [8.248964912483912]
  Sharpness-Aware Minimization (SAM) aims to enhance the generalizability by minimizing worst-case loss using one-step gradient ascent as an approximation. In this paper, we introduce a normalized Hessian trace to accurately measure the curvature of loss landscape on em both training and test sets. In particular, to counter excessive non-linearity of loss landscape, we propose Curvature Regularized SAM (CR-SAM)
  arXiv  Detail & Related papers  (2023-12-21T03:46:29Z)
• Towards Robust Offline Reinforcement Learning under Diverse Data Corruption [46.16052026620402]
  We show that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. We propose a more robust offline RL approach named Robust IQL (RIQL)
  arXiv  Detail & Related papers  (2023-10-19T17:54:39Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.