Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation

• URL: http://arxiv.org/abs/2301.11374v2
• Date: Fri, 26 May 2023 21:40:01 GMT
• Title: Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation
• Abstract summary: We present a reinforcement learning framework in which the learned policy comes with a machine-checkable certificate of provable adversarial robustness. We experimentally evaluate CAROL on four MuJoCo environments with continuous state and action spaces. CAROL learns policies that, when contrasted with policies from the state-of-the-art robust RL algorithms, exhibit: (i) markedly enhanced certified performance lower bounds; and (ii) comparable performance under empirical adversarial attacks.
• Score: 10.69970450827617
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a reinforcement learning (RL) framework in which the learned policy comes with a machine-checkable certificate of provable adversarial robustness. Our approach, called CAROL, learns a model of the environment. In each learning iteration, it uses the current version of this model and an external abstract interpreter to construct a differentiable signal for provable robustness. This signal is used to guide learning, and the abstract interpretation used to construct it directly leads to the robustness certificate returned at convergence. We give a theoretical analysis that bounds the worst-case accumulative reward of CAROL. We also experimentally evaluate CAROL on four MuJoCo environments with continuous state and action spaces. On these tasks, CAROL learns policies that, when contrasted with policies from the state-of-the-art robust RL algorithms, exhibit: (i) markedly enhanced certified performance lower bounds; and (ii) comparable performance under empirical adversarial attacks.

Related papers

• In-Context Learning as Implicit Policy Gradient [47.12608115550359]
  Large language models can iteratively improve their outputs by incorporating generated samples and their corresponding evaluation scores as in-context examples.<n>We show that score-conditioned In-Context Learning (ICL) admits a structural correspondence to policy gradient optimization.
  arXiv  Detail & Related papers  (2026-07-25T11:13:52Z)
• Learning to Watch: Active Video Anomaly Understanding via Interleaved Policy Optimization [56.150184676745205]
  Video anomaly understanding (VAU) relies on sparse, context-dependent cues.<n>We propose $Anomtext-$, a closed-loop framework that reconceptualizes video understanding as an active sequential decision-making process.<n>Inspired by human video-reviewing behavior, this framework unifies internal cognitive reasoning and strategic evidence acquisition into an interleaved policy.
  arXiv  Detail & Related papers  (2026-07-01T08:41:49Z)
• Post-Hoc Robustness for Model-Based Reinforcement Learning [1.4072254177584387]
  This work introduces post-hoc robustification of deep RL agents at inference time.<n>The goal is to improve robustness without any additional training of neural networks.
  arXiv  Detail & Related papers  (2026-06-02T11:43:13Z)
• Coupled Variational Reinforcement Learning for Language Model General Reasoning [83.82392089177841]
  We propose textitbCoupled bVari bReinforcement bLearning (CoVRL) to bridge variational inference and reinforcement learning.<n>CoVRL improves performance by 12.4% over the base model and achieves an additional 2.3% improvement over strong state-of-the-art verifier-free RL baselines.
  arXiv  Detail & Related papers  (2025-12-14T07:03:51Z)
• Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment [29.74044158672979]
  Chain-of-Thought (CoT) reasoning techniques can be applied to building steerable pluralistic models.<n>We explore several methods, including CoT prompting, fine-tuning on human-authored CoT, fine-tuning on synthetic explanations, and Reinforcement Learning with Verifiable Rewards.
  arXiv  Detail & Related papers  (2025-10-05T05:39:50Z)
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning [87.7836502955847]
  We propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning.<n>Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood.<n>We introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy.
  arXiv  Detail & Related papers  (2025-06-10T12:40:39Z)
• Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws [52.10468229008941]
  This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting.<n>We provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model.<n>Building on these insights, we introduce a novel method for Contrastive Language-Image Pretraining with a reference model, termed DRRho-CLIP.
  arXiv  Detail & Related papers  (2025-05-10T16:55:03Z)
• SAMBO-RL: Shifts-aware Model-based Offline Reinforcement Learning [9.88109749688605]
  Model-based Offline Reinforcement Learning trains policies based on offline datasets and model dynamics. This paper disentangles the problem into two key components: model bias and policy shift. We introduce Shifts-aware Model-based Offline Reinforcement Learning (SAMBO-RL)
  arXiv  Detail & Related papers  (2024-08-23T04:25:09Z)
• iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning [24.684363928059113]
  We propose an efficient representation learning method using only a self-supervised latent-state consistency loss. We achieve high performance and prevent representation collapse by quantizing the latent representation. Our method, named iQRL: implicitly Quantized Reinforcement Learning, is straightforward, compatible with any model-free RL algorithm.
  arXiv  Detail & Related papers  (2024-06-04T18:15:44Z)
• READ: Improving Relation Extraction from an ADversarial Perspective [33.44949503459933]
  We propose an adversarial training method specifically designed for relation extraction (RE) Our approach introduces both sequence- and token-level perturbations to the sample and uses a separate perturbation vocabulary to improve the search for entity and context perturbations.
  arXiv  Detail & Related papers  (2024-04-02T16:42:44Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning [26.05184273238923]
  This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL) We devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating "conservatism" into a learned reward function. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy.
  arXiv  Detail & Related papers  (2023-02-09T17:16:29Z)
• Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning [92.18524491615548]
  Contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL) We study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. Under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
  arXiv  Detail & Related papers  (2022-07-29T17:29:08Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning [55.048010996144036]
  We show that under some noise assumption, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free. We propose Spectral Dynamics Embedding (SPEDE), which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise.
  arXiv  Detail & Related papers  (2021-11-22T19:24:57Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary [86.0846119254031]
  We study the robustness of reinforcement learning with adversarially perturbed state observations. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
  arXiv  Detail & Related papers  (2021-01-21T05:38:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.