Safety by Design: Realized-Cost Constraints for Contextual Bandits with Continuous Actions

• URL: http://arxiv.org/abs/2608.26755v1
• Date: Thu, 27 Aug 2026 07:52:00 GMT
• Title: Safety by Design: Realized-Cost Constraints for Contextual Bandits with Continuous Actions
• Abstract summary: We study contextual bandits with one-dimensional continuous actions and stage-wise high-probability constraints on the realized cost.<n>We propose High-Probability Constrained UCB, an optimistic-pessimistic algorithm that explores for reward while conservatively estimating the safe action set.
• Score: 25.377745631929557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contextual bandits are a standard framework for sequential decision-making under uncertainty, with applications in clinical trials, dosage selection, recommendation systems, and autonomous systems. Safety is central in many of these applications, since a single unsafe decision in settings such as dosage selection or autonomous driving can have catastrophic consequences. A common way to model safety in bandit problems is to associate each action with both a reward signal and a cost signal, and to optimize reward subject to constraints on cost. Most existing safety-constrained bandit models enforce safety by requiring the expected cost of each action to remain below a prescribed threshold. However, this may be insufficient in heteroscedastic settings, where the chosen action affects not only the expected reward and cost, but also the variability of the observed outcomes. We study contextual bandits with one-dimensional continuous actions and stage-wise high-probability constraints on the realized cost. We propose High-Probability Constrained UCB, an optimistic-pessimistic algorithm that explores for reward while conservatively estimating the safe action set. For linear reward and cost models, we prove a tight $\tilde{\mathcal{O}}(d\sqrt{T})$ regret bound, and we extend the analysis to general function classes using the eluder dimension. Experiments show that enforcing realized-cost safety substantially reduces violations compared with expected-cost constrained baselines.

Related papers

• Robust Peak-cost Constrained Reinforcement Learning [10.897248153175935]
  We study robust peak-cost constrained reinforcement learning (RP-CRL)<n>The objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory.<n>We show that, unlike standard CMDPs, peak-cost constrained MDPs may not admit zero duality gap.
  arXiv  Detail & Related papers  (2026-07-16T21:00:50Z)
• Selective Safety Steering via Value-Filtered Decoding [54.87935112120107]
  Large language models (LLMs) are trained to align with human values, but their generations may still violate safety constraints.<n>Existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model.<n>We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses.
  arXiv  Detail & Related papers  (2026-05-14T12:13:08Z)
• Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptation [50.23374353859762]
  "Maximizing coverage" by collecting diverse, single-shot demonstrations can be self-defeating due to non-vanishing estimation noise.<n>We formalize this phenomenon as a Coverage--Density Trade-off.<n>We propose Anchor-Centric Adaptation (ACA), a two-stage framework that first stabilizes a policy skeleton through repeated demonstrations at core anchors, then expands coverage to high-risk boundaries via teacher-forced error mining and constrained residual updates.
  arXiv  Detail & Related papers  (2026-05-08T07:35:24Z)
• Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning [13.674504740468159]
  We show how a safety reachability set enforces safety constraints without unstable min/max or Lagrangian optimization.<n>We also show how this set enforces safety constraints without unstable min/max or Lagrangian optimization.<n>Our method matches or outperforms state of the art baselines while maintaining safety.
  arXiv  Detail & Related papers  (2026-03-08T11:09:33Z)
• A Simple Reduction Scheme for Constrained Contextual Bandits with Adversarial Contexts via Regression [7.798233121583888]
  We study constrained contextual bandits with adversarially chosen contexts, where each action yields a random reward and incurs a random cost.<n>We adopt the standard realizability assumption: conditioned on the observed context, rewards and costs are drawn independently from fixed distributions whose expectations belong to known function classes.
  arXiv  Detail & Related papers  (2026-02-04T20:19:55Z)
• Conformal Thinking: Risk Control for Reasoning on a Compute Budget [60.65072883773352]
  Reasoning Large Language Models (LLMs) enable test-time scaling, with dataset-level accuracy improving as the token budget increases.<n>We re-frame the budget setting problem as risk control, limiting the error rate while minimizing compute.<n>Our framework introduces an upper threshold that stops reasoning when the model is confident and a novel lower threshold that preemptively stops unsolvable instances.
  arXiv  Detail & Related papers  (2026-02-03T18:17:22Z)
• Boundary-to-Region Supervision for Offline Safe Reinforcement Learning [56.150983204962735]
  Boundary-to-Region (B2R) is a framework that enables asymmetric conditioning through cost signal realignment.<n>B2R redefines CTG as a boundary constraint under a fixed safety budget, unifying the cost distribution of all feasible trajectories.<n> Experimental results show that B2R satisfies safety constraints in 35 out of 38 safety-critical tasks.
  arXiv  Detail & Related papers  (2025-09-30T03:38:20Z)
• ConstrainedZero: Chance-Constrained POMDP Planning using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints [34.9739641898452]
  This work introduces the ConstrainedZero policy algorithm that solves CC-POMDPs in belief space by learning neural network approximations of the optimal value and policy. Results show that by separating safety constraints from the objective we can achieve a target level of safety without optimizing the balance between rewards and costs.
  arXiv  Detail & Related papers  (2024-05-01T17:17:22Z)
• SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning [64.33956692265419]
  offline safe RL is of great practical relevance for deploying agents in real-world applications. We present a novel offline safe RL approach referred to as SaFormer.
  arXiv  Detail & Related papers  (2023-01-28T13:57:01Z)
• Safe Linear Bandits over Unknown Polytopes [39.177982674455784]
  The safe linear bandit problem (SLB) is an online approach to linear programming with unknown objective and unknown roundwise constraints. We study the tradeoffs between efficacy and smooth safety costs of SLBs over polytopes.
  arXiv  Detail & Related papers  (2022-09-27T21:13:32Z)
• A Unifying Framework for Online Optimization with Long-Term Constraints [62.35194099438855]
  We study online learning problems in which a decision maker has to take a sequence of decisions subject to $m$ long-term constraints. The goal is to maximize their total reward, while at the same time achieving small cumulative violation across the $T$ rounds. We present the first best-of-both-world type algorithm for this general class problems, with no-regret guarantees both in the case in which rewards and constraints are selected according to an unknown model, and in the case in which they are selected at each round by an adversary.
  arXiv  Detail & Related papers  (2022-09-15T16:59:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.