Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signals

• URL: http://arxiv.org/abs/2510.14503v1
• Date: Thu, 16 Oct 2025 09:48:54 GMT
• Title: Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signals
• Abstract summary: This paper proposes a reversible learning framework to improve the robustness and efficiency of value based Reinforcement Learning agents.<n>The framework has two complementary core mechanisms: an empirically derived transition reversibility measure called Phi of s and a interrupts, and a selective state rollback operation.
• Score: 1.9537983097153042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper proposes a reversible learning framework to improve the robustness and efficiency of value based Reinforcement Learning agents, addressing vulnerability to value overestimation and instability in partially irreversible environments. The framework has two complementary core mechanisms: an empirically derived transition reversibility measure called Phi of s and a, and a selective state rollback operation. We introduce an online per state action estimator called Phi that quantifies the likelihood of returning to a prior state within a fixed horizon K. This measure is used to adjust the penalty term during temporal difference updates dynamically, integrating reversibility awareness directly into the value function. The system also includes a selective rollback operator. When an action yields an expected return markedly lower than its instantaneous estimated value and violates a predefined threshold, the agent is penalized and returns to the preceding state rather than progressing. This interrupts sub optimal high risk trajectories and avoids catastrophic steps. By combining reversibility aware evaluation with targeted rollback, the method improves safety, performance, and stability. In the CliffWalking v0 domain, the framework reduced catastrophic falls by over 99.8 percent and yielded a 55 percent increase in mean episode return. In the Taxi v3 domain, it suppressed illegal actions by greater than or equal to 99.9 percent and achieved a 65.7 percent improvement in cumulative reward, while also sharply reducing reward variance in both environments. Ablation studies confirm that the rollback mechanism is the critical component underlying these safety and performance gains, marking a robust step toward safe and reliable sequential decision making.

Related papers

• Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning [7.373349445059511]
  Risk-aware Q-learning (RaQL) provides a model-free, two-timescale estimator for dynamic risk objectives.<n>This paper proposes an adaptive training controller for Conditional Value-at-Risk (CVaR) RaQL and evaluates it on a daily Bitcoin trading task.
  arXiv  Detail & Related papers  (2026-08-05T00:26:22Z)
• Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms [21.868021164923515]
  We study an inference-time strategy that uses uncertainty signals to identify unreliable generation regions and roll back to earlier valid prefixes without retraining the model.<n>Our results show that the complete rollback framework improves over equal-budget restart across the evaluated benchmarks and model settings.
  arXiv  Detail & Related papers  (2026-07-31T05:26:42Z)
• Safe Remediation as Risk-Constrained Intervention Decision in Microservice Systems [2.1686718805804976]
  In modern IT operations (IT-Ops), the cost of an incorrect repair often exceeds the cost of no action at all.<n>This paper makes three contributions to close this gap: (i) we reformulate safe remediation as a risk-constrained intervention decision problem; (ii) we introduce a three-dimensional risk decomposition; and (iii) we design a context-adaptive human-in-the-loop (HITL) gate that turns from a binary failsafe into a bandwidth-aware control layer.
  arXiv  Detail & Related papers  (2026-07-22T10:43:14Z)
• Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning [81.57028614960576]
  We show that the Confidence Region possesses two critical properties: 1) High Reliability -- answers in the confidence region become highly accurate and stable, and 2) High Redundancy -- models generate unnecessary tokens long after reaching the correct answer.<n>These properties unlock more efficient and reliable inference strategies.
  arXiv  Detail & Related papers  (2026-06-01T10:11:14Z)
• Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation [13.151825012034886]
  We propose a framework that trains risk-sensitive policies through Conditional Value-at-Risk (CVaR) constrained optimization on an off-policy TD3 backbone.<n>During training, the policy is optimized under CVaR constraints on cumulative costs, promoting sensitivity to high-cost tail outcomes.<n>A key finding is that policies trained with CVaR constraints maintain larger safety margins from obstacles across evaluated states.
  arXiv  Detail & Related papers  (2026-05-13T22:53:47Z)
• On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment [54.30690671490447]
  Existing safety-alignment signals are largely response-level or off-policy.<n>FATE transforms verifier-scored failures into repair supervision without expert demonstrations.<n>FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%.
  arXiv  Detail & Related papers  (2026-05-12T09:56:28Z)
• Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization [51.11398557420066]
  We introduce Metis, a framework that reformulates jailbreaking as inference-time policy optimization.<n>Metis achieves the strongest average Attack Success Rate (ASR) among compared methods at 89.2%.<n>By replacing redundant exploration with directed optimization, Metis reduces token costs by an average of 8.2x and up to 11.4x.
  arXiv  Detail & Related papers  (2026-05-11T06:45:00Z)
• Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking [0.0]
  Real-world objectives are often uncertain, context-dependent, and internally inconsistent.<n>This mismatch can lead to alignment failures such as reward hacking, over-optimization, and overconfident behavior.<n>We introduce a dual-source uncertainty-aware reward framework that explicitly models both uncertainty in value estimation and uncertainty in human preferences.
  arXiv  Detail & Related papers  (2026-04-29T07:14:01Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Uncertainty-Aware Jamming Mitigation with Active RIS: A Robust Stackelberg Game Approach [65.06640919319413]
  This paper investigates the jamming mitigation by leveraging an active reconfigurable intelligent surface (ARIS)<n>We adopt the Stackelberg game formulation to model the strategic interaction between the legitimate side and the adversary.<n>We first derive the optimal jamming policy as the follower's best response, which is then incorporated into the legitimate-side optimization for robust anti-jamming design.
  arXiv  Detail & Related papers  (2026-02-20T12:02:01Z)
• Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity [16.835098688159004]
  Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastrophic events.<n>We develop risk-averse value and model-free Q-learning algorithms that rely on discretized augmented states.<n> Empirical results demonstrate that our algorithms successfully learn CVaR-sensitive policies and achieve effective performance-safety trade-offs.
  arXiv  Detail & Related papers  (2026-02-03T17:39:45Z)
• Illuminating the Black Box: Real-Time Monitoring of Backdoor Unlearning in CNNs via Explainable AI [0.0]
  Backdoor attacks pose severe security threats to deep neural networks by embedding malicious triggers that force misclassification.<n>This paper introduces a novel framework that integrates Gradient-weighted Class Activation Mapping (Grad-CAM) into the unlearning process to provide real-time monitoring and explainability.<n> Experiments on CIFAR-10 with BadNets attacks demonstrate that our approach reduces Attack Success Rate (ASR) from 96.51% to 5.52% while retaining 99.48% of clean accuracy (82.06%)
  arXiv  Detail & Related papers  (2025-11-26T11:29:11Z)
• SafeEvalAgent: Toward Agentic and Self-Evolving Safety Evaluation of LLMs [37.82193156438782]
  This paper introduces a new paradigm of agentic safety evaluation, reframing evaluation as a continuous and self-evolving process.<n>We propose a novel multi-agent framework SafeEvalAgent, which autonomously ingests unstructured policy documents to generate and perpetually evolve a comprehensive safety benchmark.<n>Our experiments demonstrate the effectiveness of SafeEvalAgent, showing a consistent decline in model safety as the evaluation hardens.
  arXiv  Detail & Related papers  (2025-09-30T11:20:41Z)
• Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection [18.467741067831877]
  We introduce Progressive Self-Reflection, a novel inference-time technique that empowers large language models to self-monitor and correct their outputs dynamically.<n> Experimental results demonstrate that applying our proposed method to Llama-3.1-8B-Instruct reduces the attack success rate from 77.5% to 5.9%.<n>Our approach acts as a test-time scaling method, where additional self-reflection rounds enhance safety at the cost of inference overhead.
  arXiv  Detail & Related papers  (2025-09-29T12:54:28Z)
• Aurora: Are Android Malware Classifiers Reliable and Stable under Distribution Shift? [51.12297424766236]
  AURORA is a framework to evaluate malware classifiers based on their confidence quality and operational resilience.<n>AURORA is complemented by a set of metrics designed to go beyond point-in-time performance.<n>The fragility in SOTA frameworks across datasets of varying drift suggests the need for a return to the whiteboard.
  arXiv  Detail & Related papers  (2025-05-28T20:22:43Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk [30.229387511344456]
  We propose a novel reinforcement learning algorithm of CVaR-Proximal-Policy-Optimization (CPPO) which formalizes the risk-sensitive constrained optimization problem by keeping its CVaR under a given threshold. Experimental results show that CPPO achieves a higher cumulative reward and is more robust against both observation and transition disturbances.
  arXiv  Detail & Related papers  (2022-06-09T11:57:54Z)
• Smooth Sequential Optimisation with Delayed Feedback [0.0]
  We propose a novel adaptation to shrinkage that estimates smoothed reward estimates from cumulative inputs. Using numerical simulations, we show that this adaptation retains the benefits of shrinkage, and improves the stability of reward estimation by more than 50%.
  arXiv  Detail & Related papers  (2021-06-21T17:51:36Z)
• Corruption-robust exploration in episodic reinforcement learning [76.19192549843727]
  We study multi-stage episodic reinforcement learning under adversarial corruptions in both the rewards and the transition probabilities of the underlying system. Our framework yields efficient algorithms which attain near-optimal regret in the absence of corruptions. Notably, our work provides the first sublinear regret guarantee which any deviation from purely i.i.d. transitions in the bandit-feedback model for episodic reinforcement learning.
  arXiv  Detail & Related papers  (2019-11-20T03:49:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.