LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies

• URL: http://arxiv.org/abs/2510.24983v1
• Date: Tue, 28 Oct 2025 21:26:18 GMT
• Title: LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies
• Abstract summary: Diffusion policies are competitive for offline reinforcement learning (RL)<n>We introduce LRT-Diffusion, a risk-aware sampling rule that treats each denoising step as a sequential hypothesis test.<n>This turns guidance from a fixed push into an evidence-driven adjustment with a user-interpretable risk budget.
• Score: 8.211731158152554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion policies are competitive for offline reinforcement learning (RL) but are typically guided at sampling time by heuristics that lack a statistical notion of risk. We introduce LRT-Diffusion, a risk-aware sampling rule that treats each denoising step as a sequential hypothesis test between the unconditional prior and the state-conditional policy head. Concretely, we accumulate a log-likelihood ratio and gate the conditional mean with a logistic controller whose threshold tau is calibrated once under H0 to meet a user-specified Type-I level alpha. This turns guidance from a fixed push into an evidence-driven adjustment with a user-interpretable risk budget. Importantly, we deliberately leave training vanilla (two heads with standard epsilon-prediction) under the structure of DDPM. LRT guidance composes naturally with Q-gradients: critic-gradient updates can be taken at the unconditional mean, at the LRT-gated mean, or a blend, exposing a continuum from exploitation to conservatism. We standardize states and actions consistently at train and test time and report a state-conditional out-of-distribution (OOD) metric alongside return. On D4RL MuJoCo tasks, LRT-Diffusion improves the return-OOD trade-off over strong Q-guided baselines in our implementation while honoring the desired alpha. Theoretically, we establish level-alpha calibration, concise stability bounds, and a return comparison showing when LRT surpasses Q-guidance-especially when off-support errors dominate. Overall, LRT-Diffusion is a drop-in, inference-time method that adds principled, calibrated risk control to diffusion policies for offline RL.

Related papers

• Knowing When to Ask for Help: Bayesian Self-Escalation in Hierarchical LLM Agents [0.0]
  Current agent systems decide delegation before reasoning begins.<n>We study a third regime: an agent that recognises, during its own reasoning, that it is unlikely to succeed.
  arXiv  Detail & Related papers  (2026-08-25T05:35:07Z)
• Uncertainty-Aware Sequential Decision Rules for Event-Triggered LLM Invocation in Streaming Systems [3.556355987197792]
  Streaming inference pipelines increasingly pair lightweight fast models with Large Language Models (LLMs)<n>We cast this as a risk-based sequential stopping problem, where a trigger policy fires when a risk functional over the observation history exceeds a threshold.<n>We prove six results: a minimum inter-event time bound excluding trigger; optimality of threshold policies via smooth pasting; approximate SPRT guarantees under estimated parameters.
  arXiv  Detail & Related papers  (2026-06-27T08:18:09Z)
• PromptShift-CRC: Drift-Aware Conformal Risk Control for Foundation Models Under Prompt and Domain Shift [0.0]
  Foundation models are now used in settings where the prompts they receive can change quickly.<n>Conformal prediction and conformal risk control give model-agnostic ways to control error, but they work best when the calibration data still look like the future data.<n>This paper develops PromptShift CRC, a drift-aware conformal risk control method for foundation-model outputs under prompt and domain shift.
  arXiv  Detail & Related papers  (2026-06-14T18:45:56Z)
• Rethinking the Divergence Regularization in LLM RL [56.952725733148746]
  Reinforcement learning (RL) has become a key component of post-training large language models (LLMs)<n>We propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift.<n>DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights.
  arXiv  Detail & Related papers  (2026-06-08T17:58:23Z)
• One-Way Policy Optimization for Self-Evolving LLMs [63.8638342097375]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs)<n>We propose One-Way Policy Optimization (OWPO), a method based on the principle of decoupling optimization direction from update magnitude.<n> Experimental results demonstrate that OWPO outperforms strong baselines, including DAPO, OPD, and MOPD.
  arXiv  Detail & Related papers  (2026-05-21T08:25:27Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Fairness Begins with State: Purifying Latent Preferences for Hierarchical Reinforcement Learning in Interactive Recommendation [23.667292152163068]
  We present textbfDSRM-HRL, a framework that reformulates fairness-aware recommendation as a latent state purification problem.<n>We show that DSRM-HRL effectively breaks the "rich-get-richer" feedback loop, achieving a superior frontier between recommendation utility and exposure equity.
  arXiv  Detail & Related papers  (2026-03-04T08:14:21Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Trust but Verify: Adaptive Conditioning for Reference-Based Diffusion Super-Resolution via Implicit Reference Correlation Modeling [42.10910149675583]
  Real-world degradations make correspondences between low-quality (LQ) inputs and reference (Ref) images unreliable.<n>We propose Ada-RefSR, a single-step diffusion framework guided by a "Trust but verify" principle.<n>Experiments on multiple datasets demonstrate that Ada-RefSR achieves a strong balance of fidelity, naturalness, and efficiency.
  arXiv  Detail & Related papers  (2026-02-02T09:34:57Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets [2.922743999325622]
  This paper introduces Density-Ratio Weighted Behavioral Cloning (Weighted BC)<n>Weighted BC is a robust imitation learning approach that uses a small, verified clean reference set to estimate trajectory-level density ratios via a binary discriminator.<n> Experiments demonstrate that Weighted BC maintains near-optimal performance even at high contamination ratios.
  arXiv  Detail & Related papers  (2025-10-01T21:43:04Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning [13.163511229897667]
  In offline reinforcement learning, it is necessary to manage out-of-distribution actions to prevent overestimation of value functions.<n>We propose Diffusion Actor-Critic (DAC) that formulates the Kullback-Leibler (KL) constraint policy iteration as a diffusion noise regression problem.<n>Our approach is evaluated on D4RL benchmarks and outperforms the state-of-the-art in nearly all environments.
  arXiv  Detail & Related papers  (2024-05-31T00:41:04Z)
• Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization [63.93275508300137]
  We introduce a novel risk-aware Counterfactual Learning To Rank method with theoretical guarantees for safe deployment. Our experimental results demonstrate the efficacy of our proposed method, which is effective at avoiding initial periods of bad performance when little data is available.
  arXiv  Detail & Related papers  (2023-04-26T15:54:23Z)
• Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories [88.08381083207449]
  We show the prevalence of emphgeneralization failure on controllable states from stranger agents. We propose a novel method called Self-Trajectory Augmentation (STA), which will reset the environment to the agent's old states according to the Q function during training.
  arXiv  Detail & Related papers  (2023-04-26T10:12:12Z)
• Policy Evaluation in Distributional LQR [70.63903506291383]
  We provide a closed-form expression of the distribution of the random return. We show that this distribution can be approximated by a finite number of random variables. Using the approximate return distribution, we propose a zeroth-order policy gradient algorithm for risk-averse LQR.
  arXiv  Detail & Related papers  (2023-03-23T20:27:40Z)
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning [14.432131909590824]
  Offline Reinforcement Learning aims to train effective policies using previously collected datasets. Standard off-policy RL algorithms are prone to overestimations of the values of out-of-distribution (less explored) actions. We improve the behavior regularized offline reinforcement learning and propose BRAC+.
  arXiv  Detail & Related papers  (2021-10-02T23:55:49Z)
• Detecting Rewards Deterioration in Episodic Reinforcement Learning [63.49923393311052]
  In many RL applications, once training ends, it is vital to detect any deterioration in the agent performance as soon as possible. We consider an episodic framework, where the rewards within each episode are not independent, nor identically-distributed, nor Markov. We define the mean-shift in a way corresponding to deterioration of a temporal signal (such as the rewards), and derive a test for this problem with optimal statistical power.
  arXiv  Detail & Related papers  (2020-10-22T12:45:55Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.