When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering

• URL: http://arxiv.org/abs/2602.22474v1
• Date: Wed, 25 Feb 2026 23:23:22 GMT
• Title: When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering
• Authors: Jessie Yuan, Yilin Wu, Andrea Bajcsy,
• Abstract summary: Policy steering is an emerging way to adapt robot behaviors at deployment-time.<n> Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities.<n>We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility.
• Score: 10.01278648231868
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Policy steering is an emerging way to adapt robot behaviors at deployment-time: a learned verifier analyzes low-level action samples proposed by a pre-trained policy (e.g., diffusion policy) and selects only those aligned with the task. While Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities, existing frameworks often assume these models are well-calibrated. In practice, the overconfident judgment from VLM can degrade the steering performance under both high-level semantic uncertainty in task specifications and low-level action uncertainty or incapability of the pre-trained policy. We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility, and selects an uncertainty resolution strategy: execute a high-confidence action, clarify task ambiguity via natural language queries, or ask for action interventions to correct the low-level policy when it is deemed incapable at the task. We leverage conformal prediction to calibrate the composition of the VLM and the pre-trained base policy, providing statistical assurances that the verifier selects the correct strategy. After collecting interventions during deployment, we employ residual learning to improve the capability of the pre-trained policy, enabling the system to learn continually but with minimal expensive human feedback. We demonstrate our framework through experiments in simulation and on hardware, showing that UPS can disentangle confident, ambiguous, and incapable scenarios and minimizes expensive user interventions compared to uncalibrated baselines and prior human- or robot-gated continual learning approaches. Videos can be found at https://jessie-yuan.github.io/ups/

Related papers

• LLMs for High-Frequency Decision-Making: Normalized Action Reward-Guided Consistency Policy Optimization [12.894668119938663]
  Large Language Models (LLMs) form the cornerstone of sequential decision-making agent development.<n>This paper proposes Normalized Action Reward guided Consistency Policy Optimization.<n> Experiments on UAV pursuit, a typical high-frequency task, show our method delivers superior performance on independent and composite tasks.
  arXiv  Detail & Related papers  (2026-03-03T07:22:14Z)
• VLS: Steering Pretrained Robot Policies via Vision-Language Models [31.189909515514668]
  Vision-Language Steering (VLS) is a training-free framework for inference-time adaptation of frozen generative robot policies.<n>VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy.
  arXiv  Detail & Related papers  (2026-02-03T19:50:16Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• Active Test-time Vision-Language Navigation [60.69722522420299]
  ATENA is a test-time active learning framework that enables a practical human-robot interaction via episodic feedback on uncertain navigation outcomes.<n>In particular, ATENA learns to increase certainty in successful episodes and decrease it in failed ones, improving uncertainty calibration.<n>In addition, we propose a self-active learning strategy that enables an agent to evaluate its navigation outcomes based on confident predictions.
  arXiv  Detail & Related papers  (2025-06-07T02:24:44Z)
• Learning Verifiable Control Policies Using Relaxed Verification [49.81690518952909]
  This work proposes to perform verification throughout training to aim for policies whose properties can be evaluated throughout runtime.<n>The approach is to use differentiable reachability analysis and incorporate new components into the loss function.
  arXiv  Detail & Related papers  (2025-04-23T16:54:35Z)
• How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation [17.638831964639834]
  Behavior cloning policies are increasingly successful at solving complex tasks by learning from human demonstrations. We present a framework that provides a tight lower-bound on robot performance in an arbitrary environment. In experiments we evaluate policies for visuomotor manipulation in both simulation and hardware.
  arXiv  Detail & Related papers  (2024-05-08T22:00:35Z)
• PARTNR: Pick and place Ambiguity Resolving by Trustworthy iNteractive leaRning [5.046831208137847]
  We present the PARTNR algorithm that can detect ambiguities in the trained policy by analyzing multiple modalities in the pick and place poses. PARTNR employs an adaptive, sensitivity-based, gating function that decides if additional user demonstrations are required. We demonstrate the performance of PARTNR in a table-top pick and place task.
  arXiv  Detail & Related papers  (2022-11-15T17:07:40Z)
• Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems [18.546197100318693]
  We introduce a scalable framework for supporting fine-grained exploration targets for individual domains via user-defined constraints. We present a novel meta-gradient learning approach that is scalable and practical to address this problem. We conduct extensive experiments using data from a real-world conversational AI on a set of realistic constraint benchmarks.
  arXiv  Detail & Related papers  (2022-09-17T23:44:13Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning [75.56839075060819]
  Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline.
  arXiv  Detail & Related papers  (2020-05-21T19:47:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.