The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents

• URL: http://arxiv.org/abs/2605.20544v1
• Date: Tue, 19 May 2026 22:32:44 GMT
• Title: The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents
• Abstract summary: Vision-language models (VLMs) are used as planners for embodied agents.<n>We introduce a taxonomy to categorize abstention in the context of embodied robotics.<n>We present RoboAbstention, a framework for generating abstention instructions grounded in images.
• Score: 14.695254264082273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) are used as high-level planners for embodied agents, translating natural language instructions and visual observations into action plans. While prior work has studied abstention in LLMs, existing benchmarks are largely text-only and do not capture the perceptual grounding and physical constraints inherent to embodied robotics environments. In such settings, abstention requires recognizing when instructions are ambiguous, physically infeasible, based on false premises, or otherwise unresolvable given the available sensory modalities and context. To address this gap, we introduce a taxonomy to categorize abstention in the context of embodied robotics and present RoboAbstention, a scalable and auditable framework for generating abstention instructions grounded in images gathered from five robotics datasets. RoboAbstention instantiates the taxonomy through a three-phase pipeline: (1) structured visual grounding, (2) deterministic constraint derivation, and (3) controlled instruction generation via category-specific templates. This enables the construction of a diverse dataset with verifiable abstention conditions. We evaluate several frontier VLMs and find that all models exhibit significant weaknesses in abstention, including those with advanced reasoning capabilities. The best-performing model, Gemini 2.5 Flash, abstains on only 39.0% of our 6,069 benchmark instructions, while the embodied planner Gemini Robotics ER 1.6 Preview abstains on just 16.5%. We further explore methods for improving abstention in VLM planners, such as defensive prompting and in-context learning, and find that these interventions substantially improve performance, reaching 93.6% abstention rate for Gemini Robotics ER 1.6 Preview and 88.6% for GPT 5.4 Mini, yet no approach fully solves the problem. We open-source RoboAbstention at https://purseclab.github.io/RoboAbstention/.

Related papers

• GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation [60.95802802905149]
  We present GIFT (Guided Intermediate Feature Training), an architecture-flexible framework for learning intermediate features.<n>We instantiate GIFT in a Vision-Language-Action (VLA) policy, a direct-action World-Action Model (WAM), and an inverse-dynamics WAM while retaining each model's action formulation.<n>On RoboCasa, the three GIFT variants reach 61.4%, 83.6%, and 82.3%, outperforming their counterparts by 12.6, 9.0, and 8.4 points, respectively.
  arXiv  Detail & Related papers  (2026-09-03T17:59:03Z)
• Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robots [15.891078191181697]
  Vision-Language Models (VLMs) are increasingly deployed as planners in robotic systems.<n> adversarial text placed within the robot's visual field can act as an indirect prompt injection into the VLM's reasoning stack.<n>We present a systematic study of physical prompt injection attacks against VLM-controlled sorting.
  arXiv  Detail & Related papers  (2026-08-06T07:52:55Z)
• RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation [46.68319840674789]
  We present RoboInter1.5, an extended and holistic suite of intermediate representations for both robotic manipulation and embodied world modeling.<n>RoboInter1.5 provides a unified resource of data, benchmarks, and models centered on dense manipulation-oriented intermediate representations.
  arXiv  Detail & Related papers  (2026-07-21T05:05:01Z)
• Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs [46.61554174146852]
  Task-Agnostic Pretraining (TAP) is a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data.<n>On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data.<n>On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%.
  arXiv  Detail & Related papers  (2026-07-02T17:33:37Z)
• GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation [49.16739604572808]
  Vision-Language-Action (VLA) models achieve strong benchmark performance but struggle in real-world deployment with unseen objects.<n>We argue that this stems from the lack of a unified geometry-aware manipulation representation.<n>We propose GEAR-VLA, a VLA framework for learning unified geometry-aware action representations for generalizable robotic manipulation.
  arXiv  Detail & Related papers  (2026-06-07T09:23:16Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• A Semantic Autonomy Framework for VLM-Integrated Indoor Mobile Robots: Hybrid Deterministic Reasoning and Cross-Robot Adaptive Memory [5.2851376150891864]
  This paper presents a six-layer reference framework for semantically autonomous indoor navigation.<n>A seven-step parametric resolver handles 88% of instructions in under 0.1 milliseconds without invoking a language model, camera, or GPU.<n>A five-category semantic memory framework with explicit scope taxonomy enables cross-session learning and cross-robot knowledge transfer.
  arXiv  Detail & Related papers  (2026-05-04T12:27:03Z)
• From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation [32.41847293364159]
  PRIMO R1 is a 7B framework that transforms video MLLMs into active "Critics"<n>We leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation.<n>We establish state-of-the-art performance on RoboFail benchmark with 67.0% accuracy, surpassing closed-source models like OpenAI o1 by 6.2%.
  arXiv  Detail & Related papers  (2026-03-16T17:53:28Z)
• InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation [77.07565723756119]
  We present InternVLA-A1, a vision-language-action model with dynamic predictive capabilities.<n>We pre-train these models on heterogeneous data sources over real-world robot data, synthetic simulation data, and human videos.<n>We evaluate InternVLA-A1 on 12 real-world robotic tasks and a simulation benchmark.
  arXiv  Detail & Related papers  (2026-01-05T18:54:29Z)
• InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy [138.89177083578213]
  We introduce InternVLA-M1, a unified framework for spatial grounding and robot control.<n>InternVLA-M1 employs a two-stage pipeline: (i) spatial grounding pre-training on over 2.3M spatial reasoning data, and (ii) spatially guided action post-training.<n>Results: InternVLA-M1 outperforms its variant without spatial guidance by +14.6% on SimplerEnv Google Robot, +17% on WidowX, and +4.3% on LIBERO Franka.
  arXiv  Detail & Related papers  (2025-10-15T17:30:05Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics [67.11221574129937]
  Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world.<n>We propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding.<n>RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-06-04T17:59:27Z)
• From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation [35.79160868966466]
  We propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning.<n>Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals.<n>We show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.
  arXiv  Detail & Related papers  (2025-05-13T13:20:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.