When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answering

• URL: http://arxiv.org/abs/2512.04597v1
• Date: Thu, 04 Dec 2025 09:17:40 GMT
• Title: When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answering
• Abstract summary: Embodied Question Answering (EQA) requires an agent to interpret language, perceive its environment, and navigate within 3D scenes to produce responses.<n>We focus on a minimal requirement for EQA agents, abstention: knowing when to withhold an answer.<n>We derive five representative categories requiring abstention: actionability, referential underspecification, preference dependence, information unavailability, and false presupposition.
• Score: 27.058973883094
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied Question Answering (EQA) requires an agent to interpret language, perceive its environment, and navigate within 3D scenes to produce responses. Existing EQA benchmarks assume that every question must be answered, but embodied agents should know when they do not have sufficient information to answer. In this work, we focus on a minimal requirement for EQA agents, abstention: knowing when to withhold an answer. From an initial study of 500 human queries, we find that 32.4% contain missing or underspecified context. Drawing on this initial study and cognitive theories of human communication errors, we derive five representative categories requiring abstention: actionability limitation, referential underspecification, preference dependence, information unavailability, and false presupposition. We augment OpenEQA by having annotators transform well-posed questions into ambiguous variants outlined by these categories. The resulting dataset, AbstainEQA, comprises 1,636 annotated abstention cases paired with 1,636 original OpenEQA instances for balanced evaluation. Evaluating on AbstainEQA, we find that even the best frontier model only attains 42.79% abstention recall, while humans achieve 91.17%. We also find that scaling, prompting, and reasoning only yield marginal gains, and that fine-tuned models overfit to textual cues. Together, these results position abstention as a fundamental prerequisite for reliable interaction in embodied settings and as a necessary basis for effective clarification.

Related papers

• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Implicit Intelligence -- Evaluating Agents on What Users Don't Say [0.3580891736370874]
  Implicit Intelligence is an evaluation framework testing whether AI agents can move beyond prompt-following to become genuine goal-fulfillers.<n>Our scenarios feature apparent simplicity in user requests, hidden complexity in correct solutions, and discoverability of constraints through environmental exploration.
  arXiv  Detail & Related papers  (2026-02-23T23:46:55Z)
• AURA Score: A Metric For Holistic Audio Question Answering Evaluation [57.042210272137396]
  We introduce AQEval to enable systematic benchmarking of AQA metrics.<n>It is the first benchmark of its kind, consisting of 10k model responses annotated by multiple humans for their correctness and relevance.<n>Second, we conduct a comprehensive analysis of existing AQA metrics on AQEval, highlighting weak correlation with human judgment.<n>Third, we propose a new metric - AURA score, to better evaluate open-ended model responses.
  arXiv  Detail & Related papers  (2025-10-06T15:41:34Z)
• Pathways of Thoughts: Multi-Directional Thinking for Long-form Personalized Question Answering [57.12316804290369]
  Personalization is essential for adapting question answering systems to user-specific information needs.<n>We propose Pathways of Thoughts (PoT), an inference-stage method that applies to any large language model (LLM) without requiring task-specific fine-tuning.<n>PoT consistently outperforms competitive baselines, achieving up to a 13.1% relative improvement.
  arXiv  Detail & Related papers  (2025-09-23T14:44:46Z)
• CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering [9.50840225852638]
  Conditional Ambiguous Question-Answering (CondAmbigQA) is a benchmark comprising 2,000 ambiguous queries and condition-aware evaluation metrics.<n>Experiments demonstrate that models considering conditions before answering improve answer accuracy by 11.75%, with an additional 7.15% gain when conditions are explicitly provided.
  arXiv  Detail & Related papers  (2025-02-03T17:01:51Z)
• SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis [4.368712652579087]
  SubjECTive-QA is a human annotated dataset on Earnings Call Transcripts' (ECTs)<n>The dataset includes 49,446 annotations for long-form QA pairs across six features: Assertive, Cautious, Optimistic, Specific, Clear, and Relevant.<n>Our findings are that the best-performing Pre-trained Language Model (PLM), RoBERTa-base, has similar weighted F1 scores to Llama-3-70b-Chat on features with lower subjectivity.
  arXiv  Detail & Related papers  (2024-10-28T01:17:34Z)
• EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering [21.114403949257934]
  Large vision-language models (VLMs) have shown promise for Embodied Question Answering (EQA)<n>Existing approaches either treat it as static video question answering without active exploration or restrict answers to a closed set of choices.<n>We introduce EfficientEQA, a novel framework that couples efficient exploration with free-form answer generation.<n>Our experimental results show that EfficientEQA achieves over 15% higher answer accuracy and requires over 20% fewer exploration steps than state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-26T19:48:47Z)
• Is the House Ready For Sleeptime? Generating and Evaluating Situational Queries for Embodied Question Answering [48.43453390717167]
  We present and tackle the problem of Embodied Question Answering with Situational Queries (S-EQA) in a household environment.<n>Unlike prior EQA work, situational queries require the agent to correctly identify multiple object-states and reach a consensus on their states for an answer.<n>We introduce a novel Prompt-Generate-Evaluate scheme that wraps around an LLM's output to generate unique situational queries and corresponding consensus object information.
  arXiv  Detail & Related papers  (2024-05-08T00:45:20Z)
• GSQA: An End-to-End Model for Generative Spoken Question Answering [54.418723701886115]
  We introduce the first end-to-end Generative Spoken Question Answering (GSQA) model that empowers the system to engage in abstractive reasoning. Our model surpasses the previous extractive model by 3% on extractive QA datasets. Our GSQA model shows the potential to generalize to a broad spectrum of questions, thus further expanding the spoken question answering capabilities of abstractive QA.
  arXiv  Detail & Related papers  (2023-12-15T13:33:18Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)
• Attention-Based Methods For Audio Question Answering [16.82832919748399]
  We propose neural network architectures based on self-attention and cross-attention for the audio question answering task. All our models are trained on the recently proposed Clotho-AQA dataset for both binary yes/no questions and single-word answer questions.
  arXiv  Detail & Related papers  (2023-05-31T12:00:51Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.