Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation

• URL: http://arxiv.org/abs/2510.20310v2
• Date: Mon, 27 Oct 2025 17:58:06 GMT
• Title: Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
• Abstract summary: Embodied Question Answering (EQA) requires agents to explore 3D environments to obtain observations and answer questions related to the scene.<n>We introduce ToolEQA, an agent that integrates external tools with multi-step reasoning.<n>We further design a novel EQA data generation pipeline that automatically constructs large-scale EQA tasks with reasoning trajectories and corresponding answers.
• Score: 42.392504686945614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied Question Answering (EQA) requires agents to explore 3D environments to obtain observations and answer questions related to the scene. Existing methods leverage VLMs to directly explore the environment and answer questions without explicit thinking or planning, which limits their reasoning ability and results in excessive or inefficient exploration as well as ineffective responses. In this paper, we introduce ToolEQA, an agent that integrates external tools with multi-step reasoning, where external tools can provide more useful information for completing the task, helping the model derive better exploration directions in the next step of reasoning and thus obtaining additional effective information. This enables ToolEQA to generate more accurate responses with a shorter exploration distance. To enhance the model's ability for tool-usage and multi-step reasoning, we further design a novel EQA data generation pipeline that automatically constructs large-scale EQA tasks with reasoning trajectories and corresponding answers. Based on the pipeline, we collect the EQA-RT dataset that contains about 18K tasks, divided into a training set EQA-RT-Train, and two test sets EQA-RT-Seen (scenes overlapping with the training set) and EQA-RT-Unseen (novel scenes). Experiments on EQA-RT-Seen and EQA-RT-Unseen show that ToolEQA improves the success rate by 9.2~20.2% over state-of-the-art baselines, while outperforming the zero-shot ToolEQA by 10% in success rate. In addition, ToolEQA also achieves state-of-the-art performance on the HM-EQA, OpenEQA, and EXPRESS-Bench datasets, demonstrating its generality. Our homepage see https://tooleqa.github.io.

Related papers

• IQA-T1: Tool-based Visual Evidence Reasoning for Image Quality Assessment [59.83941105390897]
  We propose IQA-T1, a tool-based visual evidence reasoning framework.<n>During inference, the model autonomously invokes specialized analysis tools to generate structured visual evidence.<n>Experiments show that IQA-T1 achieves the best overall performance across datasets.
  arXiv  Detail & Related papers  (2026-07-14T05:47:32Z)
• AURA Score: A Metric For Holistic Audio Question Answering Evaluation [57.042210272137396]
  We introduce AQEval to enable systematic benchmarking of AQA metrics.<n>It is the first benchmark of its kind, consisting of 10k model responses annotated by multiple humans for their correctness and relevance.<n>Second, we conduct a comprehensive analysis of existing AQA metrics on AQEval, highlighting weak correlation with human judgment.<n>Third, we propose a new metric - AURA score, to better evaluate open-ended model responses.
  arXiv  Detail & Related papers  (2025-10-06T15:41:34Z)
• AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation [31.02336903452371]
  AirQA is a human-annotated comprehensive paper QA dataset in the field of artificial intelligence (AI)<n>With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervention.<n>ExTrActor consistently improves the multi-turn tool-use capability of small models, enabling them to achieve performance comparable to larger ones.
  arXiv  Detail & Related papers  (2025-09-21T07:24:17Z)
• EQA-RM: A Generative Embodied Reward Model with Test-time Scaling [43.79542068802909]
  We introduce EQA-RM, a novel generative multimodal reward model specifically architected for Embodied Question Answering (EQA)<n>The generative nature of EQA-RM provides interpretable, structured reward feedback (beyond simple scalars)<n>We introduce EQARewardBench, a new benchmark built on OpenEQA for standardized EQA reward model assessment.
  arXiv  Detail & Related papers  (2025-06-12T06:25:04Z)
• Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answering [87.76784654371312]
  Embodied Question Answering requires agents to dynamically explore 3D environments, actively gather visual information, and perform multi-step reasoning to answer questions.<n>Existing datasets often introduce biases or prior knowledge, leading to disembodied reasoning.<n>We construct the largest dataset designed specifically to evaluate both exploration and reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-14T06:29:47Z)
• EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering [21.114403949257934]
  Large vision-language models (VLMs) have shown promise for Embodied Question Answering (EQA)<n>Existing approaches either treat it as static video question answering without active exploration or restrict answers to a closed set of choices.<n>We introduce EfficientEQA, a novel framework that couples efficient exploration with free-form answer generation.<n>Our experimental results show that EfficientEQA achieves over 15% higher answer accuracy and requires over 20% fewer exploration steps than state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-26T19:48:47Z)
• AQUALLM: Audio Question Answering Data Generation Using Large Language Models [2.2232550112727267]
  We introduce a scalable AQA data generation pipeline, which relies on Large Language Models (LLMs) We present three extensive and high-quality benchmark datasets for AQA. Models trained on our datasets demonstrate enhanced generalizability when compared to models trained using human-annotated AQA data.
  arXiv  Detail & Related papers  (2023-12-28T20:01:27Z)
• NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions [80.60423934589515]
  We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark. We set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. We find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning.
  arXiv  Detail & Related papers  (2021-05-18T04:56:46Z)
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA [95.9105154311491]
  We introduce two approaches to improve unsupervised Question Answering (QA) First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA) Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA.
  arXiv  Detail & Related papers  (2020-05-06T15:56:06Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.