MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents
- URL: http://arxiv.org/abs/2603.09827v2
- Date: Wed, 11 Mar 2026 02:13:29 GMT
- Title: MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents
- Abstract summary: As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future.<n>Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video.<n>We first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents.
- Score: 54.48066948369172
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future. To ensure better communication between human users and the multi-agent system, it is crucial to interpret incoming information from agents in parallel and refer to the appropriate context for each query. Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video and correctly aggregating multiple egocentric videos to construct system-level memory. In this work, we first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents. To facilitate research in this direction, we introduce MultiAgent-EgoQA (MA-EgoQA), a benchmark designed to systemically evaluate existing models in our scenario. MA-EgoQA provides 1.7k questions unique to multiple egocentric streams, spanning five categories: social interaction, task coordination, theory-of-mind, temporal reasoning, and environmental interaction. We further propose a simple baseline model for MA-EgoQA named EgoMAS, which leverages shared memory across embodied agents and agent-wise dynamic retrieval. Through comprehensive evaluation across diverse baselines and EgoMAS on MA-EgoQA, we find that current approaches are unable to effectively handle multiple egocentric streams, highlighting the need for future advances in system-level understanding across the agents. The code and benchmark are available at https://ma-egoqa.github.io.
Related papers
- EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent [40.15272756472325]
Egocentric Procedural Understanding VQA task (EgoProceVQA)<n>EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types.<n>EgoProceAgent, a self-skill-exploration agentic framework.
arXiv Detail & Related papers (2026-07-15T12:55:35Z) - GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents [4.920953895710103]
We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding.<n>We densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World.<n>From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy.
arXiv Detail & Related papers (2026-03-25T14:10:45Z) - EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT [56.24624833924252]
EgoThinker is a framework that endows MLs with robust egocentric reasoning capabilities through-temporal chain-of-thought supervision and a two-stage learning curriculum.<n>EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained-temporal localization tasks.
arXiv Detail & Related papers (2025-10-27T17:38:17Z) - EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering [59.94048858464922]
EgoCross is a benchmark to evaluate the cross-domain generalization of MLLMs in EgocentricQA.<n>EgoCross covers four diverse and challenging domains, including surgery, industry, extreme sports, and animal perspective.<n>It comprises approximately 1,000 QA pairs across 798 video clips, spanning four key QA tasks: prediction, recognition, localization, and counting.
arXiv Detail & Related papers (2025-08-14T15:11:20Z) - EgoM2P: Egocentric Multimodal Multitask Pretraining [55.259234688003545]
Building large-scale egocentric multimodal and multitask models presents unique challenges.<n> EgoM2P is a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding.<n>We will fully open-source EgoM2P to support the community and advance egocentric vision research.
arXiv Detail & Related papers (2025-06-09T15:59:25Z) - MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA [72.47344411599322]
This research aims to explore building a multimodal foundation model for egocentric video understanding.<n>We automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data.<n>We contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths.
arXiv Detail & Related papers (2024-10-09T17:59:59Z) - Multi-LLM QA with Embodied Exploration [55.581423861790945]
We investigate the use of Multi-Embodied LLM Explorers (MELE) for question-answering in an unknown environment.
Multiple LLM-based agents independently explore and then answer queries about a household environment.
We analyze different aggregation methods to generate a single, final answer for each query.
arXiv Detail & Related papers (2024-06-16T12:46:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.