Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models

• URL: http://arxiv.org/abs/2605.18431v2
• Date: Tue, 19 May 2026 05:12:41 GMT
• Title: Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models
• Abstract summary: We propose a framework for fine-grained cooperative spatial reasoning.<n> SP-CoR combines dynamics-aware multi-robot frame sampling, spectral- and physics-guided view fusion, and physics-aligned prompt distillation.<n>It consistently improves cooperative reasoning, outperforming the strongest finetuned baseline by +3.87% on Habitat and +7.12% on iGibson.
• Score: 76.29781698910202
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) have made substantial progress in egocentric video understanding, but their ability to reason cooperatively from multiple embodied viewpoints remains largely unexplored. We study this problem through multi-robot cooperative dynamic spatial reasoning, where a model must answer spatial, temporal, visibility, and coordination questions by integrating synchronized egocentric videos from a team of moving robots. To support this setting, we introduce CoopSR, the first benchmark for this task, together with EgoTeam, a multi-robot egocentric QA dataset. EgoTeam contains 114,227 QA pairs spanning 19 question types, four difficulty tiers, and three team sizes in Habitat and iGibson, along with a real-world test set of around 2,326 QAs collected using two quadruped robots. We further propose SP-CoR (Spectral and Physics-Informed Cooperative Reasoner), an MLLM framework for fine-grained cooperative spatial reasoning. SP-CoR combines dynamics-aware multi-robot frame sampling, spectral- and physics-guided view fusion, and physics-aligned prompt distillation, enabling the model to benefit from privileged robot-pose supervision during training while requiring only egocentric videos at test time. Across 22 MLLM baselines, SP-CoR consistently improves cooperative reasoning, outperforming the strongest fine-tuned baseline by +3.87% on Habitat and +7.12% on iGibson. It also shows stronger generalization to unseen team sizes and real-world robot tests. Code can be found at https://github.com/KPeng9510/seeing-together.git.

Related papers

• OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction [21.78933851010173]
  We present OmniRobotHome, the first room-scale residential platform that unifies real-time 3D human and object perception with coordinated multi-robot actuation.<n>We focus on two central problems: safety in shared human-robot environments and human-anticipatory robotic assistance.
  arXiv  Detail & Related papers  (2026-04-30T17:59:58Z)
• M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction [5.410329948686681]
  M2HRI is a multimodal multi-agent framework built on large language models that equips each robot with distinct personality and long-term memory.<n>Results show that both agent individuality and structured coordination are essential for coherent and socially appropriate multi-agent HRI.
  arXiv  Detail & Related papers  (2026-04-13T19:08:05Z)
• MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents [54.48066948369172]
  As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future.<n>Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video.<n>We first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents.
  arXiv  Detail & Related papers  (2026-03-10T15:48:35Z)
• Leveraging Adaptive Group Negotiation for Heterogeneous Multi-Robot Collaboration with Large Language Models [27.286674175117394]
  Large Language Models (LLMs) excel at reasoning and planning, their potential for coordinated control has not been fully explored.<n>Inspired by human teamwork, we present CLiMRS, an adaptive group negotiation framework among LLMs for multi-robot collaboration.<n>Our results demonstrate that leveraging human-inspired group formation and negotiation principles significantly enhances the efficiency of heterogeneous multi-robot collaboration.
  arXiv  Detail & Related papers  (2025-12-29T14:50:52Z)
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT [56.24624833924252]
  EgoThinker is a framework that endows MLs with robust egocentric reasoning capabilities through-temporal chain-of-thought supervision and a two-stage learning curriculum.<n>EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained-temporal localization tasks.
  arXiv  Detail & Related papers  (2025-10-27T17:38:17Z)
• EgoM2P: Egocentric Multimodal Multitask Pretraining [55.259234688003545]
  Building large-scale egocentric multimodal and multitask models presents unique challenges.<n> EgoM2P is a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding.<n>We will fully open-source EgoM2P to support the community and advance egocentric vision research.
  arXiv  Detail & Related papers  (2025-06-09T15:59:25Z)
• RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics [67.11221574129937]
  Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world.<n>We propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding.<n>RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2025-06-04T17:59:27Z)
• GRUtopia: Dream General Robots in a City at Scale [65.08318324604116]
  This paper introduces project GRUtopia, the first simulated interactive 3D society designed for various robots. GRScenes includes 100k interactive, finely annotated scenes, which can be freely combined into city-scale environments. GRResidents is a Large Language Model (LLM) driven Non-Player Character (NPC) system that is responsible for social interaction.
  arXiv  Detail & Related papers  (2024-07-15T17:40:46Z)
• A Learning Approach to Robot-Agnostic Force-Guided High Precision Assembly [6.062589413216726]
  We propose a learning approach to high-precision robotic assembly problems. We focus on the contact-rich phase, where the assembly pieces are in close contact with each other. Our training environment is robotless, as the end-effector is not attached to any specific robot.
  arXiv  Detail & Related papers  (2020-10-15T22:33:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.