Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning

• URL: http://arxiv.org/abs/2603.14811v1
• Date: Mon, 16 Mar 2026 04:27:53 GMT
• Title: Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learning
• Abstract summary: We introduce the Ego-to-World benchmark, which evaluates a vision-language model's ability to fuse heterogeneous viewpoints across three tasks.<n>We propose CoRL, a two-stage framework that combines Chain-of-Thought supervised fine-tuning with reinforcement learning.<n>We show that CoRL consistently surpasses strong proprietary and open-source baselines on both reasoning and perception-grounding metrics.
• Score: 61.753025885751036
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding the world from distributed, partial viewpoints is a fundamental challenge for embodied multi-agent systems. Each agent perceives the environment through an ego-centric view that is often limited by occlusion and ambiguity. To study this problem, we introduce the Ego-to-World (E2W) benchmark, which evaluates a vision-language model's ability to fuse heterogeneous viewpoints across three tasks: (i) global counting, (ii) relational location reasoning, and (iii) action-oriented grasping that requires predicting view-specific image coordinates. To address this setting, we propose CoRL, a two-stage framework that combines Chain-of-Thought supervised fine-tuning with reinforcement learning using Group-Relative Policy Optimization. Its core component, the Cross-View Spatial Reward (CVSR), provides dense task-aligned feedback by linking reasoning steps to visual evidence, ensuring coherent cross-view entity resolution, and guiding the model toward correct final predictions. Experiments on E2W show that CoRL consistently surpasses strong proprietary and open-source baselines on both reasoning and perception-grounding metrics, while ablations further confirm the necessity of each CVSR component. Beyond that, CoRL generalizes to external spatial reasoning benchmarks and enables effective real-world multi-robot manipulation with calibrated multi-camera rigs, demonstrating cross-view localization and successful grasp-and-place execution. Together, E2W and CoRL provide a principled foundation for learning world-centric scene understanding from distributed, ego-centric observations, advancing collaborative embodied AI.

Related papers

• Towards High-Resolution Visual Perception via Hierarchical Entity Exploration [36.649153640650496]
  Hierarchical Entity Exploration transforms static image understanding into query-guided entity exploration.<n>HEE is a training-free and model-agnostic framework that transforms static image understanding into dynamic, query-guided entity exploration.
  arXiv  Detail & Related papers  (2026-07-01T11:41:21Z)
• AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization [79.26935895370191]
  AnchorWorld is a framework that advances egocentric simulation through enhanced interaction integrity.<n> auxiliary training supervision incorporates viewpoints decoupled from the agent's first-person sensorium.<n>We propose a simple yet effective mechanism for customizing self-evolving worlds.
  arXiv  Detail & Related papers  (2026-06-05T14:43:13Z)
• EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding [21.745254094130075]
  This paper introduces EARL, an Egocentric Analysis-guided Reinforcement Learning framework that transfers coarse interaction semantics to query-oriented answering and grounding.<n>Specifically, EARL adopts a two-stage parsing framework including coarse-grained interpretation and fine-grained response.<n> Experiments on Ego-IRGBench show that EARL achieves 65.48% cIoU for pixel grounding, outperforming previous RL-based methods by 8.37%.
  arXiv  Detail & Related papers  (2026-05-14T12:10:27Z)
• PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning [80.94559742826083]
  Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal.<n>Recent work shows that providing a fine-grained, model-intrinsic signal effectively improves language reasoning training by providing step-level guidance without costly external models.<n>While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy.<n>We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature.
  arXiv  Detail & Related papers  (2026-05-13T12:55:18Z)
• Reference-Free Omnidirectional Stereo Matching via Multi-View Consistency Maximization [22.711843818785454]
  Free OmniMVS can aggregate pair-wise correlations into a robust, omni-aware, and global consensus.<n>To realize visibility-aware consensus, we propose a lightweight attention mechanism that adaptively fuses the correlation vectors.<n>Experiments on diverse benchmark datasets demonstrate the superiority of our method for globally consistent, visibility-aware, and scale-aware depth estimation.
  arXiv  Detail & Related papers  (2026-03-16T09:23:23Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• Keep it SymPL: Symbolic Projective Layout for Allocentric Spatial Reasoning in Vision-Language Models [5.961445903498366]
  We introduce Projective Layout (SymPL), a framework that reformulates allocentric reasoning into symbolic forms that VLMs handle well.<n>Experiments demonstrate that this reformulation substantially improves performance in both allocentric and egocentric tasks.
  arXiv  Detail & Related papers  (2026-02-22T10:18:54Z)
• RegionReasoner: Region-Grounded Multi-Round Visual Reasoning [69.75509909581133]
  RegionReasoner is a reinforcement learning framework for visual reasoning.<n>It enforces grounded reasoning by requiring each reasoning trace to explicitly cite the corresponding reference bounding boxes.<n>RegionReasoner is optimized with structured rewards combining grounding fidelity and global-local semantic alignment.
  arXiv  Detail & Related papers  (2026-02-03T16:52:16Z)
• Revisiting Multi-Task Visual Representation Learning [52.93947931352643]
  We introduce MTV, a principled multi-task visual pretraining framework.<n>We leverage high-capacity "expert" models to synthesize dense, structured pseudo-labels at scale.<n>Our results demonstrate that MTV achieves "best-of-both-worlds" performance.
  arXiv  Detail & Related papers  (2026-01-20T11:59:19Z)
• Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion [31.189038928192648]
  Co2S is a semi-supervised RS segmentation framework that fuses priors from vision-language models and self-supervised models.<n>An explicit-implicit semantic co-guidance mechanism is introduced that utilizes text embeddings and learnable queries.<n>Experiments on six popular datasets demonstrate the superiority of the proposed method.
  arXiv  Detail & Related papers  (2025-12-28T18:24:19Z)
• A Tale of Two Experts: Cooperative Learning for Source-Free Unsupervised Domain Adaptation [59.88864205383671]
  Source-Free Unsupervised Domain Adaptation (SFUDA) addresses the realistic challenge of adapting a source-trained model to a target domain without access to the source data.<n>Existing SFUDA methods either exploit only the source model's predictions or fine-tune large multimodal models.<n>We propose the Experts Cooperative Learning (EXCL) to exploit complementary insights and the latent structure of target data.
  arXiv  Detail & Related papers  (2025-09-26T11:39:50Z)
• HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Model [13.82578761807402]
  We introduce HOID-R1, the first HOI detection framework that integrates chain-of-thought (CoT) guided fine-tuning with group relative policy optimization.<n>To mitigate hallucinations in the CoT reasoning, we introduce an "MLLM-as-a-judge" mechanism that supervises the CoT outputs.<n>Experiments show that HOID-R1 achieves state-of-the-art performance on HOI detection benchmarks and outperforms existing methods in open-world generalization to novel scenarios.
  arXiv  Detail & Related papers  (2025-08-15T09:28:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.