MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding

• URL: http://arxiv.org/abs/2604.09167v1
• Date: Fri, 10 Apr 2026 09:51:42 GMT
• Title: MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding
• Abstract summary: We present MAG-3D, a training-free multi-agent framework for grounded 3D reasoning with off-the-shelf vision-language models.<n>We propose a planning agent that decomposes the task and orchestrates the overall reasoning process, a grounding agent that performs free-form 3D grounding and relevant frame retrieval from extensive 3D scene observations, and a coding agent that conducts flexible geometric reasoning and explicit verification through executable programs.
• Score: 25.15914325538431
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language models (VLMs) have achieved strong performance in multimodal understanding and reasoning, yet grounded reasoning in 3D scenes remains underexplored. Effective 3D reasoning hinges on accurate grounding: to answer open-ended queries, a model must first identify query-relevant objects and regions in a complex scene, and then reason about their spatial and geometric relationships. Recent approaches have demonstrated strong potential for grounded 3D reasoning. However, they often rely on in-domain tuning or hand-crafted reasoning pipelines, which limit their flexibility and zero-shot generalization to novel environments. In this work, we present MAG-3D, a training-free multi-agent framework for grounded 3D reasoning with off-the-shelf VLMs. Instead of relying on task-specific training or fixed reasoning procedures, MAG-3D dynamically coordinates expert agents to address the key challenges of 3D reasoning. Specifically, we propose a planning agent that decomposes the task and orchestrates the overall reasoning process, a grounding agent that performs free-form 3D grounding and relevant frame retrieval from extensive 3D scene observations, and a coding agent that conducts flexible geometric reasoning and explicit verification through executable programs. This multi-agent collaborative design enables flexible training-free 3D grounded reasoning across diverse scenes and achieves state-of-the-art performance on challenging benchmarks.

Related papers

• ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA [26.947027118604368]
  textbfViewMind3D is a fully training-free and modular framework for 3D spatial reasoning over multi-view observations of a scene.<n>We show that ViewMind3D achieves competitive performance compared to prior training-free and fine-tuned 3D-LLMs.
  arXiv  Detail & Related papers  (2026-07-30T16:14:30Z)
• Do 3D Large Language Models Really Understand 3D Spatial Relationships? [80.64317885117704]
  3D Large-Language Models claim to understand 3D worlds, especially spatial relationships among objects.<n>We find that fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input.<n>We introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning.
  arXiv  Detail & Related papers  (2026-03-06T16:04:34Z)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models [100.13033631690114]
  Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding.<n>We introduce SandboxVLM, a framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM.<n>Our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods.
  arXiv  Detail & Related papers  (2025-11-14T04:16:09Z)
• Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views [41.05815610513033]
  3DThinker is a framework that exploits the rich geometric information embedded within images while reasoning, like humans do.<n>Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training.
  arXiv  Detail & Related papers  (2025-10-21T13:36:58Z)
• SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting [85.87902260102652]
  We introduce the novel task of Sequential 3D Gaussian Affordance Reasoning.<n>We then propose SeqSplatNet, an end-to-end framework that directly maps an instruction to a sequence of 3D affordance masks.<n>Our method sets a new state-of-the-art on our challenging benchmark, effectively advancing affordance reasoning from single-step interactions to complex, sequential tasks at the scene level.
  arXiv  Detail & Related papers  (2025-07-31T17:56:55Z)
• SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning [23.6011224506759]
  We introduce a novel large vision-language model (LVLM) that addresses 3D spatial reasoning.<n>Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning.<n>Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-04-28T17:48:43Z)
• 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds [81.14476072159049]
  3D Affordance detection is a challenging problem with broad applications on various robotic tasks.<n>We reformulate the traditional affordance detection paradigm into textit Reasoning Affordance (IRAS) task.<n>We propose 3D-ADLLM, a framework designed for reasoning affordance detection in 3D open-scene.
  arXiv  Detail & Related papers  (2025-02-27T12:29:44Z)
• Multimodal 3D Reasoning Segmentation with Complex Scenes [92.92045550692765]
  We propose a 3D reasoning segmentation task for reasoning segmentation with multiple objects in scenes.<n>The task allows producing 3D segmentation masks and detailed textual explanations as enriched by 3D spatial relations among objects.<n>In addition, we design MORE3D, a novel 3D reasoning network that works with queries of multiple objects.
  arXiv  Detail & Related papers  (2024-11-21T08:22:45Z)
• ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities [23.18281583681258]
  We propose a new task called 3D reasoning grounding and introduce a new benchmark ScanReason. ScanReason provides over 10K question-answer-location pairs from five reasoning types that require the synerization of reasoning and grounding. A chain-of-grounding mechanism is proposed to further boost the performance with interleaved reasoning and grounding steps during inference.
  arXiv  Detail & Related papers  (2024-07-01T17:59:35Z)
• Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model [108.35777542298224]
  Reason3D processes point cloud data and text prompts to produce textual responses and segmentation masks.<n>We propose a hierarchical mask decoder that employs a coarse-to-fine approach to segment objects within expansive scenes.
  arXiv  Detail & Related papers  (2024-05-27T17:59:41Z)
• OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning [68.45848423501927]
  We propose a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning.<n>Our approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions.
  arXiv  Detail & Related papers  (2024-05-02T17:59:24Z)
• Think-Program-reCtify: 3D Situated Reasoning with Large Language Models [68.52240087262825]
  This work addresses the 3D situated reasoning task which aims to answer questions given egocentric observations in a 3D environment. We propose a novel framework that leverages the planning, tool usage, and reflection capabilities of large language models (LLMs) through a ThinkProgram-reCtify loop. Experiments and analysis on the SQA3D benchmark demonstrate the effectiveness, interpretability and robustness of our method.
  arXiv  Detail & Related papers  (2024-04-23T03:22:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.