3D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering

• URL: http://arxiv.org/abs/2507.12026v1
• Date: Wed, 16 Jul 2025 08:38:26 GMT
• Title: 3D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering
• Authors: Rongtao Xu, Han Gao, Mingming Yu, Dong An, Shunpeng Chen, Changwei Wang, Li Guo, Xiaodan Liang, Shibiao Xu,
• Abstract summary: 3D-MoRe is designed to generate large-scale 3D-language datasets by leveraging the strengths of foundational models.<n>The framework integrates key components, including multi-modal embedding, cross-modal interaction, and a language model decoder.<n>Using the ScanNet 3D scene dataset, along with text annotations from ScanQA and ScanRefer, 3D-MoRe generates 62,000 question-answer pairs and 73,000 object descriptions.
• Score: 52.01655676571933
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the growing need for diverse and scalable data in indoor scene tasks, such as question answering and dense captioning, we propose 3D-MoRe, a novel paradigm designed to generate large-scale 3D-language datasets by leveraging the strengths of foundational models. The framework integrates key components, including multi-modal embedding, cross-modal interaction, and a language model decoder, to process natural language instructions and 3D scene data. This approach facilitates enhanced reasoning and response generation in complex 3D environments. Using the ScanNet 3D scene dataset, along with text annotations from ScanQA and ScanRefer, 3D-MoRe generates 62,000 question-answer (QA) pairs and 73,000 object descriptions across 1,513 scenes. We also employ various data augmentation techniques and implement semantic filtering to ensure high-quality data. Experiments on ScanQA demonstrate that 3D-MoRe significantly outperforms state-of-the-art baselines, with the CIDEr score improving by 2.15\%. Similarly, on ScanRefer, our approach achieves a notable increase in CIDEr@0.5 by 1.84\%, highlighting its effectiveness in both tasks. Our code and generated datasets will be publicly released to benefit the community, and both can be accessed on the https://3D-MoRe.github.io.

Related papers

• Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation [92.17176311351469]
  We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework.<n>Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale.<n>Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs.
  arXiv  Detail & Related papers  (2025-02-04T18:18:50Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan.<n>The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• Unifying 3D Vision-Language Understanding via Promptable Queries [39.55438547712157]
  unified model for 3D vision-language (3D-VL) understanding. PQ3D is capable of using Promptable Queries to tackle a wide range of 3D-VL tasks. Tested across ten diverse 3D-VL datasets, PQ3D demonstrates impressive performance on these tasks.
  arXiv  Detail & Related papers  (2024-05-19T04:35:05Z)
• Grounded 3D-LLM with Referent Tokens [58.890058568493096]
  We propose Grounded 3D-LLM to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats.
  arXiv  Detail & Related papers  (2024-05-16T18:03:41Z)
• ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes [48.65360357173095]
  Scan Entities in 3D (ScanEnts3D) dataset provides explicit correspondences between 369k objects across 84k natural referential sentences. We show that by incorporating intuitive losses that enable learning from this novel dataset, we can significantly improve the performance of several recently introduced neural listening architectures.
  arXiv  Detail & Related papers  (2022-12-12T21:25:58Z)
• CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection [57.44434974289945]
  We propose Contextualized Multi-Stage Refinement for 3D Object Detection (CMR3D) framework. Our framework takes a 3D scene as input and strives to explicitly integrate useful contextual information of the scene. In addition to 3D object detection, we investigate the effectiveness of our framework for the problem of 3D object counting.
  arXiv  Detail & Related papers  (2022-09-13T05:26:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.