Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Abstract Overview
Imagine3D-LLM is a multi-view multimodal large language model that inserts a compact set of learnable Gaussian summary tokens between image and text tokens, then decodes those tokens into a compact 3D Gaussian Splatting scene representation before answering. The model is trained jointly with standard language modeling, photometric reconstruction of the input views, and distillation from a compact Gaussian teacher to speed convergence. The paper argues that this bottlenecked reconstruction process encourages the model to merge recurring content across views into object-level, view-consistent representations rather than relying on dense pixel-level geometry alone. Analyses in the paper further indicate that reconstruction supervision sharpens cross-view correspondences in the model's image features and leads summary tokens to group by object without explicit object supervision.
Novelty
The distinctive idea is to make an MLLM explicitly "imagine" a coarse 3D scene representation from multi-view images before producing a language answer, using Gaussian summary tokens as an internal bottleneck. Unlike approaches that inject fine-grained geometric signals or external 3D features directly, this method learns a compact object-centric 3D representation inside the LLM and conditions reasoning on it.
Results
Across seven benchmarks, the model reports consistent gains over prior 3D MLLMs and strong 2D baselines. On SPAR-Bench, Imagine3D-LLM reaches 68.5 average accuracy, exceeding 3DThinker-7B's 63.3, and on ScanNet-based tasks it improves over a controlled baseline with the same backbone, data, and schedule. Ablations indicate that the reconstruction objective is the main source of the gains, while teacher distillation primarily makes training converge faster.
Key Points
- The method appends learnable Gaussian summary tokens to multi-view visual tokens and decodes them at an intermediate LLM layer into a compact 3D Gaussian scene representation.
- A token bottleneck is used to encourage object-level grouping across viewpoints, and the paper presents analyses showing sharper cross-view correspondence and emergent object-centric clustering.
- Empirically, the model outperforms prior methods on ScanNet-based 3D reasoning, captioning, and grounding benchmarks and achieves state-of-the-art results on SPAR-Bench.