Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding

• URL: http://arxiv.org/abs/2512.12822v1
• Date: Sun, 14 Dec 2025 20:02:43 GMT
• Title: Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
• Authors: Yongyuan Liang, Xiyao Wang, Yuanchen Ju, Jianwei Yang, Furong Huang,
• Abstract summary: Lemon is a unified transformer architecture that processes 3D point cloud patches and language tokens as a single sequence.<n>To handle the complexity of 3D data, we develop a structured patchification and tokenization scheme that preserves spatial context.<n>Lemon establishes new state-of-the-art performance across comprehensive 3D understanding and reasoning tasks.
• Score: 80.66591664266744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling large multimodal models (LMMs) to 3D understanding poses unique challenges: point cloud data is sparse and irregular, existing models rely on fragmented architectures with modality-specific encoders, and training pipelines often suffer from instability and poor scalability. We introduce Lemon, a unified transformer architecture that addresses these challenges by jointly processing 3D point cloud patches and language tokens as a single sequence. Unlike prior work that relies on modality-specific encoders and cross-modal alignment modules, this design enables early spatial-linguistic fusion, eliminates redundant encoders, improves parameter efficiency, and supports more effective model scaling. To handle the complexity of 3D data, we develop a structured patchification and tokenization scheme that preserves spatial context, and a three-stage training curriculum that progressively builds capabilities from object-level recognition to scene-level spatial reasoning. Lemon establishes new state-of-the-art performance across comprehensive 3D understanding and reasoning tasks, from object recognition and captioning to spatial reasoning in 3D scenes, while demonstrating robust scaling properties as model size and training data increase. Our work provides a unified foundation for advancing 3D spatial intelligence in real-world applications.

Related papers

• SSR: Pushing the Limit of Spatial Intelligence with Structured Scene Reasoning [30.87517633729756]
  SSR is a framework designed for Structured Scene Reasoning.<n>It seamlessly integrates 2D and 3D representations via a lightweight alignment mechanism.<n>It achieves state-of-the-art performance on multiple spatial intelligence benchmarks.
  arXiv  Detail & Related papers  (2026-02-28T02:05:35Z)
• SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs [21.891285551179365]
  We introduce Spherical Coordinate-based Positional Embedding (SoPE)<n>Our method maps point-cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles.<n>This formulation preserves the inherent geometric structure of point-cloud data, enhances spatial awareness, and yields more consistent and expressive geometric representations for multimodal learning.
  arXiv  Detail & Related papers  (2026-02-26T07:42:15Z)
• SpatialMosaic: A Multiview VLM Dataset for Partial Visibility [25.874299974251965]
  We propose a scalable multi-view data generation and annotation pipeline that constructs realistic spatial reasoning QAs.<n>We introduce SpatialMosaic-Bench, a benchmark for evaluating multi-view spatial reasoning under realistic and challenging scenarios.<n>We also present SpatialMosaicVLM, a hybrid framework that integrates 3D reconstruction models as geometry encoders within Vision-Language Models.
  arXiv  Detail & Related papers  (2025-12-29T10:48:54Z)
• MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Experts [50.37005070020306]
  MoRE is a dense 3D visual foundation model based on a Mixture-of-Experts (MoE) architecture.<n>MoRE incorporates a confidence-based depth refinement module that stabilizes and refines geometric estimation.<n>It integrates dense semantic features with globally aligned 3D backbone representations for high-fidelity surface normal prediction.
  arXiv  Detail & Related papers  (2025-10-31T06:54:27Z)
• Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge [45.19482892758984]
  Affordance segmentation aims to parse 3D objects into functionally distinct parts, bridging recognition and interaction for applications in robotic manipulation, embodied AI, and AR.<n>We introduce Cross-Modal Affinity Transfer (CMAT), a pre-training strategy that aligns a 3D encoder with lifted 2D semantics and jointly optimize reconstruction, affinity, and diversity to yield semantically organized representations.<n>We further design the Cross-modal Affordance Transformer (CAST), which integrates multi-modal prompts with CMAT-pretrained features to generate precise, prompt-aware segmentation maps.
  arXiv  Detail & Related papers  (2025-10-09T15:01:26Z)
• LTM3D: Bridging Token Spaces for Conditional 3D Generation with Auto-Regressive Diffusion Framework [40.17218893870908]
  LTM3D is a Latent Token space Modeling framework for conditional 3D shape generation.<n>It integrates the strengths of diffusion and auto-regressive (AR) models.<n>LTM3D offers a generalizable framework for multi-modal, multi-representation 3D generation.
  arXiv  Detail & Related papers  (2025-05-30T06:08:45Z)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction [86.82819259860186]
  We introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning.<n>VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding.
  arXiv  Detail & Related papers  (2025-05-26T17:56:30Z)
• GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency [50.11520458252128]
  Existing 3D affordance learning methods struggle with generalization and robustness due to limited annotated data.<n>We propose GEAL, a novel framework designed to enhance the generalization and robustness of 3D affordance learning by leveraging large-scale pre-trained 2D models.<n>GEAL consistently outperforms existing methods across seen and novel object categories, as well as corrupted data.
  arXiv  Detail & Related papers  (2024-12-12T17:59:03Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.