An overview of 3D Vision-Language Models

• URL: http://arxiv.org/abs/2609.05583v1
• Date: Fri, 04 Sep 2026 15:43:51 GMT
• Title: An overview of 3D Vision-Language Models
• Abstract summary: 3D Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings.<n>Traditional 3D deep-learning models are typically trained for specific tasks, such as classification, segmentation, or detection.<n> Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations.
• Score: 3.6170225940073006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.

Related papers

• Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment [50.02189698630855]
  We propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality.<n>Our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T17:50:47Z)
• A Review of 3D Object Detection with Vision-Language Models [0.31457219084519]
  We provide the first systematic analysis dedicated to 3D object detection with vision-language models.<n>Traditional approaches using point clouds and voxel grids are compared to modern vision-language frameworks like CLIP and 3D LLMs.<n>We highlight current challenges, such as limited 3D-language datasets and computational demands.
  arXiv  Detail & Related papers  (2025-04-25T23:27:26Z)
• 3D Scene Graph Guided Vision-Language Pre-training [11.131667398927394]
  3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions.<n>Existing approaches typically follow task-specific, highly specialized paradigms.<n>This paper proposes a 3D scene graph-guided vision-language pre-training framework.
  arXiv  Detail & Related papers  (2024-11-27T16:10:44Z)
• g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks [62.74304008688472]
  Generalizable 3D-Language Feature Fields (g3D-LF) is a 3D representation model pre-trained on large-scale 3D-language dataset for embodied tasks.
  arXiv  Detail & Related papers  (2024-11-26T01:54:52Z)
• Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Image [70.02187124865627]
  Open-vocabulary 3D object detection (OV-3DDet) aims to localize and recognize both seen and previously unseen object categories within any new 3D scene. We leverage a vision foundation model to provide image-wise guidance for discovering novel classes in 3D scenes. We demonstrate significant improvements in accuracy and generalization, highlighting the potential of foundation models in advancing open-vocabulary 3D object detection.
  arXiv  Detail & Related papers  (2024-07-07T04:50:04Z)
• VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision Understanding [47.58359136198136]
  VisionGPT-3D provides a versatile multimodal framework building upon the strengths of multimodal foundation models. It seamlessly integrates various SOTA vision models and brings the automation in the selection of SOTA vision models. It identifies the suitable 3D mesh creation algorithms corresponding to 2D depth maps analysis, generates optimal results based on diverse multimodal inputs.
  arXiv  Detail & Related papers  (2024-03-14T16:13:00Z)
• Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment [24.63428589906294]
  We propose a weakly supervised approach for 3D visual grounding based on Visual Linguistic Alignment.<n>Our 3D-VLA exploits the superior ability of current large-scale vision-language models on aligning the semantics between texts and 2D images.<n>During the inference stage, the learned text-3D correspondence will help us ground the text queries to the 3D target objects even without 2D images.
  arXiv  Detail & Related papers  (2023-12-15T09:08:14Z)
• JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues [68.76032126906743]
  We introduce JM3D, a comprehensive approach integrating point cloud, text, and image. Key contributions include the Structured Multimodal Organizer (SMO), enriching vision-language representation with multiple views and hierarchical text. Our advanced model, JM3D-LLM, marries 3D representation with large language models via efficient fine-tuning.
  arXiv  Detail & Related papers  (2023-10-14T06:13:20Z)
• Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes [68.61199623705096]
  Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore. We propose a novel 3D pre-training Vision-Language method, namely Multi-CLIP, that enables a model to learn language-grounded and transferable 3D scene point cloud representations.
  arXiv  Detail & Related papers  (2023-06-04T11:08:53Z)
• CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes [68.61199623705096]
  We design a novel 3D pre-training Vision-Language method that helps a model learn semantically meaningful and transferable 3D scene point cloud representations. We inject the representational power of the popular CLIP model into our 3D encoder by aligning the encoded 3D scene features with the corresponding 2D image and text embeddings. We evaluate our model's 3D world reasoning capability on the downstream task of 3D Visual Question Answering.
  arXiv  Detail & Related papers  (2023-04-12T16:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.