Cross-Attentive Multiview Fusion of Vision-Language Embeddings

• URL: http://arxiv.org/abs/2604.12551v1
• Date: Tue, 14 Apr 2026 10:25:32 GMT
• Title: Cross-Attentive Multiview Fusion of Vision-Language Embeddings
• Abstract summary: We introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints.<n>We leverage multiview consistency as a self-supervision signal for this fusion.<n>Our Cross-Attentive Multiview Fusion consistently outperforms naive averaging or single-view descriptor selection.
• Score: 28.984855441235776
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language models have been key to the development of open-vocabulary 2D semantic segmentation. Lifting these models from 2D images to 3D scenes, however, remains a challenging problem. Existing approaches typically back-project and average 2D descriptors across views, or heuristically select a single representative one, often resulting in suboptimal 3D representations. In this work, we introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints and fuses them into a unified per-3D-instance embedding. As a second contribution, we leverage multiview consistency as a self-supervision signal for this fusion, which significantly improves performance when added to a standard supervised target-class loss. Our Cross-Attentive Multiview Fusion, which we denote with its acronym CAMFusion, not only consistently outperforms naive averaging or single-view descriptor selection, but also achieves state-of-the-art results on 3D semantic and instance classification benchmarks, including zero-shot evaluations on out-of-domain datasets.

Related papers

• Inclusive Interactive Collisions for Multi-View Consistent Compositional 3D Generation [25.750694245523217]
  We propose I2C-3D, a novel optimization-based method to generate multi-view consistent compositional 3D assets with reasonable interactions.<n>Specifically, we propose an Inclusive Interactive Collisions strategy to guide Gaussian primitives appearing in reasonable interaction regions naturally.<n>To enhance multi-view consistency, Multi-View Adaptive Score Distillation Sampling is devised to distill multi-view consistency prior and layout prior from pre-trained diffusion model.
  arXiv  Detail & Related papers  (2026-06-23T06:48:22Z)
• DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation [58.47973015036709]
  DisPOSE is a self-supervised framework that approximates the inherently discrete multi-view person-assignment problem.<n>By employing differentiable Sinkhorn projections during denoising, our model learns to guide solutions toward valid and feasible assignments.<n>The proposed approach outperforms current state-of-the-art self-supervised methods on standard datasets.
  arXiv  Detail & Related papers  (2026-06-05T16:14:39Z)
• Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation [9.296275675671636]
  We propose a hierarchical approach for input context collection at both the view and token levels.<n>Specifically, we combine pixel features with camera parameters and assess view importance based on both semantic content and geometric position.<n>We evaluate our framework on three widely used benchmarks, demonstrating significant improvements over existing tuning-free methods and performance comparable to training-based approaches.
  arXiv  Detail & Related papers  (2026-06-02T03:38:51Z)
• Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding [34.1504914582344]
  3D Visual Grounding aims to localize objects in 3D scenes via natural language descriptions.<n>We propose "Think, Act, Build (TAB)", a generative 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams.<n>To overcome the multi-view coverage deficit caused by strict VLM semantic tracking, we introduce the Semantic-Anchored Geometric Expansion.
  arXiv  Detail & Related papers  (2026-04-01T06:12:16Z)
• Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation [91.2768117730855]
  Generalised 3D Referring Expression (3D-GRES) localizes objects in 3D scenes based on natural language, even when descriptions match multiple or zero targets.<n>Existing methods rely solely on sparse point clouds, lacking rich visual semantics for fine-grained descriptions.<n>We propose HCF-RES, a multi-modal framework with two key innovations.
  arXiv  Detail & Related papers  (2026-03-06T13:09:29Z)
• OpenInsGaussian: Open-vocabulary Instance Gaussian Segmentation with Context-aware Cross-view Fusion [89.98812408058336]
  We introduce textbfOpenInsGaussian, an textbfOpen-vocabulary textbfInstance textbfGaussian segmentation framework with Context-aware Cross-view Fusion.<n>OpenInsGaussian achieves state-of-the-art results in open-vocabulary 3D Gaussian segmentation, outperforming existing baselines by a large margin.
  arXiv  Detail & Related papers  (2025-10-21T03:24:12Z)
• Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusion [73.11061598576798]
  Camera-based occupancy prediction is a mainstream approach for 3D perception in autonomous driving.<n>textbfCIGOcc is a two-stage occupancy prediction framework based on multi-level representation fusion.<n>textbfCIGOcc extracts segmentation, graphics, and depth features from an input image and introduces a deformable multi-level fusion mechanism.
  arXiv  Detail & Related papers  (2025-10-15T06:37:33Z)
• PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum [20.206273757144547]
  PGOV3D is a novel framework that introduces a Partial-to-Global curriculum for improving open-vocabulary 3D semantic segmentation.<n>We pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry.<n>In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex.
  arXiv  Detail & Related papers  (2025-06-30T08:13:07Z)
• DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields [68.94868475824575]
  This paper introduces a novel approach capable of generating infinite, high-quality 3D-consistent 2D annotations alongside 3D point cloud segmentations. We leverage the strong semantic prior within a 3D generative model to train a semantic decoder. Once trained, the decoder efficiently generalizes across the latent space, enabling the generation of infinite data.
  arXiv  Detail & Related papers  (2023-11-18T21:58:28Z)
• Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation [72.94143731623117]
  Existing methods simply align 3D representations with single-view 2D images and coarse-grained parent category text. Insufficient synergy neglects the idea that a robust 3D representation should align with the joint vision-language space. We propose a multi-view joint modality modeling approach, termed JM3D, to obtain a unified representation for point cloud, text, and image.
  arXiv  Detail & Related papers  (2023-08-06T01:11:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.