MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes

• URL: http://arxiv.org/abs/2606.31533v1
• Date: Tue, 30 Jun 2026 11:46:25 GMT
• Title: MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
• Abstract summary: MV-GEL is a framework for localizing fine-grained geometric entities on polygon meshes from natural language queries.<n>GELviews is a prompt-conditioned ranking module that prioritizes viewpoints based on language prompted observability of geometric CAD entities.<n>Experiments show up to a 1.7X improvement in face-level IoU and over 4.5X gains in edge-level F1 compared to vanilla baselines.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Identifying and grounding precise geometric entities, such as edges, planar regions, and curved surfaces within 3D objects, is foundational to computer-aided design (CAD), robotic manipulation, and scientific simulation. Although modern Vision Language Models (VLMs) have advanced referring segmentation (RIS) in the image domain, extending such language-driven localization to structured 3D geometry is substantially harder. The 3D object appearance is highly sensitive to viewpoints; a single perspective may render a target entity clearly observable, while another may suffer from severe occlusion or foreshortening. In this work, we attempt to solve these challenges with MV-GEL (Multi-View Geometric Entity Localization), a framework for localizing fine-grained geometric entities on polygon meshes from natural language queries. Our key insight is that reliable CAD entity (i.e., faces, edges or solids) localization depends on selecting views that make the queried entity maximally interpretable. We introduce GELviews, a prompt-conditioned ranking module that prioritizes viewpoints based on language prompted observability of geometric CAD entities. Selected views are processed by a VLM-based reasoning segmentation backbone, and predicted masks are lifted to the corresponding meshes via geometry-aware ray casting. Our framework is completely CAD agnostic and relies only on 3D meshes. Experiments show up to a 1.7X improvement in face-level IoU and over 4.5X gains in edge-level F1 compared to vanilla baselines, substantially outperforming CLIP-based and random view sampling, particularly for thin and view-sensitive structures.The dataset, code and trained checkpoints are available at https://github.com/kbali1297/MV-GEL.

Related papers

• G$^2$TAM: Geometry Grounded Track Anything Model [49.05553245076823]
  We propose a unified framework for promptable instance tracking in 3D using only unordered RGB images or videos.<n>G$2$TAM employs spatially aligned geometric representations as implicit memory, ensuring stable instance identity and localization across frames and views.<n>At its core is a cross-modal spatial encoder that integrates visual and textual prompts into a shared geometric space, enabling end-to-end spatial reconstruction and instance-consistent mask prediction.
  arXiv  Detail & Related papers  (2026-07-04T09:32:13Z)
• SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning [22.547972947051765]
  Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems.<n>We propose SpatialStack, a hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy.<n>Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-28T22:49:40Z)
• Make Geometry Matter for Spatial Reasoning [62.61667611352403]
  Vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited.<n>Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs.<n>We propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens.
  arXiv  Detail & Related papers  (2026-03-27T17:45:12Z)
• Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps [77.63233146945718]
  We introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images.<n>Our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-24T10:05:32Z)
• MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation [59.75554954111619]
  We introduce Multi-view 3D Referring Expression (MV-3DRES), where the model must recover scene structure and segment the referred object directly from sparse multi-view images.<n>We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning.<n>Experiments show that MVGGT establishes the first strong baseline and achieves both high accuracy and fast inference, outperforming existing alternatives.
  arXiv  Detail & Related papers  (2026-01-11T11:44:07Z)
• 3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation [17.294440057314812]
  Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks.<n>We propose Geometric Distillation, a framework that injects human-inspired geometric cues into pretrained VLMs.<n>Our method shapes representations to be geometry-aware while remaining compatible with natural image-text inputs.
  arXiv  Detail & Related papers  (2025-06-11T15:56:59Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• 3D Part Segmentation via Geometric Aggregation of 2D Visual Features [57.20161517451834]
  Supervised 3D part segmentation models are tailored for a fixed set of objects and parts, limiting their transferability to open-set, real-world scenarios.<n>Recent works have explored vision-language models (VLMs) as a promising alternative, using multi-view rendering and textual prompting to identify object parts.<n>To address these limitations, we propose COPS, a COmprehensive model for Parts that blends semantics extracted from visual concepts and 3D geometry to effectively identify object parts.
  arXiv  Detail & Related papers  (2024-12-05T15:27:58Z)
• Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding [11.416392706435415]
  Zero-shot 3D point cloud understanding can be achieved via 2D Vision-Language Models (VLMs) Existing strategies directly map Vision-Language Models from 2D pixels of rendered or captured views to 3D points, overlooking the inherent and expressible point cloud geometric structure. We introduce the first training-free aggregation technique that leverages the point cloud's 3D geometric structure to improve the quality of the transferred Vision-Language Models.
  arXiv  Detail & Related papers  (2023-12-04T12:30:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.