3D Segmentation Using Viewpoint-Dependent Spatial Relationships

• URL: http://arxiv.org/abs/2605.15708v1
• Date: Fri, 15 May 2026 07:58:44 GMT
• Title: 3D Segmentation Using Viewpoint-Dependent Spatial Relationships
• Abstract summary: We introduce a viewpoint-aware 3D segmentation dataset containing 220k benchmark samples.<n>In this dataset, target objects can only be identified through observer-centric spatial relations.<n>We introduce a viewpoint representation that encodes camera poses and conditions the model on the observation viewpoint.
• Score: 55.198821645924234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in 3D datasets and multimodal models have greatly improved natural language 3D scene understanding. However, most 3D referring segmentation methods do not explicitly represent the observer viewpoint, making spatial relations such as "left," "right," "front," and "behind" ambiguous and difficult to evaluate. We introduce a viewpoint-aware 3D referring segmentation dataset containing 220k benchmark samples, and scalable to tens of millions of viewpoint-conditioned samples through dense viewpoint sampling. In this dataset, target objects can only be identified through observer-centric spatial relations, making viewpoint-conditioned grounding necessary. We construct the benchmark by leveraging camera poses to automatically annotate observer-centric relations (left/right, front/behind) together with viewpoint-independent relations (above/under). Using this benchmark, we evaluate several existing 3D large multimodal models in a zero-shot setting and find that current models struggle with viewpoint-dependent spatial instructions. We further study how explicit viewpoint information can be incorporated into 3D large multimodal models. We introduce a viewpoint representation that encodes camera poses and conditions the model on the observation viewpoint, improving segmentation accuracy on viewpoint-dependent relations and increasing mIoU from 0.30 to 0.47 compared to a model without viewpoint conditioning. The dataset, code, and trained models will be made publicly available upon acceptance.

Related papers

• G$^2$TAM: Geometry Grounded Track Anything Model [49.05553245076823]
  We propose a unified framework for promptable instance tracking in 3D using only unordered RGB images or videos.<n>G$2$TAM employs spatially aligned geometric representations as implicit memory, ensuring stable instance identity and localization across frames and views.<n>At its core is a cross-modal spatial encoder that integrates visual and textual prompts into a shared geometric space, enabling end-to-end spatial reconstruction and instance-consistent mask prediction.
  arXiv  Detail & Related papers  (2026-07-04T09:32:13Z)
• MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement [2.1383141762884152]
  We propose MoViD, a viewpoint-invariant 3D human pose estimation framework that disentangles viewpoint information from motion features.<n>MoViD reduces pose estimation error by over 24.2% compared to state-of-the-art methods.<n>MoViD achieves real-time inference at 15 FPS on NVIDIA edge devices.
  arXiv  Detail & Related papers  (2026-03-29T09:08:57Z)
• Depth Any Panoramas: A Foundation Model for Panoramic Depth Estimation [68.95366581365829]
  We present a panoramic metric depth foundation model that generalizes across diverse scene distances.<n>We collect a large-scale dataset by combining public datasets, high-quality synthetic data from our UE5 simulator and text-to-image models, and real panoramic images from the web.
  arXiv  Detail & Related papers  (2025-12-18T18:59:29Z)
• Can Generative Video Models Help Pose Estimation? [42.10672365565019]
  Pairwise pose estimation from images with little or no overlap is an open challenge in computer vision.<n>Inspired by the human ability to infer spatial relationships from diverse scenes, we propose a novel approach, InterPose.<n>We propose a video model to hallucinate intermediate frames between two input images, effectively creating a dense, visual transition.
  arXiv  Detail & Related papers  (2024-12-20T18:58:24Z)
• View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields [52.08335264414515]
  We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene. Our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency.
  arXiv  Detail & Related papers  (2024-05-30T04:14:58Z)
• S$^2$Contact: Graph-based Network for 3D Hand-Object Contact Estimation with Semi-Supervised Learning [70.72037296392642]
  We propose a novel semi-supervised framework that allows us to learn contact from monocular images. Specifically, we leverage visual and geometric consistency constraints in large-scale datasets for generating pseudo-labels. We show benefits from using a contact map that rules hand-object interactions to produce more accurate reconstructions.
  arXiv  Detail & Related papers  (2022-08-01T14:05:23Z)
• Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
  A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
  arXiv  Detail & Related papers  (2021-03-12T15:30:02Z)
• SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection [9.924083358178239]
  We propose two variants of self-attention for contextual modeling in 3D object detection. We first incorporate the pairwise self-attention mechanism into the current state-of-the-art BEV, voxel and point-based detectors. Next, we propose a self-attention variant that samples a subset of the most representative features by learning deformations over randomly sampled locations.
  arXiv  Detail & Related papers  (2021-01-07T18:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.