Joint 2D-3D Segmentation and Association in Street-level Imaging

• URL: http://arxiv.org/abs/2605.26725v1
• Date: Tue, 26 May 2026 09:04:51 GMT
• Title: Joint 2D-3D Segmentation and Association in Street-level Imaging
• Abstract summary: This work presents a unified framework for joint 2D-3D segmentation and association.<n>A 3D-driven association mechanism replaces traditional 2D multi-object tracking.<n> Experiments demonstrate substantially improved coverage of ground-truth sequences and more robust identity retention.
• Score: 18.70317918218011
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate interpretation of street-level imagery is essential for large-scale urban mapping and the creation of Spatial Digital Twin (SDT) environments. This work presents a unified framework for joint 2D-3D segmentation and association that integrates visual semantics with multi-view geometric reasoning. Unlike conventional approaches that rely heavily on sequential frames for temporal tracking, our method leverages zero-shot detection and segmentation together with structure-from-motion reconstruction to establish stable cross-view correspondences. A 3D-driven association mechanism replaces traditional 2D multi-object tracking, using geometric consistency to guide identity preservation across wide-baseline viewpoints and varying imaging conditions. By combining 2D texture cues with global 3D context, the proposed pipeline is well-suited for scalable street-level processing and can be used for a variety of object types. Experiments demonstrate substantially improved coverage of ground-truth sequences and more robust identity retention compared to state-of-the-art 2D-only tracking methods, achieving a 22% performance gain in challenging urban scenarios.

Related papers

• Reliability-Aware 3D Geometric Injection for Universal Person Re-identification [47.874819213762926]
  We propose a Universal Monocular 3D-Enhanced ReID framework driven by a Consistency-Aware Reliability Gate and Dual-Stream Residual Fusion.<n>Our method improves challenging, structure-sensitive scenarios while preserving competitive performance on clean domains.
  arXiv  Detail & Related papers  (2026-07-21T08:54:25Z)
• VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction [59.303842406260124]
  VGGT-Edit is a feed-forward framework for text-conditioned native 3D scene editing.<n>We introduce depth-synchronized text injection to align semantic guidance with the backbone's spatial poses.<n> VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed.
  arXiv  Detail & Related papers  (2026-05-14T17:59:04Z)
• PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views [41.6619757077912]
  We propose Pose-Aware 3D Shape Retrieval (PASR), a framework that formulates retrieval as a feature-level analysis-by-language problem.<n>By aligning pose-conditioned 3D projections with 2D feature maps, our method bridges the gap between real-world images and synthetic meshes.<n>PASR substantially outperforms existing methods on both clean and occluded 3D shape retrieval datasets by a wide margin.
  arXiv  Detail & Related papers  (2026-04-24T15:32:03Z)
• Hierarchical Image-Guided 3D Point Cloud Segmentation in Industrial Scenes via Multi-View Bayesian Fusion [4.679314646805623]
  3D segmentation is critical for understanding complex scenes with dense layouts and multi-scale objects.<n>Existing 3D point-based methods require costly annotations, while image-guided methods often suffer from semantic inconsistencies across views.<n>We propose a hierarchical image-guided 3D segmentation framework that progressively refines segmentation from instance-level to part-level.
  arXiv  Detail & Related papers  (2025-12-07T15:15:52Z)
• HD$^2$-SSC: High-Dimension High-Density Semantic Scene Completion for Autonomous Driving [52.959716866316604]
  Camera-based 3D semantic scene completion (SSC) plays a crucial role in autonomous driving.<n>Existing SSC methods suffer from the inherent input-output dimension gap and annotation-reality density gap.<n>We propose a corresponding High- Dimension High-Density Semantic Scene Completion framework with expanded pixel semantics and refined voxel occupancies.
  arXiv  Detail & Related papers  (2025-11-11T07:24:35Z)
• Self-Supervised Cross-Modal Learning for Image-to-Point Cloud Registration [22.360139236823155]
  CrossI2P is a self-supervised framework that unifies cross-modal learning and two-stage registration in a single end-to-end pipeline.<n>We show that CrossI2P outperforms state-of-the-art methods by 23.7% on the KITTI Odometry benchmark and by 37.9% on nuScenes.
  arXiv  Detail & Related papers  (2025-09-19T11:29:22Z)
• CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction [25.468907201804093]
  Large Reconstruction Models (LRMs) have shown great promise in leveraging multi-view images generated by 2D diffusion models to extract 3D content.<n>However, 2D diffusion models often struggle to produce dense images with strong multi-view consistency.<n>We present CDI3D, a feed-forward framework designed for efficient, high-quality image-to-3D generation with view.
  arXiv  Detail & Related papers  (2025-03-11T03:08:43Z)
• Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects [67.96148051569993]
  This paper introduces a new neuro-implicit method that can reconstruct the geometry and appearance of two objects undergoing close interactions while disjoining both in 3D.<n>The framework is end-to-end trainable and supervised using a novel alpha-blending regularisation.<n>We introduce a new dataset consisting of close interactions between a human and an object and also evaluate on two scenes of humans performing martial arts.
  arXiv  Detail & Related papers  (2025-02-19T18:59:56Z)
• GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency [50.11520458252128]
  Existing 3D affordance learning methods struggle with generalization and robustness due to limited annotated data.<n>We propose GEAL, a novel framework designed to enhance the generalization and robustness of 3D affordance learning by leveraging large-scale pre-trained 2D models.<n>GEAL consistently outperforms existing methods across seen and novel object categories, as well as corrupted data.
  arXiv  Detail & Related papers  (2024-12-12T17:59:03Z)
• T-3DGS: Removing Transient Objects for 3D Scene Reconstruction [83.05271859398779]
  Transient objects in video sequences can significantly degrade the quality of 3D scene reconstructions.<n>We propose T-3DGS, a novel framework that robustly filters out transient distractors during 3D reconstruction using Gaussian Splatting.
  arXiv  Detail & Related papers  (2024-11-29T07:45:24Z)
• ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction [62.599588577671796]
  We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality.
  arXiv  Detail & Related papers  (2023-11-29T20:30:18Z)
• JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery [84.67823511418334]
  This paper presents 3D JOint contrastive learning with TRansformers framework for handling occluded 3D human mesh recovery. Our method includes an encoder-decoder transformer architecture to fuse 2D and 3D representations for achieving 2D$&$3D aligned results.
  arXiv  Detail & Related papers  (2023-07-31T02:58:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.