DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving

• URL: http://arxiv.org/abs/2607.26165v1
• Date: Tue, 28 Jul 2026 18:16:00 GMT
• Title: DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving
• Abstract summary: Video panoptic segmentation (DVPS) unifies metric depth, semantic segmentation, and instance trajectories.<n>Existing approaches often rely on computationally expensive, multi-stage pipelines or offline tracking, rendering them unsuitable for real-time decision-making.<n>We propose DVPSFormer, a unified online architecture designed for efficient 4D scene understanding.
• Score: 89.17723223258513
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safe autonomous navigation requires a holistic understanding of dynamic environments, necessitating the simultaneous estimation of metric depth, semantic segmentation, and instance trajectories. While depth-aware video panoptic segmentation (DVPS) unifies these tasks, existing approaches often rely on computationally expensive, multi-stage pipelines or offline tracking, rendering them unsuitable for real-time decision-making. To address this, we propose DVPSFormer, a unified online architecture designed for efficient 4D scene understanding. Central to our approach is explicit scene discretization (ESD), a novel mechanism that leverages segmentation queries to represent foreground and background regions, enabling a discrete-to-continuous (D2C) depth head to decode metric depth in a single pass. This tightly couples semantic and geometric learning while significantly reducing latency. Furthermore, we propose an online majority voting (OMV) mechanism that exploits temporal consistency to refine classification during instance tracking. DVPSFormer establishes a new state-of-the-art on the Cityscapes-DVPS and SemKITTI-DVPS benchmarks, offering a streamlined solution for online robotic perception. Code and models are available at https://royyang0714.github.io/DVPSFormer.

Related papers

• DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale [63.05446464787182]
  We propose an alternative Vision-Geometry-Action paradigm that advocates dense 3D geometry as the critical cue for autonomous driving.<n>We introduce a streaming Driving Visual Geometry Transformer (DVGT-2), which processes inputs in an online manner and jointly outputs dense geometry and trajectory planning for the current frame.<n>Despite the faster speed, DVGT-2 achieves superior geometry reconstruction performance on various datasets.
  arXiv  Detail & Related papers  (2026-04-01T12:21:26Z)
• SPORTS: Simultaneous Panoptic Odometry, Rendering, Tracking and Segmentation for Urban Scenes Understanding [0.0]
  This paper proposes a novel framework, named SPORTS, for holistic scene understanding.<n>It integrates Video Panoptic (VPS), Visual Odometry (VO), and Scene Rendering tasks into an iterative and unified perspective.<n>Our attention-based feature fusion outperforms most existing state-of-the-art synthesis methods on the odometry, tracking, segmentation, and novel view tasks.
  arXiv  Detail & Related papers  (2025-10-14T17:28:19Z)
• MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion [17.50161162624179]
  Self-driving cars rely on reliable semantic environment perception for decision making. We propose MCDS-VSS, a structured filter model that learns in a self-supervised manner to estimate scene geometry and ego-motion of the camera. Our model parses automotive scenes into multiple interpretable representations such as scene geometry, ego-motion, and object motion.
  arXiv  Detail & Related papers  (2024-05-30T10:33:14Z)
• Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes [70.08318779492944]
  We are the first to harness vanishing point (VP) priors for more effective segmentation. Our novel, efficient network for VSS, named VPSeg, incorporates two modules that utilize exactly this pair of static and dynamic VP priors.
  arXiv  Detail & Related papers  (2024-01-27T01:01:58Z)
• Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation [76.68301884987348]
  We propose a simple yet effective approach for self-supervised video object segmentation (VOS) Our key insight is that the inherent structural dependencies present in DINO-pretrained Transformers can be leveraged to establish robust-temporal segmentation correspondences in videos. Our method demonstrates state-of-the-art performance across multiple unsupervised VOS benchmarks and excels in complex real-world multi-object video segmentation tasks.
  arXiv  Detail & Related papers  (2023-11-29T18:47:17Z)
• DVIS: Decoupled Video Instance Segmentation Framework [15.571072365208872]
  Video instance segmentation (VIS) is a critical task with diverse applications, including autonomous driving and video editing. Existing methods often underperform on complex and long videos in real world, primarily due to two factors. We propose a decoupling strategy for VIS by dividing it into three independent sub-tasks: segmentation, tracking, and refinement.
  arXiv  Detail & Related papers  (2023-06-06T05:24:15Z)
• A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design [77.34726150561087]
  We conduct a survey on the most relevant and recent advances in Deep Semantic in the context of vision for autonomous vehicles. Our main objective is to provide a comprehensive discussion on the main methods, advantages, limitations, results and challenges faced from each perspective.
  arXiv  Detail & Related papers  (2023-03-08T01:29:55Z)
• MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation [3.2489082010225494]
  We propose a novel solution with a multi-task network that performs monocular depth estimation and video panoptic segmentation. We introduce panoptic-guided depth losses and a novel panoptic masking scheme for moving objects to avoid corrupting the training signal.
  arXiv  Detail & Related papers  (2022-10-14T07:00:42Z)
• PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation [41.85216306978024]
  We propose a unified framework for depth-aware panoptic segmentation (DPS) We generate instance-specific kernels to predict depth and segmentation masks for each instance. We add additional instance-level depth cues to assist with supervising the depth learning via a new depth loss.
  arXiv  Detail & Related papers  (2022-06-01T13:00:49Z)
• Polyline Based Generative Navigable Space Segmentation for Autonomous Visual Navigation [57.3062528453841]
  We propose a representation-learning-based framework to enable robots to learn the navigable space segmentation in an unsupervised manner. We show that the proposed PSV-Nets can learn the visual navigable space with high accuracy, even without any single label.
  arXiv  Detail & Related papers  (2021-10-29T19:50:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.