MT-Depth: Multi-task Instance feature analysis for the Depth Completion

• URL: http://arxiv.org/abs/2512.04734v2
• Date: Tue, 09 Dec 2025 08:31:36 GMT
• Title: MT-Depth: Multi-task Instance feature analysis for the Depth Completion
• Authors: Abdul Haseeb Nizamani, Dandi Zhou, Xinhai Sun,
• Abstract summary: We introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions.<n>Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head.<n>We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower Root Mean Squared Error (RMSE) compared to both a U-Net-only baseline and previous semantic-guided methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Depth completion plays a vital role in 3D perception systems, especially in scenarios where sparse depth data must be densified for tasks such as autonomous driving, robotics, and augmented reality. While many existing approaches rely on semantic segmentation to guide depth completion, they often overlook the benefits of object-level understanding. In this work, we introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions. Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head. The instance segmentation branch generates per-image foreground masks that guide the depth branch via cross-attention, allowing the network to focus on object-centric regions during refinement. We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower Root Mean Squared Error (RMSE) compared to both a U-Net-only baseline and previous semantic-guided methods, while maintaining competitive Mean Absolute Error (MAE). Qualitative and quantitative results demonstrate that the proposed model effectively enhances depth accuracy near object boundaries, occlusions, and thin structures. Our findings suggest that incorporating instance-aware cues offers a promising direction for improving depth completion without relying on dense semantic labels.

Related papers

• Towards 3D Object-Centric Feature Learning for Semantic Scene Completion [18.41627244498394]
  Vision-based 3D Semantic Scene Completion (SSC) has received growing attention due to its potential in autonomous driving.<n>We propose Ocean, an object-centric prediction framework that decomposes the scene into individual object instances.<n>We show that Ocean achieves state-of-the-art performance, with mIoU scores of 17.40 and 20.28, respectively.
  arXiv  Detail & Related papers  (2025-11-17T06:28:26Z)
• Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning [3.4174356345935393]
  We propose MoDOT, a novel method that jointly estimates depth and OBs from a single image.<n>MoDOT incorporates a new module, CASM, which combines cross-attention and multi-scale strip convolutions to leverage mid-level OB features.<n>Experiments demonstrate the mutual benefits of jointly estimating depth and OBs, and validate the effectiveness of MoDOT's design.
  arXiv  Detail & Related papers  (2025-05-27T14:15:19Z)
• Depth Anything with Any Prior [64.39991799606146]
  Prior Depth Anything is a framework that combines incomplete but precise metric information in depth measurement with relative but complete geometric structures in depth prediction.<n>We develop a conditioned monocular depth estimation (MDE) model to refine the inherent noise of depth priors.<n>Our model showcases impressive zero-shot generalization across depth completion, super-resolution, and inpainting over 7 real-world datasets.
  arXiv  Detail & Related papers  (2025-05-15T17:59:50Z)
• MICDrop: Masking Image and Depth Features via Complementary Dropout for Domain-Adaptive Semantic Segmentation [155.0797148367653]
  Unsupervised Domain Adaptation (UDA) is the task of bridging the domain gap between a labeled source domain and an unlabeled target domain. We propose to leverage geometric information, i.e., depth predictions, as depth discontinuities often coincide with segmentation boundaries. We show that our method can be plugged into various recent UDA methods and consistently improve results across standard UDA benchmarks.
  arXiv  Detail & Related papers  (2024-08-29T12:15:10Z)
• Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation [34.786268652516355]
  Scene segmentation via unsupervised domain adaptation (UDA) enables the transfer of knowledge acquired from source synthetic data to real-world target data. We propose a depth-aware framework to explicitly leverage depth estimation to mix the categories and facilitate the two complementary tasks, i.e., segmentation and depth learning. In particular, the framework contains a Depth-guided Contextual Filter (DCF) forndata augmentation and a cross-task encoder for contextual learning.
  arXiv  Detail & Related papers  (2023-11-21T15:39:21Z)
• SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion [18.19171031755595]
  We propose a new end-to-end model for performing semantic segmentation and depth completion jointly. Our approach relies on RGB and sparse depth as inputs to our model and produces a dense depth map and the corresponding semantic segmentation image. Experiments done on Virtual KITTI 2 dataset, demonstrate and provide further evidence, that combining both tasks, semantic segmentation and depth completion, in a multi-task network can effectively improve the performance of each task.
  arXiv  Detail & Related papers  (2022-09-01T11:52:11Z)
• X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation [69.9604394044652]
  We propose a novel method to improve the self-supervised training of monocular depth via cross-task knowledge distillation. During training, we utilize a pretrained semantic segmentation teacher network and transfer its semantic knowledge to the depth network. We extensively evaluate the efficacy of our proposed approach on the KITTI benchmark and compare it with the latest state of the art.
  arXiv  Detail & Related papers  (2021-10-24T19:47:14Z)
• Self-Guided Instance-Aware Network for Depth Completion and Enhancement [6.319531161477912]
  Existing methods directly interpolate the missing depth measurements based on pixel-wise image content and the corresponding neighboring depth values. We propose a novel self-guided instance-aware network (SG-IANet) that utilize self-guided mechanism to extract instance-level features that is needed for depth restoration.
  arXiv  Detail & Related papers  (2021-05-25T19:41:38Z)
• Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion [56.85837052421469]
  Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion.
  arXiv  Detail & Related papers  (2021-03-30T21:22:26Z)
• SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images [94.36401543589523]
  We introduce the concept of semantic objectness to exploit the geometric relationship of these two tasks. We then propose a Semantic Object and Depth Estimation Network (SOSD-Net) based on the objectness assumption. To the best of our knowledge, SOSD-Net is the first network that exploits the geometry constraint for simultaneous monocular depth estimation and semantic segmentation.
  arXiv  Detail & Related papers  (2021-01-19T02:41:03Z)
• Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation [90.87105131054419]
  We present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled image sequences. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains.
  arXiv  Detail & Related papers  (2020-12-19T21:18:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.