Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

• URL: http://arxiv.org/abs/2005.14553v2
• Date: Thu, 7 Jan 2021 15:26:43 GMT
• Title: Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation
• Authors: Christos Sakaridis, Dengxin Dai, Luc Van Gool
• Abstract summary: We develop a curriculum framework to adapt semantic segmentation models from day to night without using nighttime annotations. We also design a new evaluation framework to address the substantial uncertainty of semantics in nighttime images.
• Score: 107.33492779588641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the problem of semantic nighttime image segmentation and improve the state-of-the-art, by adapting daytime models to nighttime without using nighttime annotations. Moreover, we design a new evaluation framework to address the substantial uncertainty of semantics in nighttime images. Our central contributions are: 1) a curriculum framework to gradually adapt semantic segmentation models from day to night through progressively darker times of day, exploiting cross-time-of-day correspondences between daytime images from a reference map and dark images to guide the label inference in the dark domains; 2) a novel uncertainty-aware annotation and evaluation framework and metric for semantic segmentation, including image regions beyond human recognition capability in the evaluation in a principled fashion; 3) the Dark Zurich dataset, comprising 2416 unlabeled nighttime and 2920 unlabeled twilight images with correspondences to their daytime counterparts plus a set of 201 nighttime images with fine pixel-level annotations created with our protocol, which serves as a first benchmark for our novel evaluation. Experiments show that our map-guided curriculum adaptation significantly outperforms state-of-the-art methods on nighttime sets both for standard metrics and our uncertainty-aware metric. Furthermore, our uncertainty-aware evaluation reveals that selective invalidation of predictions can improve results on data with ambiguous content such as our benchmark and profit safety-oriented applications involving invalid inputs.

Related papers

• Exploring Reliable Matching with Phase Enhancement for Night-time Semantic Segmentation [58.180226179087086]
  We propose a novel end-to-end optimized approach, named NightFormer, tailored for night-time semantic segmentation. Specifically, we design a pixel-level texture enhancement module to acquire texture-aware features hierarchically with phase enhancement and amplified attention. Our proposed method performs favorably against state-of-the-art night-time semantic segmentation methods.
  arXiv  Detail & Related papers  (2024-08-25T13:59:31Z)
• Self-Supervised Monocular Depth Estimation in the Dark: Towards Data Distribution Compensation [24.382795861986803]
  Using night images for self-supervision is unreliable because the photometric consistency assumption is usually violated in the videos taken under complex lighting conditions. We propose a self-supervised nighttime monocular depth estimation method that does not use any night images during training.
  arXiv  Detail & Related papers  (2024-04-22T03:39:03Z)
• Introspective Deep Metric Learning [91.47907685364036]
  We propose an introspective deep metric learning framework for uncertainty-aware comparisons of images. The proposed IDML framework improves the performance of deep metric learning through uncertainty modeling.
  arXiv  Detail & Related papers  (2023-09-11T16:21:13Z)
• Similarity Min-Max: Zero-Shot Day-Night Domain Adaptation [52.923298434948606]
  Low-light conditions not only hamper human visual experience but also degrade the model's performance on downstream vision tasks. This paper challenges a more complicated scenario with border applicability, i.e., zero-shot day-night domain adaptation. We propose a similarity min-max paradigm that considers them under a unified framework.
  arXiv  Detail & Related papers  (2023-07-17T18:50:15Z)
• From colouring-in to pointillism: revisiting semantic segmentation supervision [48.637031591058175]
  We propose a pointillist approach for semantic segmentation annotation, where only point-wise yes/no questions are answered. We collected and released 22.6M point labels over 4,171 classes on the Open Images dataset.
  arXiv  Detail & Related papers  (2022-10-25T16:42:03Z)
• Introspective Deep Metric Learning for Image Retrieval [80.29866561553483]
  We argue that a good similarity model should consider the semantic discrepancies with caution to better deal with ambiguous images for more robust training. We propose to represent an image using not only a semantic embedding but also an accompanying uncertainty embedding, which describes the semantic characteristics and ambiguity of an image, respectively. The proposed IDML framework improves the performance of deep metric learning through uncertainty modeling and attains state-of-the-art results on the widely used CUB-200-2011, Cars196, and Stanford Online Products datasets.
  arXiv  Detail & Related papers  (2022-05-09T17:51:44Z)
• Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation [17.874336775904272]
  We propose a novel domain adaptation framework via cross-domain correlation distillation, called CCDistill. We extract the content and style knowledge contained in features, calculate the degree of inherent or illumination difference between two images. Experiments on Dark Zurich and ACDC demonstrate that CCDistill achieves the state-of-the-art performance for nighttime semantic segmentation.
  arXiv  Detail & Related papers  (2022-05-02T12:42:04Z)
• Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation [17.066753214406525]
  We propose a domain-separated network for self-supervised depth estimation of all-day images. Our approach achieves state-of-the-art depth estimation results for all-day images on the challenging Oxford RobotCar dataset.
  arXiv  Detail & Related papers  (2021-08-17T13:52:19Z)
• DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation [18.43890050736093]
  We propose a novel domain adaptation network (DANNet) for nighttime semantic segmentation. It employs an adversarial training with a labeled daytime dataset and an unlabeled dataset that contains coarsely aligned day-night image pairs. Our method achieves state-of-the-art performance for nighttime semantic segmentation.
  arXiv  Detail & Related papers  (2021-04-22T02:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.