HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images

• URL: http://arxiv.org/abs/2003.04645v1
• Date: Tue, 10 Mar 2020 11:36:42 GMT
• Title: HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images
• Authors: Johan Vertens, Jannik Z\"urn, Wolfram Burgard
• Abstract summary: Real-world driving scenarios entail adverse environmental conditions such as nighttime illumination or glare. We propose a multimodal semantic segmentation model that can be applied during daytime and nighttime. Besides RGB images, we leverage thermal images, making our network significantly more robust.
• Score: 26.749261270690425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The majority of learning-based semantic segmentation methods are optimized for daytime scenarios and favorable lighting conditions. Real-world driving scenarios, however, entail adverse environmental conditions such as nighttime illumination or glare which remain a challenge for existing approaches. In this work, we propose a multimodal semantic segmentation model that can be applied during daytime and nighttime. To this end, besides RGB images, we leverage thermal images, making our network significantly more robust. We avoid the expensive annotation of nighttime images by leveraging an existing daytime RGB-dataset and propose a teacher-student training approach that transfers the dataset's knowledge to the nighttime domain. We further employ a domain adaptation method to align the learned feature spaces across the domains and propose a novel two-stage training scheme. Furthermore, due to a lack of thermal data for autonomous driving, we present a new dataset comprising over 20,000 time-synchronized and aligned RGB-thermal image pairs. In this context, we also present a novel target-less calibration method that allows for automatic robust extrinsic and intrinsic thermal camera calibration. Among others, we employ our new dataset to show state-of-the-art results for nighttime semantic segmentation.

Related papers

• Exploring Reliable Matching with Phase Enhancement for Night-time Semantic Segmentation [58.180226179087086]
  We propose a novel end-to-end optimized approach, named NightFormer, tailored for night-time semantic segmentation. Specifically, we design a pixel-level texture enhancement module to acquire texture-aware features hierarchically with phase enhancement and amplified attention. Our proposed method performs favorably against state-of-the-art night-time semantic segmentation methods.
  arXiv  Detail & Related papers  (2024-08-25T13:59:31Z)
• RHRSegNet: Relighting High-Resolution Night-Time Semantic Segmentation [0.0]
  Night time semantic segmentation is a crucial task in computer vision, focusing on accurately classifying and segmenting objects in low-light conditions. We propose RHRSegNet, implementing a relighting model over a High-Resolution Network for semantic segmentation. Our proposed model increases the HRnet segmentation performance by 5% in low-light or nighttime images.
  arXiv  Detail & Related papers  (2024-07-08T15:07:09Z)
• Similarity Min-Max: Zero-Shot Day-Night Domain Adaptation [52.923298434948606]
  Low-light conditions not only hamper human visual experience but also degrade the model's performance on downstream vision tasks. This paper challenges a more complicated scenario with border applicability, i.e., zero-shot day-night domain adaptation. We propose a similarity min-max paradigm that considers them under a unified framework.
  arXiv  Detail & Related papers  (2023-07-17T18:50:15Z)
• Does Thermal Really Always Matter for RGB-T Salient Object Detection? [153.17156598262656]
  This paper proposes a network named TNet to solve the RGB-T salient object detection (SOD) task. In this paper, we introduce a global illumination estimation module to predict the global illuminance score of the image. On the other hand, we introduce a two-stage localization and complementation module in the decoding phase to transfer object localization cue and internal integrity cue in thermal features to the RGB modality.
  arXiv  Detail & Related papers  (2022-10-09T13:50:12Z)
• Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation [17.874336775904272]
  We propose a novel domain adaptation framework via cross-domain correlation distillation, called CCDistill. We extract the content and style knowledge contained in features, calculate the degree of inherent or illumination difference between two images. Experiments on Dark Zurich and ACDC demonstrate that CCDistill achieves the state-of-the-art performance for nighttime semantic segmentation.
  arXiv  Detail & Related papers  (2022-05-02T12:42:04Z)
• Bi-Mix: Bidirectional Mixing for Domain Adaptive Nighttime Semantic Segmentation [83.97914777313136]
  In autonomous driving, learning a segmentation model that can adapt to various environmental conditions is crucial. In this paper, we study the problem of Domain Adaptive Nighttime Semantic (DANSS), which aims to learn a discriminative nighttime model. We propose a novel Bi-Mix framework for DANSS, which can contribute to both image translation and segmentation adaptation processes.
  arXiv  Detail & Related papers  (2021-11-19T17:39:47Z)
• Meta-UDA: Unsupervised Domain Adaptive Thermal Object Detection using Meta-Learning [64.92447072894055]
  Infrared (IR) cameras are robust under adverse illumination and lighting conditions. We propose an algorithm meta-learning framework to improve existing UDA methods. We produce a state-of-the-art thermal detector for the KAIST and DSIAC datasets.
  arXiv  Detail & Related papers  (2021-10-07T02:28:18Z)
• DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation [18.43890050736093]
  We propose a novel domain adaptation network (DANNet) for nighttime semantic segmentation. It employs an adversarial training with a labeled daytime dataset and an unlabeled dataset that contains coarsely aligned day-night image pairs. Our method achieves state-of-the-art performance for nighttime semantic segmentation.
  arXiv  Detail & Related papers  (2021-04-22T02:49:28Z)
• Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme [68.8204255655161]
  In this report we present methods that we tried to achieve that goal. Our models are trained on a rendered dataset of artificial locations with varied scene content, light source location and color temperature. With this dataset, we used a network with illumination estimation component aiming to infer and replace light conditions in the latent space representation of the concerned scenes.
  arXiv  Detail & Related papers  (2020-06-03T15:25:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.