Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation

• URL: http://arxiv.org/abs/2411.17610v1
• Date: Tue, 26 Nov 2024 17:21:53 GMT
• Title: Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation
• Authors: Niharika Hegde, Shishir Muralidhara, René Schuster, Didier Stricker,
• Abstract summary: In autonomous driving, diverse sensors contribute to robustness against adverse weather and lighting conditions. In the context of continual learning, incremental learning is especially challenging for considerably large domain shifts. We propose the use of a modified Relevance Mapping Network (RMN) to incrementally learn new modalities while preserving performance on previously learned modalities.
• Score: 13.137604149999973
• License:
• Abstract: In autonomous driving, environment perception has significantly advanced with the utilization of deep learning techniques for diverse sensors such as cameras, depth sensors, or infrared sensors. The diversity in the sensor stack increases the safety and contributes to robustness against adverse weather and lighting conditions. However, the variance in data acquired from different sensors poses challenges. In the context of continual learning (CL), incremental learning is especially challenging for considerably large domain shifts, e.g. different sensor modalities. This amplifies the problem of catastrophic forgetting. To address this issue, we formulate the concept of modality-incremental learning and examine its necessity, by contrasting it with existing incremental learning paradigms. We propose the use of a modified Relevance Mapping Network (RMN) to incrementally learn new modalities while preserving performance on previously learned modalities, in which relevance maps are disjoint. Experimental results demonstrate that the prevention of shared connections in this approach helps alleviate the problem of forgetting within the constraints of a strict continual learning framework.

Related papers

• MSSIDD: A Benchmark for Multi-Sensor Denoising [55.41612200877861]
  We introduce a new benchmark, the Multi-Sensor SIDD dataset, which is the first raw-domain dataset designed to evaluate the sensor transferability of denoising models. We propose a sensor consistency training framework that enables denoising models to learn the sensor-invariant features.
  arXiv  Detail & Related papers  (2024-11-18T13:32:59Z)
• Efficient and robust Sensor Placement in Complex Environments [1.1421942894219899]
  This paper addresses the problem of efficient and unobstructed surveillance or communication in complex environments. We propose a greedy algorithm to achieve the objective. Deep learning techniques are used to accelerate the evaluation of the objective function.
  arXiv  Detail & Related papers  (2023-09-15T17:10:19Z)
• Machine Learning Based Compensation for Inconsistencies in Knitted Force Sensors [1.0742675209112622]
  Knitted sensors frequently suffer from inconsistencies due to innate effects such as offset, relaxation, and drift. In this paper, we demonstrate a method for counteracting this by applying processing using a minimal artificial neural network (ANN) By training a three-layer ANN with a total of 8 neurons, we manage to significantly improve the mapping between sensor reading and actuation force.
  arXiv  Detail & Related papers  (2023-06-21T09:19:33Z)
• TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation [6.458496335718508]
  We propose a novel Transformer-based Adversarial learning framework for human activity recognition using wearable sensors via Self-KnowledgE Distillation (TASKED) In the proposed method, we adopt the teacher-free self-knowledge distillation to improve the stability of the training procedure and the performance of human activity recognition.
  arXiv  Detail & Related papers  (2022-09-14T11:08:48Z)
• Learning Online Multi-Sensor Depth Fusion [100.84519175539378]
  SenFuNet is a depth fusion approach that learns sensor-specific noise and outlier statistics. We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets.
  arXiv  Detail & Related papers  (2022-04-07T10:45:32Z)
• Improving robustness of jet tagging algorithms with adversarial training [56.79800815519762]
  We investigate the vulnerability of flavor tagging algorithms via application of adversarial attacks. We present an adversarial training strategy that mitigates the impact of such simulated attacks.
  arXiv  Detail & Related papers  (2022-03-25T19:57:19Z)
• Bayesian Imitation Learning for End-to-End Mobile Manipulation [80.47771322489422]
  Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities. We show that using the Variational Information Bottleneck to regularize convolutional neural networks improves generalization to held-out domains. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities.
  arXiv  Detail & Related papers  (2022-02-15T17:38:30Z)
• Plants Don't Walk on the Street: Common-Sense Reasoning for Reliable Semantic Segmentation [0.7696728525672148]
  We propose to use a partly human-designed, partly learned set of rules to describe relations between objects of a traffic scene on a high level of abstraction. In doing so, we improve and robustify existing deep neural networks consuming low-level sensor information.
  arXiv  Detail & Related papers  (2021-04-19T12:51:06Z)
• Improving Robustness of Learning-based Autonomous Steering Using Adversarial Images [58.287120077778205]
  We introduce a framework for analyzing robustness of the learning algorithm w.r.t varying quality in the image input for autonomous driving. Using the results of sensitivity analysis, we propose an algorithm to improve the overall performance of the task of "learning to steer"
  arXiv  Detail & Related papers  (2021-02-26T02:08:07Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.