Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance

• URL: http://arxiv.org/abs/2505.22154v1
• Date: Wed, 28 May 2025 09:18:55 GMT
• Title: Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance
• Authors: Chao Tian, Chao Yang, Guoqing Zhu, Qiang Wang, Zhenyu He,
• Abstract summary: RGB-Thermal (RGB-T) object detection utilizes thermal infrared (TIR) images to complement RGB data, improving robustness in challenging conditions.<n>Traditional RGB-T detectors assume balanced training data, where both modalities contribute equally.<n>This paper addresses these challenges by proposing a novel base-and-auxiliary detector architecture.
• Score: 10.189143024910598
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: RGB-Thermal (RGB-T) object detection utilizes thermal infrared (TIR) images to complement RGB data, improving robustness in challenging conditions. Traditional RGB-T detectors assume balanced training data, where both modalities contribute equally. However, in real-world scenarios, modality degradation-due to environmental factors or technical issues-can lead to extreme modality imbalance, causing out-of-distribution (OOD) issues during testing and disrupting model convergence during training. This paper addresses these challenges by proposing a novel base-and-auxiliary detector architecture. We introduce a modality interaction module to adaptively weigh modalities based on their quality and handle imbalanced samples effectively. Additionally, we leverage modality pseudo-degradation to simulate real-world imbalances in training data. The base detector, trained on high-quality pairs, provides a consistency constraint for the auxiliary detector, which receives degraded samples. This framework enhances model robustness, ensuring reliable performance even under severe modality degradation. Experimental results demonstrate the effectiveness of our method in handling extreme modality imbalances~(decreasing the Missing Rate by 55%) and improving performance across various baseline detectors.

Related papers

• Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimization [23.328511708942045]
  Heterogeneity-aware Distributional Framework (HDF) designed to enhance time-frequency modeling and mitigate imbalance caused by hard samples.<n>Time-Frequency Distributional Attention Module (DAM) captures both temporal consistency and frequency robustness.<n> adaptive optimization module Distribution-aware Scaling Module (DSM) introduced to dynamically balance classification and contrastive losses.
  arXiv  Detail & Related papers  (2025-07-21T16:21:47Z)
• DINAMO: Dynamic and INterpretable Anomaly MOnitoring for Large-Scale Particle Physics Experiments [0.0]
  We present novel, interpretable, robust, and scalable DQM algorithms designed to automate anomaly detection.<n>Our approach constructs evolving histogram templates with built-in uncertainties, featuring a statistical variant.<n>Experiments on synthetic datasets demonstrate the high accuracy, adaptability, and interpretability of these methods.
  arXiv  Detail & Related papers  (2025-01-31T15:51:41Z)
• Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution [48.34173818491552]
  Diffusion Prior Interpolation (DPI) can balance consistency and diversity and can be seamlessly integrated into pre-trained models.<n>In extensive experiments conducted on synthetic and real datasets, DPI demonstrates superiority over SOTA FSR methods.
  arXiv  Detail & Related papers  (2024-12-21T09:28:44Z)
• Energy Score-based Pseudo-Label Filtering and Adaptive Loss for Imbalanced Semi-supervised SAR target recognition [1.2035771704626825]
  Existing semi-supervised SAR ATR algorithms show low recognition accuracy in the case of class imbalance. This work offers a non-balanced semi-supervised SAR target recognition approach using dynamic energy scores and adaptive loss.
  arXiv  Detail & Related papers  (2024-11-06T14:45:16Z)
• Automatically Adaptive Conformal Risk Control [49.95190019041905]
  We propose a methodology for achieving approximate conditional control of statistical risks by adapting to the difficulty of test samples.<n>Our framework goes beyond traditional conditional risk control based on user-provided conditioning events to the algorithmic, data-driven determination of appropriate function classes for conditioning.
  arXiv  Detail & Related papers  (2024-06-25T08:29:32Z)
• Feature Corrective Transfer Learning: End-to-End Solutions to Object Detection in Non-Ideal Visual Conditions [11.90136900277127]
  "Feature Corrective Transfer Learning" is a novel approach to facilitate the end-to-end detection of objects in challenging scenarios. Non-ideal images are processed by comparing their feature maps against those from the initial ideal RGB model. This approach refines the model's ability to perform object detection across varying conditions through direct feature map correction.
  arXiv  Detail & Related papers  (2024-04-17T09:58:53Z)
• Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection [22.60228799622782]
  Key bottleneck in object detection in IR images is lack of sufficient labeled training data. We seek to leverage cues from the RGB modality to scale object detectors to the IR modality, while preserving model performance in the RGB modality. We first pretrain these factor matrices on the RGB modality, for which plenty of training data are assumed to exist and then augment only a few trainable parameters for training on the IR modality to avoid over-fitting.
  arXiv  Detail & Related papers  (2023-09-28T16:55:52Z)
• Single-Trajectory Distributionally Robust Reinforcement Learning [21.955807398493334]
  We propose Distributionally Robust RL (DRRL) to enhance performance across a range of environments. Existing DRRL algorithms are either model-based or fail to learn from a single sample trajectory. We design a first fully model-free DRRL algorithm, called distributionally robust Q-learning with single trajectory (DRQ)
  arXiv  Detail & Related papers  (2023-01-27T14:08:09Z)
• Improve Noise Tolerance of Robust Loss via Noise-Awareness [60.34670515595074]
  We propose a meta-learning method which is capable of adaptively learning a hyper parameter prediction function, called Noise-Aware-Robust-Loss-Adjuster (NARL-Adjuster for brevity) Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance.
  arXiv  Detail & Related papers  (2023-01-18T04:54:58Z)
• Model-based Deep Learning Receiver Design for Rate-Splitting Multiple Access [65.21117658030235]
  This work proposes a novel design for a practical RSMA receiver based on model-based deep learning (MBDL) methods. The MBDL receiver is evaluated in terms of uncoded Symbol Error Rate (SER), throughput performance through Link-Level Simulations (LLS) and average training overhead. Results reveal that the MBDL outperforms by a significant margin the SIC receiver with imperfect CSIR.
  arXiv  Detail & Related papers  (2022-05-02T12:23:55Z)
• Self-Damaging Contrastive Learning [92.34124578823977]
  Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution. This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes. Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
  arXiv  Detail & Related papers  (2021-06-06T00:04:49Z)
• Learning perturbation sets for robust machine learning [97.6757418136662]
  We use a conditional generator that defines the perturbation set over a constrained region of the latent space. We measure the quality of our learned perturbation sets both quantitatively and qualitatively. We leverage our learned perturbation sets to train models which are empirically and certifiably robust to adversarial image corruptions and adversarial lighting variations.
  arXiv  Detail & Related papers  (2020-07-16T16:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.