Enhancing Traffic Object Detection in Variable Illumination with RGB-Event Fusion

• URL: http://arxiv.org/abs/2311.00436v2
• Date: Sun, 15 Sep 2024 11:47:20 GMT
• Title: Enhancing Traffic Object Detection in Variable Illumination with RGB-Event Fusion
• Authors: Zhanwen Liu, Nan Yang, Yang Wang, Yuke Li, Xiangmo Zhao, Fei-Yue Wang,
• Abstract summary: Traffic object detection under variable illumination is challenging due to the information loss caused by the limited dynamic range of conventional frame-based cameras. We propose a novel Structure-aware Fusion Network (SFNet) that extracts sharp and complete object structures from the event stream. Our proposed SFNet can overcome the perceptual boundaries of conventional cameras and outperform the frame-based method by 8.0% in mAP50 and 5.9% in mAP50:95.
• Score: 29.117211261620934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traffic object detection under variable illumination is challenging due to the information loss caused by the limited dynamic range of conventional frame-based cameras. To address this issue, we introduce bio-inspired event cameras and propose a novel Structure-aware Fusion Network (SFNet) that extracts sharp and complete object structures from the event stream to compensate for the lost information in images through cross-modality fusion, enabling the network to obtain illumination-robust representations for traffic object detection. Specifically, to mitigate the sparsity or blurriness issues arising from diverse motion states of traffic objects in fixed-interval event sampling methods, we propose the Reliable Structure Generation Network (RSGNet) to generate Speed Invariant Frames (SIF), ensuring the integrity and sharpness of object structures. Next, we design a novel Adaptive Feature Complement Module (AFCM) which guides the adaptive fusion of two modality features to compensate for the information loss in the images by perceiving the global lightness distribution of the images, thereby generating illumination-robust representations. Finally, considering the lack of large-scale and high-quality annotations in the existing event-based object detection datasets, we build a DSEC-Det dataset, which consists of 53 sequences with 63,931 images and more than 208,000 labels for 8 classes. Extensive experimental results demonstrate that our proposed SFNet can overcome the perceptual boundaries of conventional cameras and outperform the frame-based method by 8.0% in mAP50 and 5.9% in mAP50:95. Our code and dataset will be available at https://github.com/YN-Yang/SFNet.

Related papers

• TransBridge: Boost 3D Object Detection by Scene-Level Completion with Transformer Decoder [66.22997415145467]
  This paper presents a joint completion and detection framework that improves the detection feature in sparse areas.<n> Specifically, we propose TransBridge, a novel transformer-based up-sampling block that fuses the features from the detection and completion networks.<n>The results show that our framework consistently improves end-to-end 3D object detection, with the mean average precision (mAP) ranging from 0.7 to 1.5 across multiple methods.
  arXiv  Detail & Related papers  (2025-12-12T00:08:03Z)
• IrisNet: Infrared Image Status Awareness Meta Decoder for Infrared Small Targets Detection [92.56025546608699]
  IrisNet is a novel meta-learned framework that adapts detection strategies to the input infrared image status.<n>Our approach establishes a dynamic mapping between infrared image features and entire decoder parameters.<n> Experiments on NUDT-SIRST, NUAA-SIRST, and IRSTD-1K datasets demonstrate the superiority of our IrisNet.
  arXiv  Detail & Related papers  (2025-11-25T13:53:54Z)
• Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios [23.41380544271609]
  Dynamic range of conventional RGB cameras reduces global contrast and causes loss of high-frequency details.<n>We propose a motion cue fusion network (MCFNet) which achieves optimal cross-modal feature fusion under challenging lighting.<n>MCFNet significantly outperforms existing methods in various poor lighting and fast moving traffic scenarios.
  arXiv  Detail & Related papers  (2025-08-14T14:48:21Z)
• Beyond RGB and Events: Enhancing Object Detection under Adverse Lighting with Monocular Normal Maps [6.240947520777607]
  We introduce NRE-Net, a novel multi-modal detection framework.<n>It fuses three complementary modalities: monocularly predicted surface normal maps, RGB images, and event streams.<n>NRE-Net significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-04T07:19:20Z)
• LSFDNet: A Single-Stage Fusion and Detection Network for Ships Using SWIR and LWIR [16.16208006025223]
  Short-wave infrared (SWIR) and long-wave infrared (LWIR) are used in ship detection.<n>We propose a novel single-stage image fusion detection algorithm called LSFDNet.<n>This algorithm leverages feature interaction between the image fusion and object detection subtask networks.<n>We validated the superiority of our proposed single-stage fusion detection algorithm on two datasets.
  arXiv  Detail & Related papers  (2025-07-28T07:13:55Z)
• DFVO: Learning Darkness-free Visible and Infrared Image Disentanglement and Fusion All at Once [57.15043822199561]
  A Darkness-Free network is proposed to handle Visible and infrared image disentanglement and fusion all at Once (DFVO)<n>DFVO employs a cascaded multi-task approach to replace the traditional two-stage cascaded training (enhancement and fusion)<n>Our proposed approach outperforms state-of-the-art alternatives in terms of qualitative and quantitative evaluations.
  arXiv  Detail & Related papers  (2025-05-07T15:59:45Z)
• A Robust Deep Networks based Multi-Object MultiCamera Tracking System for City Scale Traffic [2.024925013349319]
  The proposed framework is evaluated on the 5th AI City Challenge dataset (Track 3), comprising 46 camera feeds. The framework achieves competitive performance with an IDF1 score of 0.8289, and precision and recall scores of 0.9026 and 0.8527 respectively.
  arXiv  Detail & Related papers  (2025-05-01T14:00:25Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [63.87313550399871]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability. We propose Self-supervised Transfer (PST) and FrequencyDe-coupled Fusion module (FreDF) PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models. FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection [17.406051477690134]
  Event cameras output sparse and asynchronous events, providing a potential solution to solve these problems. We propose a novel hierarchical feature refinement network for event-frame fusion. Our method exhibits significantly better robustness when introducing 15 different corruption types to the frame images.
  arXiv  Detail & Related papers  (2024-07-17T14:09:46Z)
• Enhanced Automotive Object Detection via RGB-D Fusion in a DiffusionDet Framework [0.0]
  Vision-based autonomous driving requires reliable and efficient object detection. This work proposes a DiffusionDet-based framework that exploits data fusion from the monocular camera and depth sensor to provide the RGB and depth (RGB-D) data. By integrating the textural and color features from RGB images with the spatial depth information from the LiDAR sensors, the proposed framework employs a feature fusion that substantially enhances object detection of automotive targets.
  arXiv  Detail & Related papers  (2024-06-05T10:24:00Z)
• Deformable Convolutions and LSTM-based Flexible Event Frame Fusion Network for Motion Deblurring [7.187030024676791]
  Event cameras differ from conventional RGB cameras in that they produce asynchronous data sequences. While RGB cameras capture every frame at a fixed rate, event cameras only capture changes in the scene, resulting in sparse and asynchronous data output. Recent state-of-the-art CNN-based deblurring solutions produce multiple 2-D event frames based on the accumulation of event data over a time period. It is particularly useful for scenarios in which exposure times vary depending on factors such as lighting conditions or the presence of fast-moving objects in the scene.
  arXiv  Detail & Related papers  (2023-06-01T15:57:12Z)
• Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation [79.02808071245634]
  Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars.
  arXiv  Detail & Related papers  (2023-03-17T12:12:41Z)
• High-resolution Iterative Feedback Network for Camouflaged Object Detection [128.893782016078]
  Spotting camouflaged objects that are visually assimilated into the background is tricky for object detection algorithms. We aim to extract the high-resolution texture details to avoid the detail degradation that causes blurred vision in edges and boundaries. We introduce a novel HitNet to refine the low-resolution representations by high-resolution features in an iterative feedback manner.
  arXiv  Detail & Related papers  (2022-03-22T11:20:21Z)
• TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers [49.689566246504356]
  We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. TransFusion achieves state-of-the-art performance on large-scale datasets. We extend the proposed method to the 3D tracking task and achieve the 1st place in the leaderboard of nuScenes tracking.
  arXiv  Detail & Related papers  (2022-03-22T07:15:13Z)
• MEFNet: Multi-scale Event Fusion Network for Motion Deblurring [62.60878284671317]
  Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network.
  arXiv  Detail & Related papers  (2021-11-30T23:18:35Z)
• Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architectures [7.565038387344594]
  We present a sensor fusion framework for energy-efficient optical flow estimation using both frame- and event-based sensors. Our network is end-to-end trained using unsupervised learning to avoid expensive video annotations.
  arXiv  Detail & Related papers  (2021-03-19T02:03:33Z)
• Dense Attention Fluid Network for Salient Object Detection in Optical Remote Sensing Images [193.77450545067967]
  We propose an end-to-end Dense Attention Fluid Network (DAFNet) for salient object detection in optical remote sensing images (RSIs) A Global Context-aware Attention (GCA) module is proposed to adaptively capture long-range semantic context relationships. We construct a new and challenging optical RSI dataset for SOD that contains 2,000 images with pixel-wise saliency annotations.
  arXiv  Detail & Related papers  (2020-11-26T06:14:10Z)
• Feature Flow: In-network Feature Flow Estimation for Video Object Detection [56.80974623192569]
  Optical flow is widely used in computer vision tasks to provide pixel-level motion information. A common approach is to:forward optical flow to a neural network and fine-tune this network on the task dataset. We propose a novel network (IFF-Net) with an textbfIn-network textbfFeature textbfFlow estimation module for video object detection.
  arXiv  Detail & Related papers  (2020-09-21T07:55:50Z)
• Dual Semantic Fusion Network for Video Object Detection [35.175552056938635]
  We propose a dual semantic fusion network (DSFNet) to fully exploit both frame-level and instance-level semantics in a unified fusion framework without external guidance. The proposed DSFNet can generate more robust features through the multi-granularity fusion and avoid being affected by the instability of external guidance.
  arXiv  Detail & Related papers  (2020-09-16T06:49:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.