Towards Robust Event-based Networks for Nighttime via Unpaired Day-to-Night Event Translation

• URL: http://arxiv.org/abs/2407.10703v1
• Date: Mon, 15 Jul 2024 13:18:37 GMT
• Title: Towards Robust Event-based Networks for Nighttime via Unpaired Day-to-Night Event Translation
• Authors: Yuhwan Jeong, Hoonhee Cho, Kuk-Jin Yoon,
• Abstract summary: Event cameras with high dynamic range ensure scene capture even in low-light conditions. Night events exhibit patterns different from those captured during the day. This difference causes performance degradation when applying night events to a model trained solely on day events.
• Score: 42.961332723891324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras with high dynamic range ensure scene capture even in low-light conditions. However, night events exhibit patterns different from those captured during the day. This difference causes performance degradation when applying night events to a model trained solely on day events. This limitation persists due to a lack of annotated night events. To overcome the limitation, we aim to alleviate data imbalance by translating annotated day data into night events. However, generating events from different modalities challenges reproducing their unique properties. Accordingly, we propose an unpaired event-to-event day-to-night translation model that effectively learns to map from one domain to another using Diffusion GAN. The proposed translation model analyzes events in spatio-temporal dimension with wavelet decomposition and disentangled convolution layers. We also propose a new temporal contrastive learning with a novel shuffling and sampling strategy to regularize temporal continuity. To validate the efficacy of the proposed methodology, we redesign metrics for evaluating events translated in an unpaired setting, aligning them with the event modality for the first time. Our framework shows the successful day-to-night event translation while preserving the characteristics of events. In addition, through our translation method, we facilitate event-based modes to learn about night events by translating annotated day events into night events. Our approach effectively mitigates the performance degradation of applying real night events to downstream tasks. The code is available at https://github.com/jeongyh98/UDNET.

Related papers

• Asynchronous Event Error-Minimizing Noise for Safeguarding Event Dataset [57.17622057280324]
  Unlearnable Examples are proposed to prevent the unauthorized exploitation of image datasets.<n>We propose the first unlearnable event stream generation method to prevent unauthorized training from event datasets.
  arXiv  Detail & Related papers  (2025-07-08T07:21:59Z)
• Seeing Motion at Nighttime with an Event Camera [17.355331119296782]
  Event cameras react to dynamic with higher temporal resolution (microsecond) and higher dynamic range (120dB) We propose a nighttime event reconstruction network (NER-Net) which mainly includes a learnable event timestamps calibration module (LETC) We construct a paired real-light event dataset (RLED) through a co-axial imaging, including 64,200 spatially and temporally aligned image GTs and low-light events.
  arXiv  Detail & Related papers  (2024-04-18T03:58:27Z)
• Improving Event Definition Following For Zero-Shot Event Detection [66.27883872707523]
  Existing approaches on zero-shot event detection usually train models on datasets annotated with known event types. We aim to improve zero-shot event detection by training models to better follow event definitions.
  arXiv  Detail & Related papers  (2024-03-05T01:46:50Z)
• Event Camera Data Dense Pre-training [10.918407820258246]
  This paper introduces a self-supervised learning framework designed for pre-training neural networks tailored to dense prediction tasks using event camera data. For training our framework, we curate a synthetic event camera dataset featuring diverse scene and motion patterns.
  arXiv  Detail & Related papers  (2023-11-20T04:36:19Z)
• Continual Event Extraction with Semantic Confusion Rectification [50.59450741139265]
  We study continual event extraction, which aims to extract incessantly emerging event information while avoiding forgetting. We observe that the semantic confusion on event types stems from the annotations of the same text being updated over time. This paper proposes a novel continual event extraction model with semantic confusion rectification.
  arXiv  Detail & Related papers  (2023-10-24T02:48:50Z)
• Disentangled Contrastive Image Translation for Nighttime Surveillance [87.03178320662592]
  Nighttime surveillance suffers from degradation due to poor illumination and arduous human annotations. Existing methods rely on multi-spectral images to perceive objects in the dark, which are troubled by low resolution and color absence. We argue that the ultimate solution for nighttime surveillance is night-to-day translation, or Night2Day. This paper contributes a new surveillance dataset called NightSuR. It includes six scenes to support the study on nighttime surveillance.
  arXiv  Detail & Related papers  (2023-07-11T06:40:27Z)
• Unifying Event Detection and Captioning as Sequence Generation via Pre-Training [53.613265415703815]
  We propose a unified pre-training and fine-tuning framework to enhance the inter-task association between event detection and captioning. Our model outperforms the state-of-the-art methods, and can be further boosted when pre-trained on extra large-scale video-text data.
  arXiv  Detail & Related papers  (2022-07-18T14:18:13Z)
• Event Transformer [43.193463048148374]
  Event camera's low power consumption and ability to capture microsecond brightness make it attractive for various computer vision tasks. Existing event representation methods typically convert events into frames, voxel grids, or spikes for deep neural networks (DNNs) This work introduces a novel token-based event representation, where each event is considered a fundamental processing unit termed an event-token.
  arXiv  Detail & Related papers  (2022-04-11T15:05:06Z)
• Adversarial Attack for Asynchronous Event-based Data [0.19580473532948398]
  We generate adversarial examples and then train the robust models for event-based data for the first time. Our algorithm achieves an attack success rate of 97.95% on the N-Caltech101 dataset.
  arXiv  Detail & Related papers  (2021-12-27T06:23:43Z)
• Learning Constraints and Descriptive Segmentation for Subevent Detection [74.48201657623218]
  We propose an approach to learning and enforcing constraints that capture dependencies between subevent detection and EventSeg prediction. We adopt Rectifier Networks for constraint learning and then convert the learned constraints to a regularization term in the loss function of the neural model.
  arXiv  Detail & Related papers  (2021-09-13T20:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.