From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization

• URL: http://arxiv.org/abs/2406.08090v2
• Date: Thu, 12 Sep 2024 12:18:21 GMT
• Title: From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization
• Authors: Ziran Zhang, Yongrui Ma, Yueting Chen, Feng Zhang, Jinwei Gu, Tianfan Xue, Shi Guo,
• Abstract summary: We propose a novel per-scene optimization strategy tailored for low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments.
• Score: 29.197409507402465
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video Frame Interpolation (VFI) is important for video enhancement, frame rate up-conversion, and slow-motion generation. The introduction of event cameras, which capture per-pixel brightness changes asynchronously, has significantly enhanced VFI capabilities, particularly for high-speed, nonlinear motions. However, these event-based methods encounter challenges in low-light conditions, notably trailing artifacts and signal latency, which hinder their direct applicability and generalization. Addressing these issues, we propose a novel per-scene optimization strategy tailored for low-light conditions. This approach utilizes the internal statistics of a sequence to handle degraded event data under low-light conditions, improving the generalizability to different lighting and camera settings. To evaluate its robustness in low-light condition, we further introduce EVFI-LL, a unique RGB+Event dataset captured under low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments. Project page: https://naturezhanghn.github.io/sim2real.

Related papers

• Event-guided Low-light Video Semantic Segmentation [6.938849566816958]
  Event cameras can capture motion dynamics, filter out temporal-redundant information, and are robust to lighting conditions. We propose EVSNet, a lightweight framework that leverages event modality to guide the learning of a unified illumination-invariant representation. Specifically, we leverage a Motion Extraction Module to extract short-term and long-term temporal motions from event modality and a Motion Fusion Module to integrate image features and motion features adaptively.
  arXiv  Detail & Related papers  (2024-11-01T14:54:34Z)
• Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions [56.84882059011291]
  We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
  arXiv  Detail & Related papers  (2024-09-26T15:57:20Z)
• Event-assisted Low-Light Video Object Segmentation [47.28027938310957]
  Event cameras offer promise in enhancing object visibility and aiding VOS methods under such low-light conditions. This paper introduces a pioneering framework tailored for low-light VOS, leveraging event camera data to elevate segmentation accuracy.
  arXiv  Detail & Related papers  (2024-04-02T13:41:22Z)
• Implicit Event-RGBD Neural SLAM [54.74363487009845]
  Implicit neural SLAM has achieved remarkable progress recently. Existing methods face significant challenges in non-ideal scenarios. We propose EN-SLAM, the first event-RGBD implicit neural SLAM framework.
  arXiv  Detail & Related papers  (2023-11-18T08:48:58Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• Revisiting Event-based Video Frame Interpolation [49.27404719898305]
  Dynamic vision sensors or event cameras provide rich complementary information for video frame. estimating optical flow from events is arguably more difficult than from RGB information. We propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages.
  arXiv  Detail & Related papers  (2023-07-24T06:51:07Z)
• PL-EVIO: Robust Monocular Event-based Visual Inertial Odometry with Point and Line Features [3.6355269783970394]
  Event cameras are motion-activated sensors that capture pixel-level illumination changes instead of the intensity image with a fixed frame rate. We propose a robust, high-accurate, and real-time optimization-based monocular event-based visual-inertial odometry (VIO) method.
  arXiv  Detail & Related papers  (2022-09-25T06:14:12Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.