Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras

• URL: http://arxiv.org/abs/2503.17262v1
• Date: Fri, 21 Mar 2025 16:04:13 GMT
• Title: Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras
• Authors: Shuang Guo, Friedhelm Hamann, Guillermo Gallego,
• Abstract summary: Event cameras rely on motion to obtain information about scene appearance.<n>We propose an unsupervised learning framework that jointly estimates optical flow (motion) and image intensity (appearance) with a single network.
• Score: 11.125596394858192
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Event cameras rely on motion to obtain information about scene appearance. In other words, for event cameras, motion and appearance are seen both or neither, which are encoded in the output event stream. Previous works consider recovering these two visual quantities as separate tasks, which does not fit with the nature of event cameras and neglects the inherent relations between both tasks. In this paper, we propose an unsupervised learning framework that jointly estimates optical flow (motion) and image intensity (appearance), with a single network. Starting from the event generation model, we newly derive the event-based photometric error as a function of optical flow and image intensity, which is further combined with the contrast maximization framework, yielding a comprehensive loss function that provides proper constraints for both flow and intensity estimation. Exhaustive experiments show that our model achieves state-of-the-art performance for both optical flow (achieves 20% and 25% improvement in EPE and AE respectively in the unsupervised learning category) and intensity estimation (produces competitive results with other baselines, particularly in high dynamic range scenarios). Last but not least, our model achieves shorter inference time than all the other optical flow models and many of the image reconstruction models, while they output only one quantity. Project page: https://github.com/tub-rip/e2fai

Related papers

• One Diffusion to Generate Them All [54.82732533013014]
  OneDiffusion is a versatile, large-scale diffusion model that supports bidirectional image synthesis and understanding. It enables conditional generation from inputs such as text, depth, pose, layout, and semantic maps. OneDiffusion allows for multi-view generation, camera pose estimation, and instant personalization using sequential image inputs.
  arXiv  Detail & Related papers  (2024-11-25T12:11:05Z)
• Learning Dense and Continuous Optical Flow from an Event Camera [28.77846425802558]
  Event cameras such as DAVIS can simultaneously output high temporal resolution events and low frame-rate intensity images. Most of the existing optical flow estimation methods are based on two consecutive image frames and can only estimate discrete flow at a fixed time interval. We propose a novel deep learning-based dense and continuous optical flow estimation framework from a single image with event streams.
  arXiv  Detail & Related papers  (2022-11-16T17:53:18Z)
• Motion-inductive Self-supervised Object Discovery in Videos [99.35664705038728]
  We propose a model for processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation. We demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets.
  arXiv  Detail & Related papers  (2022-10-01T08:38:28Z)
• Secrets of Event-Based Optical Flow [13.298845944779108]
  Event cameras respond to scene dynamics and offer advantages to estimate motion. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. Our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark.
  arXiv  Detail & Related papers  (2022-07-20T16:40:38Z)
• Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion [47.57998625129672]
  We introduce multi-scale feature-level fusion and computing one-shot non-linear inter-frame motion from events and images. We show that our method improves the reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS score.
  arXiv  Detail & Related papers  (2022-03-31T17:14:58Z)
• MEFNet: Multi-scale Event Fusion Network for Motion Deblurring [62.60878284671317]
  Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network.
  arXiv  Detail & Related papers  (2021-11-30T23:18:35Z)
• Dense Optical Flow from Event Cameras [55.79329250951028]
  We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras. Our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC.
  arXiv  Detail & Related papers  (2021-08-24T07:39:08Z)
• Learning Monocular Dense Depth from Events [53.078665310545745]
  Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames. Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction. We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
  arXiv  Detail & Related papers  (2020-10-16T12:36:23Z)
• Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy [0.0]
  Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. We propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art.
  arXiv  Detail & Related papers  (2020-09-17T13:30:05Z)
• Single Image Optical Flow Estimation with an Event Camera [38.92408855196647]
  Event cameras are bio-inspired sensors that report intensity changes in microsecond resolution. We propose a single image (potentially blurred) and events based optical flow estimation approach.
  arXiv  Detail & Related papers  (2020-04-01T11:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.