Taming Contrast Maximization for Learning Sequential, Low-latency, Event-based Optical Flow

• URL: http://arxiv.org/abs/2303.05214v2
• Date: Wed, 27 Sep 2023 15:21:35 GMT
• Title: Taming Contrast Maximization for Learning Sequential, Low-latency, Event-based Optical Flow
• Authors: Federico Paredes-Vall\'es, Kirk Y. W. Scheper, Christophe De Wagter, Guido C. H. E. de Croon
• Abstract summary: Event cameras have gained significant traction since they open up new avenues for low-latency and low-power solutions to complex computer vision problems. To unlock these solutions, it is necessary to develop algorithms that can leverage the unique nature of event data. In this work, we propose a novel self-supervised learning pipeline for the estimation of event-based optical flow.
• Score: 18.335337530059867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras have recently gained significant traction since they open up new avenues for low-latency and low-power solutions to complex computer vision problems. To unlock these solutions, it is necessary to develop algorithms that can leverage the unique nature of event data. However, the current state-of-the-art is still highly influenced by the frame-based literature, and usually fails to deliver on these promises. In this work, we take this into consideration and propose a novel self-supervised learning pipeline for the sequential estimation of event-based optical flow that allows for the scaling of the models to high inference frequencies. At its core, we have a continuously-running stateful neural model that is trained using a novel formulation of contrast maximization that makes it robust to nonlinearities and varying statistics in the input events. Results across multiple datasets confirm the effectiveness of our method, which establishes a new state of the art in terms of accuracy for approaches trained or optimized without ground truth.

Related papers

• Event-Stream Super Resolution using Sigma-Delta Neural Network [0.10923877073891444]
  Event cameras present unique challenges due to their low resolution and sparse, asynchronous nature of the data they collect. Current event super-resolution algorithms are not fully optimized for the distinct data structure produced by event cameras. Research proposes a method that integrates binary spikes with Sigma Delta Neural Networks (SDNNs)
  arXiv  Detail & Related papers  (2024-08-13T15:25:18Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Revisit Event Generation Model: Self-Supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations [11.874972134063638]
  This paper proposes a novel SSL event-to-video reconstruction approach, dubbed EvINR, which eliminates the need for labeled data or optical flow estimation. We use an implicit neural representation (INR), which takes in coordinate $(x, y, t)$ and predicts intensity values, to represent the event generation equation. To make EvINR feasible for online requisites, we propose several acceleration techniques that substantially expedite the training process.
  arXiv  Detail & Related papers  (2024-07-26T04:18:10Z)
• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation [34.529280562470746]
  We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model by 29%.
  arXiv  Detail & Related papers  (2024-07-15T15:18:28Z)
• Event-Aided Time-to-Collision Estimation for Autonomous Driving [28.13397992839372]
  We present a novel method that estimates the time to collision using a neuromorphic event-based camera. The proposed algorithm consists of a two-step approach for efficient and accurate geometric model fitting on event data. Experiments on both synthetic and real data demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-07-10T02:37:36Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective [52.497514255040514]
  This work develops a new approach that enables data-driven methods to continuously learn and optimize resource allocation strategies in a dynamic environment. We propose to build the notion of continual learning into wireless system design, so that the learning model can incrementally adapt to the new episodes. Our design is based on a novel bilevel optimization formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2021-05-03T07:23:39Z)
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment [55.91291559442884]
  This work develops a methodology that enables data-driven methods to continuously learn and optimize in a dynamic environment. We propose to build the notion of continual learning into the modeling process of learning wireless systems. Our design is based on a novel min-max formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2020-11-16T08:24:34Z)
• Learning Monocular Dense Depth from Events [53.078665310545745]
  Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames. Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction. We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
  arXiv  Detail & Related papers  (2020-10-16T12:36:23Z)
• Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy [0.0]
  Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. We propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art.
  arXiv  Detail & Related papers  (2020-09-17T13:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.