Perturbed State Space Feature Encoders for Optical Flow with Event Cameras

• URL: http://arxiv.org/abs/2504.10669v1
• Date: Mon, 14 Apr 2025 19:45:07 GMT
• Title: Perturbed State Space Feature Encoders for Optical Flow with Event Cameras
• Authors: Gokul Raju Govinda Raju, Nikola Zubić, Marco Cannici, Davide Scaramuzza,
• Abstract summary: Event-based cameras offer significant advantages over traditional cameras for optical flow estimation.<n>Current neural networks adopted for event-based optical flow still face temporal and spatial reasoning limitations.<n>We propose Perturbed State Space Feature challenges for multi-frame optical flow with event cameras.
• Score: 17.147140984254655
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With their motion-responsive nature, event-based cameras offer significant advantages over traditional cameras for optical flow estimation. While deep learning has improved upon traditional methods, current neural networks adopted for event-based optical flow still face temporal and spatial reasoning limitations. We propose Perturbed State Space Feature Encoders (P-SSE) for multi-frame optical flow with event cameras to address these challenges. P-SSE adaptively processes spatiotemporal features with a large receptive field akin to Transformer-based methods, while maintaining the linear computational complexity characteristic of SSMs. However, the key innovation that enables the state-of-the-art performance of our model lies in our perturbation technique applied to the state dynamics matrix governing the SSM system. This approach significantly improves the stability and performance of our model. We integrate P-SSE into a framework that leverages bi-directional flows and recurrent connections, expanding the temporal context of flow prediction. Evaluations on DSEC-Flow and MVSEC datasets showcase P-SSE's superiority, with 8.48% and 11.86% improvements in EPE performance, respectively.

Related papers

• OptFormer: Optical Flow-Guided Attention and Phase Space Reconstruction for SST Forecasting [4.206799880454911]
  We propose OptFormer, a novel optical encoder-decoder model that integrates phase-space reconstruction with a motion-aware attention mechanism guided by flow.<n>Experiments on NOAA SST datasets across multiple spatial scales demonstrate that OptFormer achieves superior performance under a 1:1 training-to-prediction setting.
  arXiv  Detail & Related papers  (2025-12-29T22:27:15Z)
• Inertia-Informed Orientation Priors for Event-Based Optical Flow Estimation [7.36599004748324]
  Event cameras directly encode motion within a scene.<n>Many learning-based and model-based methods exist that estimate event-based optical flow.<n>We introduce a novel biologically dense-inspired hybrid CM that couples spatially visual and sparse motion cues.
  arXiv  Detail & Related papers  (2025-11-17T04:39:18Z)
• Inference-Time Gaze Refinement for Micro-Expression Recognition: Enhancing Event-Based Eye Tracking with Motion-Aware Post-Processing [2.5465367830324905]
  Event-based eye tracking holds significant promise for fine-grained cognitive state inference.<n>We introduce a model-agnostic, inference-time refinement framework to enhance the output of existing event-based gaze estimation models.
  arXiv  Detail & Related papers  (2025-06-14T14:48:11Z)
• Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems [49.819436680336786]
  Gaussian process state-space models (GPSSMs) have emerged as a powerful framework for modeling dynamical systems.<n>We propose an efficient transformed Gaussian process state-space model (ETGPSSM) to address these limitations.<n>Our approach leverages a single shared Gaussian process (GP) combined with normalizing flows and Bayesian neural networks, enabling efficient modeling of complex, high-dimensional state transitions.
  arXiv  Detail & Related papers  (2025-03-24T03:19:45Z)
• SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation [10.696635172502141]
  Event cameras generate asynchronous and sparse event streams capturing changes in light intensity. Spiking neural networks (SNNs) share similar asynchronous and sparse characteristics and are well-suited for event cameras. We propose two solutions for fast and robust optical flow estimation for event cameras: STTFlowNet and SDFlowformer.
  arXiv  Detail & Related papers  (2024-09-06T07:48:18Z)
• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation [34.529280562470746]
  We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model by 29%.
  arXiv  Detail & Related papers  (2024-07-15T15:18:28Z)
• SFANet: Spatial-Frequency Attention Network for Weather Forecasting [54.470205739015434]
  Weather forecasting plays a critical role in various sectors, driving decision-making and risk management. Traditional methods often struggle to capture the complex dynamics of meteorological systems. We propose a novel framework designed to address these challenges and enhance the accuracy of weather prediction.
  arXiv  Detail & Related papers  (2024-05-29T08:00:15Z)
• Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidance [18.861739607401322]
  This paper proposes a two-stage framework named ST-PAD for capturing-temporal fluid dynamics modeling. In the upstream stage, we design a vector reconstruction module with temporal characteristics. In the downstream stage, a diffusion probability network involving parameters is utilized to generate high-quality future states of fluids.
  arXiv  Detail & Related papers  (2024-03-18T14:57:47Z)
• Generative Modeling with Phase Stochastic Bridges [49.4474628881673]
  Diffusion models (DMs) represent state-of-the-art generative models for continuous inputs. We introduce a novel generative modeling framework grounded in textbfphase space dynamics Our framework demonstrates the capability to generate realistic data points at an early stage of dynamics propagation.
  arXiv  Detail & Related papers  (2023-10-11T18:38:28Z)
• Asynchronous Optimisation for Event-based Visual Odometry [53.59879499700895]
  Event cameras open up new possibilities for robotic perception due to their low latency and high dynamic range. We focus on event-based visual odometry (VO) We propose an asynchronous structure-from-motion optimisation back-end.
  arXiv  Detail & Related papers  (2022-03-02T11:28:47Z)
• hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow [0.0]
  Event-based vision sensors produce asynchronous event streams with high temporal resolution based on changes in the visual scene. Existing solutions for calculating optical flow from event data fail to capture the true direction of motion due to the aperture problem. We present a hardware realization of the fARMS algorithm allowing for real-time computation of true flow on low-power, embedded platforms.
  arXiv  Detail & Related papers  (2021-12-13T16:27:17Z)
• Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution [52.899234731501075]
  Space-time video super-resolution (ST-VSR) simultaneously increases the spatial resolution and frame rate for a given video. Existing methods typically suffer from difficulties in how to efficiently leverage information from a large range of neighboring frames. We propose a coarse-to-fine bidirectional recurrent neural network instead of using ConvLSTM to leverage knowledge between adjacent frames.
  arXiv  Detail & Related papers  (2021-10-13T15:21:30Z)
• PCA Event-Based Otical Flow for Visual Odometry [0.0]
  We present a Principal Component Analysis approach to the problem of event-based optical flow estimation. We show that the best variant of our proposed method, dedicated to the real-time context of visual odometry, is about two times faster compared to state-of-the-art implementations.
  arXiv  Detail & Related papers  (2021-05-08T18:30:44Z)
• Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization [59.9673626329892]
  We exploit the global relationship between optical flow and camera motion using epipolar geometry. We use implicit differentiation to enable back-propagation through the lower-level geometric optimization layer independent of its implementation.
  arXiv  Detail & Related papers  (2020-02-26T22:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.