SuperEvent: Cross-Modal Learning of Event-based Keypoint Detection

• URL: http://arxiv.org/abs/2504.00139v1
• Date: Mon, 31 Mar 2025 18:46:02 GMT
• Title: SuperEvent: Cross-Modal Learning of Event-based Keypoint Detection
• Abstract summary: SuperEvent is a data-driven approach to predict stable keypoints with expressive descriptors.<n>We demonstrate the usefulness of SuperEvent by its integration into a modern sparse keypoint and descriptor-based SLAM framework.
• Score: 13.42760841894735
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event-based keypoint detection and matching holds significant potential, enabling the integration of event sensors into highly optimized Visual SLAM systems developed for frame cameras over decades of research. Unfortunately, existing approaches struggle with the motion-dependent appearance of keypoints and the complex noise prevalent in event streams, resulting in severely limited feature matching capabilities and poor performance on downstream tasks. To mitigate this problem, we propose SuperEvent, a data-driven approach to predict stable keypoints with expressive descriptors. Due to the absence of event datasets with ground truth keypoint labels, we leverage existing frame-based keypoint detectors on readily available event-aligned and synchronized gray-scale frames for self-supervision: we generate temporally sparse keypoint pseudo-labels considering that events are a product of both scene appearance and camera motion. Combined with our novel, information-rich event representation, we enable SuperEvent to effectively learn robust keypoint detection and description in event streams. Finally, we demonstrate the usefulness of SuperEvent by its integration into a modern sparse keypoint and descriptor-based SLAM framework originally developed for traditional cameras, surpassing the state-of-the-art in event-based SLAM by a wide margin. Source code and multimedia material are available at smartroboticslab.github.io/SuperEvent.

Related papers

• EventGait: Towards Robust Gait Recognition with Event Streams [50.890621860023]
  Event cameras offer microsecond temporal resolution and high dynamic range, naturally capturing robust dynamic cues and suppressing static noise.<n>We propose textbfEventGait, an end-to-end dual-stream framework that separately models motion and shape while preserving the advantages of events.<n>Our approach sets a new state of the art on both synthesized and real-world event-based gait benchmarks, highlighting the robustness and potential of event-driven gait analysis.
  arXiv  Detail & Related papers  (2026-05-21T08:12:04Z)
• Match-Any-Events: Zero-Shot Motion-Robust Feature Matching Across Wide Baselines for Event Cameras [40.06828305096689]
  We introduce the first event matching model that achieves cross-dataset wide-baseline correspondence in a zero-shot manner.<n>We introduce a motion-robust and computationally efficient attention backbone that learns multi-timescale features from event streams.<n>Our framework achieves a 37.7% improvement over the previous best event feature matching methods.
  arXiv  Detail & Related papers  (2026-04-20T18:48:53Z)
• EventFlash: Towards Efficient MLLMs for Event-Based Vision [55.65520031675231]
  Event-based multimodal language models (LMLMs) enable robust perception in high-speed and low-light scenarios.<n>We build EventMind, a large-scale and scene-diverse dataset with over 500k instruction sets.<n>We present an adaptive temporal window aggregation module for efficient temporal sampling, which adaptively compresses temporal tokens.<n>We believe EventFlash serves as an efficient foundation model for event-based vision.
  arXiv  Detail & Related papers  (2026-02-03T08:06:45Z)
• EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation [13.258464981482788]
  Event cameras produce asynchronous event streams that are spatially sparse yet temporally dense.<n>We propose a novel hypergraph-guided event stream completion mechanism, which connects tokens across different times and locations.<n>The proposed method can flexibly incorporate RGB tokens as nodes in the hypergraph within this completion framework, enabling multi-modal hypergraph-based information completion.
  arXiv  Detail & Related papers  (2025-11-26T14:30:04Z)
• EventSTU: Event-Guided Efficient Spatio-Temporal Understanding for Video Large Language Models [56.16721798968254]
  We propose an event-guided, training-free framework for efficient understanding, named EventSTU.<n>In the temporal domain, we design a coarse-to-fine sampling algorithm that the change-triggered property of event cameras to eliminate redundant large frames.<n>In the spatial domain, we achieves an adaptive token pruning algorithm that leverages the saliency of events as a zero-cost prior to guide spatial reduction.
  arXiv  Detail & Related papers  (2025-11-24T09:30:02Z)
• Event-based Facial Keypoint Alignment via Cross-Modal Fusion Attention and Self-Supervised Multi-Event Representation Learning [16.170645576584487]
  Event cameras offer unique advantages for facial keypoint alignment under challenging conditions.<n>We propose a novel framework based on cross-modal fusion attention (CMFA) and self-supervised multi-event representation learning (SSMER) for event-based facial keypoint alignment.
  arXiv  Detail & Related papers  (2025-09-29T16:00:50Z)
• SuperEIO: Self-Supervised Event Feature Learning for Event Inertial Odometry [6.552812892993662]
  Event cameras asynchronously output low-latency event streams, promising for state estimation in high-speed motion and challenging lighting conditions.<n>We propose SuperEIO, a novel framework that leverages the learning-based event-only detection and IMU measurements to achieve eventinertial odometry.<n>We evaluate our method extensively on multiple public datasets, demonstrating its superior accuracy and robustness compared to other state-of-the-art event-based methods.
  arXiv  Detail & Related papers  (2025-03-29T03:58:15Z)
• EventCrab: Harnessing Frame and Point Synergy for Event-based Action Recognition and Beyond [61.10181853363728]
  Event-based Action Recognition (EAR) possesses the advantages of high-temporal and privacy preservation compared with traditional action recognition.<n>We present EventCrab, a framework that adeptly integrates the "lighter" frame-specific networks for dense event frames with the "heavier" point-specific networks for sparse event points.<n>Experiments on four datasets demonstrate the significant performance of our proposed EventCrab.
  arXiv  Detail & Related papers  (2024-11-27T13:28:57Z)
• Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking [32.86991031493605]
  Event-based bionic camera captures dynamic scenes with high temporal resolution and high dynamic range. We propose a dynamic event subframe splitting strategy to split the event stream into more fine-grained event clusters. Based on this, we design an event-based sparse attention mechanism to enhance the interaction of event features in temporal and spatial dimensions.
  arXiv  Detail & Related papers  (2024-09-26T06:12:08Z)
• Event-to-Video Conversion for Overhead Object Detection [7.744259147081667]
  Event cameras complicate downstream image processing, especially for complex tasks such as object detection. We show that there is a significant gap in performance between dense event representations and corresponding RGB frames. We apply event-to-video conversion models that convert event streams into gray-scale video to close this gap.
  arXiv  Detail & Related papers  (2024-02-09T22:07:39Z)
• SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features [52.213656737672935]
  SpikeMOT is an event-based multi-object tracker. SpikeMOT uses spiking neural networks to extract sparsetemporal features from event streams associated with objects.
  arXiv  Detail & Related papers  (2023-09-29T05:13:43Z)
• Event-based Simultaneous Localization and Mapping: A Comprehensive Survey [52.73728442921428]
  Review of event-based vSLAM algorithms that exploit the benefits of asynchronous and irregular event streams for localization and mapping tasks. Paper categorizes event-based vSLAM methods into four main categories: feature-based, direct, motion-compensation, and deep learning methods.
  arXiv  Detail & Related papers  (2023-04-19T16:21:14Z)
• Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation [79.02808071245634]
  Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars.
  arXiv  Detail & Related papers  (2023-03-17T12:12:41Z)
• Avoiding Post-Processing with Event-Based Detection in Biomedical Signals [69.34035527763916]
  We propose an event-based modeling framework that directly works with events as learning targets. We show that event-based modeling (without post-processing) performs on par with or better than epoch-based modeling with extensive post-processing.
  arXiv  Detail & Related papers  (2022-09-22T13:44:13Z)
• Event Transformer [43.193463048148374]
  Event camera's low power consumption and ability to capture microsecond brightness make it attractive for various computer vision tasks. Existing event representation methods typically convert events into frames, voxel grids, or spikes for deep neural networks (DNNs) This work introduces a novel token-based event representation, where each event is considered a fundamental processing unit termed an event-token.
  arXiv  Detail & Related papers  (2022-04-11T15:05:06Z)
• Learning Constraints and Descriptive Segmentation for Subevent Detection [74.48201657623218]
  We propose an approach to learning and enforcing constraints that capture dependencies between subevent detection and EventSeg prediction. We adopt Rectifier Networks for constraint learning and then convert the learned constraints to a regularization term in the loss function of the neural model.
  arXiv  Detail & Related papers  (2021-09-13T20:50:37Z)
• VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows [93.54888104118822]
  We propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods.
  arXiv  Detail & Related papers  (2021-08-11T03:55:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.