Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model

• URL: http://arxiv.org/abs/2605.23790v1
• Date: Fri, 22 May 2026 15:52:01 GMT
• Title: Exploring deep learning for Event-Based Saliency Prediction with a Transformer-based model
• Abstract summary: Saliency prediction has been extensively studied in RGB images and videos as a computational model of human visual attention.<n>We introduce SEST (Swin Event-based Saliency Transformer), a transformer-based model for saliency prediction from event data.<n>We introduce two new benchmark datasets, N-DHF1K and N-UCF Sports, generated from large-scale RGB saliency benchmarks.
• Score: 3.1333320740278623
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Saliency prediction has been extensively studied in RGB images and videos as a computational model of human visual attention. In contrast, predicting saliency from event-based data remains largely unexplored, despite the biological inspiration and favorable sensing properties of event cameras. Two obstacles have held this direction back: the absence of large-scale event saliency datasets, and the lack of a strong baseline. In this paper, we introduce SEST (Swin Event-based Saliency Transformer), a transformer-based model for saliency prediction from event data, bridging the data scarcity barrier through event-native pretraining and synthetic supervision. SEST leverages a self-supervised pretrained event-based Swin Transformer backbone combined with a lightweight CNN decoder to produce dynamic saliency maps. To address the scarcity of annotated event-based saliency data, we introduce two new benchmark datasets, N-DHF1K and N-UCF Sports, generated from large-scale RGB saliency benchmarks. Experimental results show that SEST clearly outperforms existing event-based saliency methods and narrows the performance gap with state-of-the-art RGB models. Zero-shot evaluation on a real event camera dataset further demonstrates that our model trained on synthetic data remains transferable on real event streams. To the best of our knowledge, this work is the first to apply deep learning to event-based saliency prediction, opening a new research direction at the intersection of event-based vision and neuromorphic visual attention.

Related papers

• EventGait: Towards Robust Gait Recognition with Event Streams [50.890621860023]
  Event cameras offer microsecond temporal resolution and high dynamic range, naturally capturing robust dynamic cues and suppressing static noise.<n>We propose textbfEventGait, an end-to-end dual-stream framework that separately models motion and shape while preserving the advantages of events.<n>Our approach sets a new state of the art on both synthesized and real-world event-based gait benchmarks, highlighting the robustness and potential of event-driven gait analysis.
  arXiv  Detail & Related papers  (2026-05-21T08:12:04Z)
• Towards Video Anomaly Detection from Event Streams: A Baseline and Benchmark Datasets [71.53287557600177]
  We take the first major step toward establishing event-based anomaly detection as a unified research direction.<n>We first construct multiple eventstream based benchmarks for video anomaly detection, featuring synchronized event and RGB recordings.<n>We then propose an EVent-centric Video Anomaly Detection framework, namely EWAD, with three key innovations.
  arXiv  Detail & Related papers  (2026-03-26T03:33:33Z)
• Ev4DGS: Novel-view Rendering of Non-Rigid Objects from Monocular Event Streams [34.02850677061797]
  Ev4DGS is the first approach for novel view rendering of non-rigidly deforming objects in the explicit observation space.<n>Our method regresses a deformable 3D Gaussian Splatting representation through 1) a loss relating the outputs of the estimated model with the 2D event observation space, and 2) a coarse 3D deformation model trained from binary masks generated from events.
  arXiv  Detail & Related papers  (2025-10-13T17:59:55Z)
• ControlEvents: Controllable Synthesis of Event Camera Datawith Foundational Prior from Image Diffusion Models [61.17744115607788]
  We present a diffusion-based generative model designed to synthesize high-quality event data guided by diverse control signals.<n>We demonstrate the effectiveness of our approach by synthesizing event data for visual recognition, 2D skeleton estimation, and 3D body pose estimation.
  arXiv  Detail & Related papers  (2025-09-26T19:22:07Z)
• Leveraging RGB Images for Pre-Training of Event-Based Hand Pose Estimation [64.8814078041756]
  RPEP is the first pre-training method for event-based 3D hand pose estimation using labeled RGB images and unpaired, unlabeled event data.<n>Our model significantly outperforms state-of-the-art methods on real event data, achieving up to 24% improvement on EvRealHands.
  arXiv  Detail & Related papers  (2025-09-21T07:07:49Z)
• SuperEIO: Self-Supervised Event Feature Learning for Event Inertial Odometry [6.552812892993662]
  Event cameras asynchronously output low-latency event streams, promising for state estimation in high-speed motion and challenging lighting conditions.<n>We propose SuperEIO, a novel framework that leverages the learning-based event-only detection and IMU measurements to achieve eventinertial odometry.<n>We evaluate our method extensively on multiple public datasets, demonstrating its superior accuracy and robustness compared to other state-of-the-art event-based methods.
  arXiv  Detail & Related papers  (2025-03-29T03:58:15Z)
• Evaluating Image-Based Face and Eye Tracking with Event Cameras [9.677797822200965]
  Event Cameras, also known as Neuromorphic sensors, capture changes in local light intensity at the pixel level, producing asynchronously generated data termed events'' This data format mitigates common issues observed in conventional cameras, like under-sampling when capturing fast-moving objects. We evaluate the viability of integrating conventional algorithms with event-based data, transformed into a frame format.
  arXiv  Detail & Related papers  (2024-08-19T20:27:08Z)
• Event Camera Data Dense Pre-training [10.918407820258246]
  This paper introduces a self-supervised learning framework designed for pre-training neural networks tailored to dense prediction tasks using event camera data. For training our framework, we curate a synthetic event camera dataset featuring diverse scene and motion patterns.
  arXiv  Detail & Related papers  (2023-11-20T04:36:19Z)
• Implicit Event-RGBD Neural SLAM [54.74363487009845]
  Implicit neural SLAM has achieved remarkable progress recently. Existing methods face significant challenges in non-ideal scenarios. We propose EN-SLAM, the first event-RGBD implicit neural SLAM framework.
  arXiv  Detail & Related papers  (2023-11-18T08:48:58Z)
• EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields [80.94515892378053]
  EvDNeRF is a pipeline for generating event data and training an event-based dynamic NeRF. NeRFs offer geometric-based learnable rendering, but prior work with events has only considered reconstruction of static scenes. We show that by training on varied batch sizes of events, we can improve test-time predictions of events at fine time resolutions.
  arXiv  Detail & Related papers  (2023-10-03T21:08:41Z)
• Improve Event Extraction via Self-Training with Gradient Guidance [10.618929821822892]
  We propose a Self-Training with Feedback (STF) framework to overcome the main factor that hinders the progress of event extraction. STF consists of (1) a base event extraction model trained on existing event annotations and then applied to large-scale unlabeled corpora to predict new event mentions as pseudo training samples, and (2) a novel scoring model that takes in each new predicted event trigger, an argument, its argument role, as well as their paths in the AMR graph to estimate a compatibility score. Experimental results on three benchmark datasets, including ACE05-E, ACE05-E+, and ERE
  arXiv  Detail & Related papers  (2022-05-25T04:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.