Pedestrian Crossing Intent Classification From Event-Based Vision Using Convolutional Spiking Neural Networks With Temporal Augmentation

• URL: http://arxiv.org/abs/2609.13328v1
• Date: Fri, 11 Sep 2026 02:34:24 GMT
• Title: Pedestrian Crossing Intent Classification From Event-Based Vision Using Convolutional Spiking Neural Networks With Temporal Augmentation
• Abstract summary: We present an end-to-end pipeline that converts real-world driving footage into synthetic vision sensor (DVS) event streams.<n>We also train a novel convolutional spiking neural network (Conv-SNN) with clip-consistent DVS augmentation to classify pedestrian crossing intent as binary.<n>The trained model achieves 95.83% accuracy and F1 = 0.9695 on the JAAD DVS test set, 97.79% accuracy and F1 = 0.9478 on normal CARLA DVS, and 94.78% accuracy and F1 = 0.8369 on adverse-weather CARLA DVS.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Anticipating whether a pedestrian will cross the road is safety-critical for autonomous vehicles, requiring real-time inference under challenging conditions including motion blur, high dynamic range, and class imbalance. Conventional frame-based deep networks process redundant RGB data at fixed frame rates, limiting their temporal resolution and energy efficiency. In this work we present an end-to-end pipeline that (i) converts real-world driving footage from the Joint Attention in Autonomous Driving (JAAD) dataset into synthetic dynamic vision sensor (DVS) event streams using the v2e simulator, (ii) augments training with the CARLA-simulated DVS sequences of the DVS-PedX dataset under both normal and adverse weather conditions, and (iii) trains a novel convolutional spiking neural network (Conv-SNN) with clip-consistent DVS augmentation to classify pedestrian crossing intent as binary: crossing or non-crossing. We detail all architectural decisions, the exact leaky-integrate-and-fire neuron dynamics with surrogate-gradient learning, the class-balanced loss formulation, JAAD oversampling at 6x, and a 70/15/15 stratified splitting protocol. The trained model achieves 95.83% accuracy and F1 = 0.9695 on the JAAD DVS test set, 97.79% accuracy and F1 = 0.9478 on normal CARLA DVS, and 94.78% accuracy and F1 = 0.8369 on adverse-weather CARLA DVS, all from a 1.07M-parameter architecture trained on CPU. Compared to prior frame-based approaches on JAAD, our method closes or surpasses the reported accuracy while operating natively on sparse temporal representations. We include a thorough analysis of the convergence behaviour across all 15 training epochs, domain transfer characteristics, and a quantitative comparison with representative related work.

Related papers

• Event-Based Motion Estimation via Oriented Distance Fields [55.23416502666468]
  Event-based motion estimation is central to tasks that demand high temporal resolution and robustness to fast motion.<n>We propose Oriented Distance Field Motion Estimation (ODF Motion Estimation), which replaces this optimization with a single averaging step over a precomputed field of event distance vectors.<n>On public and self-collected datasets, ODF motion estimation reaches sub-pixel accuracy at the lowest latency among compared methods.
  arXiv  Detail & Related papers  (2026-08-25T08:27:42Z)
• Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction [4.8915598762204135]
  ADAPT (Adaptive Domain-Aware Pedestrian Crossing Transformer) is a multimodal framework for predicting pedestrian crossing intention.<n> ADAPT processes four aligned visual modalities, including RGB images, local depth maps, global semantic maps, and global depth maps.<n>Experiments on the JAAD and PIE benchmark datasets demonstrate that ADAPT consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-14T03:05:09Z)
• EagleVision: A Multi-Task Benchmark for Cross-Domain Perception in High-Speed Autonomous Racing [9.483261915968617]
  EagleVision is a unified LiDAR-based benchmark for 3D detection and trajectory prediction in high-speed racing.<n>It provides newly annotated 3D bounding boxes for the Indy Autonomous Challenge dataset and the A2RL Real competition dataset.<n>Using a dataset-centric transfer framework, we quantify cross-domain generalization across urban, simulator, and real racing domains.
  arXiv  Detail & Related papers  (2026-04-13T12:42:18Z)
• Physics-Informed Reinforcement Learning of Spatial Density Velocity Potentials for Map-Free Racing [73.88859384645264]
  Out-Of-Distribution (OOD) generalization to various racetracks utilizes Machine Learning (ML) to encode the mathematical relation between sensor data and vehicle actuation for end-to-end control.<n>This paper presents a DRL method that parameterizes nonlinear vehicle dynamics from the spectral distribution of depth measurements with a non-geometric, physics-informed reward, to infer vehicle time-optimal and overtaking racing controls with an Artificial Neural Network (ANN)<n>The policy outperforms human demonstrations by 12% in OOD tracks on proportionally scaled hardware, by maximizing the friction circle with tire dynamics that resemble an empirical Pacejka tire model.
  arXiv  Detail & Related papers  (2026-04-10T17:12:07Z)
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail [85.47497935739936]
  Alpamayo-R1 (AR1) is a vision-language-action model that integrates Chain of Causation reasoning with trajectory planning.<n>We show AR1 achieves 12% improvement in planning accuracy on challenging cases compared to a trajectory-only baseline.<n>We plan to release AR1 models and a subset of the CoC in a future update.
  arXiv  Detail & Related papers  (2025-10-30T01:25:34Z)
• KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models [19.625631486595505]
  This paper introduces KEPT, a knowledge-enhanced vision-language framework.<n>It predicts ego trajectories directly from consecutive front-view driving frames.<n>It achieves state-of-the-art performance across open-loop protocols.
  arXiv  Detail & Related papers  (2025-09-03T03:10:42Z)
• DEEGITS: Deep Learning based Framework for Measuring Heterogenous Traffic State in Challenging Traffic Scenarios [0.0]
  This paper presents DEEGITS (Deep Heterogeneous Traffic State Measurement), a comprehensive framework that leverages state-of-the-art convolutional neural network (CNN) techniques to accurately and rapidly detect vehicles and pedestrians. In this study, we enhance the training dataset through data fusion, enabling simultaneous detection vehicles and pedestrians. The framework is tested to measure heterogeneous traffic states in mixed traffic conditions.
  arXiv  Detail & Related papers  (2024-11-13T04:49:32Z)
• Real-time Object Detection for Streaming Perception [84.2559631820007]
  Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. We build a simple and effective framework for streaming perception. Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
  arXiv  Detail & Related papers  (2022-03-23T11:33:27Z)
• Real Time Monocular Vehicle Velocity Estimation using Synthetic Data [78.85123603488664]
  We look at the problem of estimating the velocity of road vehicles from a camera mounted on a moving car. We propose a two-step approach where first an off-the-shelf tracker is used to extract vehicle bounding boxes and then a small neural network is used to regress the vehicle velocity.
  arXiv  Detail & Related papers  (2021-09-16T13:10:27Z)
• Is attention to bounding boxes all you need for pedestrian action prediction? [1.3999481573773074]
  We present a framework based on multiple variations of the Transformer models to reason attentively about the dynamic evolution of the pedestrians' past trajectory. We prove that using only bounding boxes as input to our model can outperform the previous state-of-the-art models. Our model has similarly reached high accuracy (91 and F1-score (0.91) on this dataset.
  arXiv  Detail & Related papers  (2021-07-16T17:47:32Z)
• Prediction of Traffic Flow via Connected Vehicles [77.11902188162458]
  We propose a Short-term Traffic flow Prediction framework so that transportation authorities take early actions to control flow and prevent congestion. We anticipate flow at future time frames on a target road segment based on historical flow data and innovative features such as real time feeds and trajectory data provided by Connected Vehicles (CV) technology. We show how this novel approach allows advanced modelling by integrating into the forecasting of flow, the impact of various events that CV realistically encountered on segments along their trajectory.
  arXiv  Detail & Related papers  (2020-07-10T16:00:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.