Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware

• URL: http://arxiv.org/abs/2312.00425v2
• Date: Wed, 17 Apr 2024 06:26:04 GMT
• Title: Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware
• Abstract summary: This paper introduces a neuromorphic methodology for eye tracking, harnessing pure event data captured by a Dynamic Vision Sensor (DVS) camera. The framework integrates a directly trained Spiking Neuron Network (SNN) regression model and leverages a state-of-the-art low power edge neuromorphic processor - Speck.
• Score: 10.702811797536667
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a neuromorphic methodology for eye tracking, harnessing pure event data captured by a Dynamic Vision Sensor (DVS) camera. The framework integrates a directly trained Spiking Neuron Network (SNN) regression model and leverages a state-of-the-art low power edge neuromorphic processor - Speck, collectively aiming to advance the precision and efficiency of eye-tracking systems. First, we introduce a representative event-based eye-tracking dataset, "Ini-30", which was collected with two glass-mounted DVS cameras from thirty volunteers. Then,a SNN model, based on Integrate And Fire (IAF) neurons, named "Retina", is described , featuring only 64k parameters (6.63x fewer than the latest) and achieving pupil tracking error of only 3.24 pixels in a 64x64 DVS input. The continous regression output is obtained by means of convolution using a non-spiking temporal 1D filter slided across the output spiking layer. Finally, we evaluate Retina on the neuromorphic processor, showing an end-to-end power between 2.89-4.8 mW and a latency of 5.57-8.01 mS dependent on the time window. We also benchmark our model against the latest event-based eye-tracking method, "3ET", which was built upon event frames. Results show that Retina achieves superior precision with 1.24px less pupil centroid error and reduced computational complexity with 35 times fewer MAC operations. We hope this work will open avenues for further investigation of close-loop neuromorphic solutions and true event-based training pursuing edge performance.

Related papers

• EdgeSpike: Spiking Neural Networks for Low-Power Autonomous Sensing in Edge IoT Architectures [0.0]
  EdgeSpike is a co-designed spiking neural network (SNN) framework for autonomous low-power sensing in edge Internet of Things (IoT) architectures.<n>It unifies a hybrid surrogate-gradient and direct-encoding training pipeline, a hardware-aware neural architecture search, and an event-driven runtime.<n>It achieves a mean classification accuracy of 91.4%, within 1.2 percentage points (pp) of strong INT8 convolutional neural network (CNN) baselines.
  arXiv  Detail & Related papers  (2026-04-29T05:15:28Z)
• No Dense Tensors Needed: Fully Sparse Object Detection on Event-Camera Voxel Grids [0.3823356975862005]
  Event cameras produce asynchronous, high-dynamic-range streams well suited for detecting small, fast-moving drones.<n>Most event-based detectors convert the sparse event stream into dense tensors, discarding the representational efficiency of neuromorphic sensing.<n>We propose SparseVoxelDet, the first fully sparse object detector for event cameras, in which backbone feature extraction, feature pyramid fusion, and the detection head all operate exclusively on occupied voxel positions.
  arXiv  Detail & Related papers  (2026-03-23T07:12:14Z)
• Neuromorphic Eye Tracking for Low-Latency Pupil Detection [37.091037454305024]
  Eye tracking for wearable systems demands low latency and milliwatt-level power.<n>Neuromorphic sensors and spiking neural networks (SNNs) offer a promising alternative.<n>This paper presents a neuromorphic version of top-performing event-based eye-tracking models.
  arXiv  Detail & Related papers  (2025-12-10T11:30:21Z)
• Privacy-preserving fall detection at the edge using Sony IMX636 event-based vision sensor and Intel Loihi 2 neuromorphic processor [1.4594259556596754]
  Fall detection for elderly care using non-invasive vision-based systems remains an important yet unsolved problem.<n>We propose a neuromorphic fall detection system that integrates the Sony IMX636 event-based vision sensor with the Intel Loihi 2 neuromorphic processor.<n>Our smart security camera proof-of-concept highlights the potential of integrating neuromorphic sensing and processing for edge AI applications.
  arXiv  Detail & Related papers  (2025-11-27T15:44:55Z)
• Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection [15.154553304520164]
  Spiking Neural Networks (SNNs) have bio-plaus and low-power advantages over Artificial Neural Networks (ANNs) In this work, we focus on bridging the performance gap between ANNs and SNNs on object detection. We design a SpikeYOLO architecture to solve this problem by simplifying the vanilla YOLO and incorporating meta SNN blocks.
  arXiv  Detail & Related papers  (2024-07-30T10:04:16Z)
• TENNs-PLEIADES: Building Temporal Kernels with Orthogonal Polynomials [1.1970409518725493]
  We focus on interfacing these networks with event-based data to perform online classification and detection with low latency. We experimented with three event-based benchmarks and obtained state-of-the-art results on all three by large margins with significantly smaller memory and compute costs.
  arXiv  Detail & Related papers  (2024-05-20T17:06:24Z)
• Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN [8.613703056677457]
  Eye-tracking technology is integral to numerous consumer electronics applications, particularly in virtual and augmented reality (VR/AR) Yet, achieving optimal performance across all these fronts presents a formidable challenge. We tackle this challenge through a synergistic software/ hardware co-design of the system with an event camera. Our system achieves 81% p5 accuracy, 99.5% p10 accuracy, and 3.71 Meanean Distance with 0.7 ms latency while only consuming 2.29 mJ per inference.
  arXiv  Detail & Related papers  (2024-04-22T15:28:42Z)
• Severity classification of ground-glass opacity via 2-D convolutional neural network and lung CT scans: a 3-day exploration [0.0]
  Ground-glass opacity is a hallmark of numerous lung diseases, including patients with COVID19 and pneumonia, pulmonary fibrosis, and tuberculosis. This note presents experimental results of a proof-of-concept framework that got implemented and tested over three days as driven by the third challenge entitled "COVID-19 Competition" As part of the challenge requirement, the source code produced during the course of this exercise is posted at https://github.com/lisatwyw/cov19.
  arXiv  Detail & Related papers  (2023-03-23T22:35:37Z)
• Recurrent Vision Transformers for Object Detection with Event Cameras [62.27246562304705]
  We present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection. Our study brings new insights into effective design choices that can be fruitful for research beyond event-based vision.
  arXiv  Detail & Related papers  (2022-12-11T20:28:59Z)
• Pushing the Limits of Asynchronous Graph-based Object Detection with Event Cameras [62.70541164894224]
  We introduce several architecture choices which allow us to scale the depth and complexity of such models while maintaining low computation. Our method runs 3.7 times faster than a dense graph neural network, taking only 8.4 ms per forward pass.
  arXiv  Detail & Related papers  (2022-11-22T15:14:20Z)
• LiteDepth: Digging into Fast and Accurate Depth Estimation on Mobile Devices [45.84356762066717]
  We develop an end-to-end learning-based model with a tiny weight size (1.4MB) and a short inference time (27FPS on Raspberry Pi 4. We propose a simple yet effective data augmentation strategy, called R2 crop, to boost the model performance. Notably, our solution named LiteDepth ranks 2nd in the MAI&AIM2022 Monocular Depth Estimation Challenge, with a si-RMSE of 0.311, an RMSE of 3.79, and the inference time is 37$ms$ tested on the Raspberry Pi 4.
  arXiv  Detail & Related papers  (2022-09-02T11:38:28Z)
• EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm [111.17100512647619]
  This paper explains the rationality of Vision Transformer by analogy with the proven practical evolutionary algorithm (EA) We propose a novel pyramid EATFormer backbone that only contains the proposed EA-based transformer (EAT) block. Massive quantitative and quantitative experiments on image classification, downstream tasks, and explanatory experiments demonstrate the effectiveness and superiority of our approach.
  arXiv  Detail & Related papers  (2022-06-19T04:49:35Z)
• Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining [58.10436813430554]
  Mini-batch training of graph neural networks (GNNs) requires a lot of computation and data movement. We argue in favor of performing mini-batch training with neighborhood sampling in a distributed multi-GPU environment. We present a sequence of improvements to mitigate these bottlenecks, including a performance-engineered neighborhood sampler. We also conduct an empirical analysis that supports the use of sampling for inference, showing that test accuracies are not materially compromised.
  arXiv  Detail & Related papers  (2021-10-16T02:41:35Z)
• Fast Motion Understanding with Spatiotemporal Neural Networks and Dynamic Vision Sensors [99.94079901071163]
  This paper presents a Dynamic Vision Sensor (DVS) based system for reasoning about high speed motion. We consider the case of a robot at rest reacting to a small, fast approaching object at speeds higher than 15m/s. We highlight the results of our system to a toy dart moving at 23.4m/s with a 24.73deg error in $theta$, 18.4mm average discretized radius prediction error, and 25.03% median time to collision prediction error.
  arXiv  Detail & Related papers  (2020-11-18T17:55:07Z)
• Tracking Objects as Points [83.9217787335878]
  We present a simultaneous detection and tracking algorithm that is simpler, faster, and more accurate than the state of the art. Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. CenterTrack is simple, online (no peeking into the future), and real-time.
  arXiv  Detail & Related papers  (2020-04-02T17:58:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.