Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset

• URL: http://arxiv.org/abs/2403.05839v3
• Date: Wed, 06 Aug 2025 10:45:11 GMT
• Title: Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset
• Authors: Xiao Wang, Xufeng Lou, Shiao Wang, Ju Huang, Lan Chen, Bo Jiang,
• Abstract summary: We propose a new long-term, large-scale frame-event visual object tracking dataset, termed FELT.<n>We also propose a novel Associative Memory Transformer based RGB-Event long-term visual tracker, termed AMTTrack.
• Score: 9.366068518600583
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing event stream based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear. In this paper, we first propose a new long-term, large-scale frame-event visual object tracking dataset, termed FELT. It contains 1,044 long-term videos that involve 1.9 million RGB frames and event stream pairs, 60 different target objects, and 14 challenging attributes. To build a solid benchmark, we retrain and evaluate 21 baseline trackers on our dataset for future work to compare. In addition, we propose a novel Associative Memory Transformer based RGB-Event long-term visual tracker, termed AMTTrack. It follows a one-stream tracking framework and aggregates the multi-scale RGB/event template and search tokens effectively via the Hopfield retrieval layer. The framework also embodies another aspect of associative memory by maintaining dynamic template representations through an associative memory update scheme, which addresses the appearance variation in long-term tracking. Extensive experiments on FELT, FE108, VisEvent, and COESOT datasets fully validated the effectiveness of our proposed tracker. Both the dataset and source code will be released on https://github.com/Event-AHU/FELT_SOT_Benchmark

Related papers

• Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking [9.353589376846902]
  We propose an efficient RGB-Event object tracking framework based on the linear-complexity Vision Mamba network.<n>The source code and pre-trained models will be released at https://github.com/Event-AHU/Mamba_FETrack.
  arXiv  Detail & Related papers  (2025-06-30T12:24:01Z)
• Adversarial Attack for RGB-Event based Visual Object Tracking [39.35874495297647]
  We propose a cross-modal adversarial attack algorithm for RGB-Event visual tracking. We evaluate the proposed approach against attacks on three widely used RGB-Event Tracking datasets.
  arXiv  Detail & Related papers  (2025-04-19T23:35:19Z)
• BlinkTrack: Feature Tracking over 100 FPS via Events and Images [50.98675227695814]
  We propose a novel framework, BlinkTrack, which integrates event data with RGB images for high-frequency feature tracking. Our method extends the traditional Kalman filter into a learning-based framework, utilizing differentiable Kalman filters in both event and image branches. Experimental results indicate that BlinkTrack significantly outperforms existing event-based methods.
  arXiv  Detail & Related papers  (2024-09-26T15:54:18Z)
• TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking [30.89375068036783]
  Existing approaches perform event feature extraction for RGB-E tracking using traditional appearance models. We propose an Event backbone (Pooler) to obtain a high-quality feature representation that is cognisant of the intrinsic characteristics of the event data. Our method significantly outperforms state-of-the-art trackers on two widely used RGB-E tracking datasets.
  arXiv  Detail & Related papers  (2024-05-08T12:19:08Z)
• SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features [52.213656737672935]
  SpikeMOT is an event-based multi-object tracker. SpikeMOT uses spiking neural networks to extract sparsetemporal features from event streams associated with objects.
  arXiv  Detail & Related papers  (2023-09-29T05:13:43Z)
• Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline [38.42400442371156]
  Existing works either utilize aligned RGB and event data for accurate tracking or directly learn an event-based tracker. We propose a novel hierarchical knowledge distillation framework that can fully utilize multi-modal / multi-view information during training to facilitate knowledge transfer. We propose the first large-scale high-resolution ($1280 times 720$) dataset named EventVOT. It contains 1141 videos and covers a wide range of categories such as pedestrians, vehicles, UAVs, ping pongs, etc.
  arXiv  Detail & Related papers  (2023-09-26T01:42:26Z)
• GLEN: General-Purpose Event Detection for Thousands of Types [80.99866527772512]
  We build a general-purpose event detection dataset GLEN, which covers 205K event mentions with 3,465 different types. GLEN is 20x larger in ontology than today's largest event dataset. We also propose a new multi-stage event detection model CEDAR specifically designed to handle the large size in GLEN.
  arXiv  Detail & Related papers  (2023-03-16T05:36:38Z)
• DIVOTrack: A Novel Dataset and Baseline Method for Cross-View Multi-Object Tracking in DIVerse Open Scenes [74.64897845999677]
  We introduce a new cross-view multi-object tracking dataset for DIVerse Open scenes with dense tracking pedestrians. Our DIVOTrack has fifteen distinct scenarios and 953 cross-view tracks, surpassing all cross-view multi-object tracking datasets currently available. Furthermore, we provide a novel baseline cross-view tracking method with a unified joint detection and cross-view tracking framework named CrossMOT.
  arXiv  Detail & Related papers  (2023-02-15T14:10:42Z)
• Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric [53.88188265943762]
  We propose a single-stage backbone network for Color-Event Unified Tracking (CEUTrack), which achieves the above functions simultaneously. Our proposed CEUTrack is simple, effective, and efficient, which achieves over 75 FPS and new SOTA performance.
  arXiv  Detail & Related papers  (2022-11-20T16:01:31Z)
• RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking [30.448658049744775]
  Given a limited amount of annotated RGB-D tracking data, most state-of-the-art RGB-D trackers are simple extensions of high-performance RGB-only trackers. To address the dataset deficiency issue, a new RGB-D dataset named RGBD1K is released in this paper.
  arXiv  Detail & Related papers  (2022-08-21T03:07:36Z)
• Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline [80.13652104204691]
  In this paper, we construct a large-scale benchmark with high diversity for visible-thermal UAV tracking (VTUAV) We provide a coarse-to-fine attribute annotation, where frame-level attributes are provided to exploit the potential of challenge-specific trackers. In addition, we design a new RGB-T baseline, named Hierarchical Multi-modal Fusion Tracker (HMFT), which fuses RGB-T data in various levels.
  arXiv  Detail & Related papers  (2022-04-08T15:22:33Z)
• DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion [56.1428110894411]
  We propose a large-scale dataset for multi-human tracking, where humans have similar appearance, diverse motion and extreme articulation. As the dataset contains mostly group dancing videos, we name it "DanceTrack" We benchmark several state-of-the-art trackers on our dataset and observe a significant performance drop on DanceTrack when compared against existing benchmarks.
  arXiv  Detail & Related papers  (2021-11-29T16:49:06Z)
• Learning Dynamic Compact Memory Embedding for Deformable Visual Object Tracking [82.34356879078955]
  We propose a compact memory embedding to enhance the discrimination of the segmentation-based deformable visual tracking method. Our method outperforms the excellent segmentation-based trackers, i.e., D3S and SiamMask on DAVIS 2017 benchmark.
  arXiv  Detail & Related papers  (2021-11-23T03:07:12Z)
• Multi-Object Tracking and Segmentation with a Space-Time Memory Network [12.043574473965318]
  We propose a method for multi-object tracking and segmentation based on a novel memory-based mechanism to associate tracklets. The proposed tracker, MeNToS, addresses particularly the long-term data association problem.
  arXiv  Detail & Related papers  (2021-10-21T17:13:17Z)
• Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking [20.66906781151]
  In multi-object tracking, the tracker maintains in its memory the appearance and motion information for each object in the scene. Many approaches model each target in isolation and lack the ability to use all the targets in the scene to jointly update the memory. We propose a training strategy adapted to multi-track pooling which generates hard tracking episodes online.
  arXiv  Detail & Related papers  (2021-01-28T18:12:39Z)
• Learning Spatio-Appearance Memory Network for High-Performance Visual Tracking [79.80401607146987]
  Existing object tracking usually learns a bounding-box based template to match visual targets across frames, which cannot accurately learn a pixel-wise representation. This paper presents a novel segmentation-based tracking architecture, which is equipped with a local-temporal memory network to learn accurate-temporal correspondence.
  arXiv  Detail & Related papers  (2020-09-21T08:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.