Event Camera Data Pre-training

• URL: http://arxiv.org/abs/2301.01928v3
• Date: Thu, 20 Jul 2023 05:21:04 GMT
• Title: Event Camera Data Pre-training
• Authors: Yan Yang and Liyuan Pan and Liu Liu
• Abstract summary: Our model is a self-supervised learning framework, and uses paired event camera data and natural RGB images for training. We achieve top-1 accuracy at 64.83% on the N-ImageNet dataset.
• Score: 14.77724035068357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes a pre-trained neural network for handling event camera data. Our model is a self-supervised learning framework, and uses paired event camera data and natural RGB images for training. Our method contains three modules connected in a sequence: i) a family of event data augmentations, generating meaningful event images for self-supervised training; ii) a conditional masking strategy to sample informative event patches from event images, encouraging our model to capture the spatial layout of a scene and accelerating training; iii) a contrastive learning approach, enforcing the similarity of embeddings between matching event images, and between paired event and RGB images. An embedding projection loss is proposed to avoid the model collapse when enforcing the event image embedding similarities. A probability distribution alignment loss is proposed to encourage the event image to be consistent with its paired RGB image in the feature space. Transfer learning performance on downstream tasks shows the superiority of our method over state-of-the-art methods. For example, we achieve top-1 accuracy at 64.83% on the N-ImageNet dataset.

Related papers

• EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting [76.02450110026747]
  Event cameras, inspired by biological vision, record pixel-wise intensity changes asynchronously with high temporal resolution. We propose Event-Aided Free-Trajectory 3DGS, which seamlessly integrates the advantages of event cameras into 3DGS. We evaluate our method on the public Tanks and Temples benchmark and a newly collected real-world dataset, RealEv-DAVIS.
  arXiv  Detail & Related papers  (2024-10-20T13:44:24Z)
• Evaluating Image-Based Face and Eye Tracking with Event Cameras [9.677797822200965]
  Event Cameras, also known as Neuromorphic sensors, capture changes in local light intensity at the pixel level, producing asynchronously generated data termed events'' This data format mitigates common issues observed in conventional cameras, like under-sampling when capturing fast-moving objects. We evaluate the viability of integrating conventional algorithms with event-based data, transformed into a frame format.
  arXiv  Detail & Related papers  (2024-08-19T20:27:08Z)
• EZSR: Event-based Zero-Shot Recognition [21.10165234725309]
  This paper studies zero-shot object recognition using event camera data. We develop an event encoder without relying on additional reconstruction networks. We achieve 47.84% zero-shot accuracy on the N-ImageNet dataset.
  arXiv  Detail & Related papers  (2024-07-31T14:06:06Z)
• MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images [57.71600854525037]
  We propose a Fuse-Describe-Match strategy for 6D pose estimation from RGB-D images. MatchU is a generic approach that fuses 2D texture and 3D geometric cues for 6D pose prediction of unseen objects.
  arXiv  Detail & Related papers  (2024-03-03T14:01:03Z)
• Segment Any Events via Weighted Adaptation of Pivotal Tokens [85.39087004253163]
  This paper focuses on the nuanced challenge of tailoring the Segment Anything Models (SAMs) for integration with event data. We introduce a multi-scale feature distillation methodology to optimize the alignment of token embeddings originating from event data with their RGB image counterparts.
  arXiv  Detail & Related papers  (2023-12-24T12:47:08Z)
• Event Camera Data Dense Pre-training [10.918407820258246]
  This paper introduces a self-supervised learning framework designed for pre-training neural networks tailored to dense prediction tasks using event camera data. For training our framework, we curate a synthetic event camera dataset featuring diverse scene and motion patterns.
  arXiv  Detail & Related papers  (2023-11-20T04:36:19Z)
• GET: Group Event Transformer for Event-Based Vision [82.312736707534]
  Event cameras are a type of novel neuromorphic sen-sor that has been gaining increasing attention. We propose a novel Group-based vision Transformer backbone for Event-based vision, called Group Event Transformer (GET) GET de-couples temporal-polarity information from spatial infor-mation throughout the feature extraction process.
  arXiv  Detail & Related papers  (2023-10-04T08:02:33Z)
• Masked Event Modeling: Self-Supervised Pretraining for Event Cameras [41.263606382601886]
  Masked Event Modeling (MEM) is a self-supervised framework for events. MEM pretrains a neural network on unlabeled events, which can originate from any event camera recording. Our method reaches state-of-the-art classification accuracy across three datasets.
  arXiv  Detail & Related papers  (2022-12-20T15:49:56Z)
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation [57.22705137545853]
  We propose a task transfer method that allows models to be trained directly with labeled images and unlabeled event data. We leverage the generative event model to split event features into content and motion features. Our approach unlocks the vast amount of existing image datasets for the training of event-based neural networks.
  arXiv  Detail & Related papers  (2021-09-06T17:31:37Z)
• Matching Neuromorphic Events and Color Images via Adversarial Learning [49.447580124957966]
  We propose the Event-Based Image Retrieval (EBIR) problem to exploit the cross-modal matching task. We address the EBIR problem by proposing neuromorphic Events-Color image Feature Learning (ECFL) We also contribute to the community N-UKbench and EC180 dataset to promote the development of EBIR problem.
  arXiv  Detail & Related papers  (2020-03-02T02:48:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.