Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequences

• URL: http://arxiv.org/abs/2302.09043v3
• Date: Wed, 8 Nov 2023 18:57:10 GMT
• Title: Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequences
• Authors: Christopher Lang, Alexander Braun, Lars Schillingmann, Karsten Haug, Abhinav Valada
• Abstract summary: We propose TempO, a temporal ordering pretext task for pre-training region-level feature representations for perception tasks. We embed each frame by an unordered set of proposal feature vectors, a representation that is natural for object detection or tracking systems. Extensive evaluations on the BDD100K, nuImages, and MOT17 datasets show that our TempO pre-training approach outperforms single-frame self-supervised learning methods.
• Score: 49.91741677556553
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised feature learning enables perception systems to benefit from the vast raw data recorded by vehicle fleets worldwide. While video-level self-supervised learning approaches have shown strong generalizability on classification tasks, the potential to learn dense representations from sequential data has been relatively unexplored. In this work, we propose TempO, a temporal ordering pretext task for pre-training region-level feature representations for perception tasks. We embed each frame by an unordered set of proposal feature vectors, a representation that is natural for object detection or tracking systems, and formulate the sequential ordering by predicting frame transition probabilities in a transformer-based multi-frame architecture whose complexity scales less than quadratic with respect to the sequence length. Extensive evaluations on the BDD100K, nuImages, and MOT17 datasets show that our TempO pre-training approach outperforms single-frame self-supervised learning methods as well as supervised transfer learning initialization strategies, achieving an improvement of +0.7% in mAP for object detection and +2.0% in the HOTA score for multi-object tracking.

Related papers

• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)
• Self-Supervised Multi-Object Tracking For Autonomous Driving From Consistency Across Timescales [53.55369862746357]
  Self-supervised multi-object trackers have tremendous potential as they enable learning from raw domain-specific data. However, their re-identification accuracy still falls short compared to their supervised counterparts. We propose a training objective that enables self-supervised learning of re-identification features from multiple sequential frames.
  arXiv  Detail & Related papers  (2023-04-25T20:47:29Z)
• Towards Sequence-Level Training for Visual Tracking [60.95799261482857]
  This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning. Four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training.
  arXiv  Detail & Related papers  (2022-08-11T13:15:36Z)
• Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning [45.13060970066485]
  We propose a self-supervised hierarchical pre-training scheme incorporated into a hierarchical Transformer-based skeleton sequence encoder (Hi-TRS) Under both supervised and semi-supervised evaluation protocols, our method achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-07-20T04:21:05Z)
• Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain [25.84756140221655]
  ConSecutive PreTraining (CSPT) is proposed based on the idea of not stopping pretraining in natural language processing (NLP) The proposed CSPT also can release the huge potential of unlabeled data for task-aware model training. The results show that by utilizing the proposed CSPT for task-aware model training, almost all downstream tasks in RSD can outperform the previous method of supervised pretraining-then-fine-tuning.
  arXiv  Detail & Related papers  (2022-07-08T12:32:09Z)
• SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning [114.58986229852489]
  In this paper, we explore the basic and generic supervision in the sequence from spatial, sequential and temporal perspectives. We derive a particular form named Contrastive Learning (SeCo) SeCo shows superior results under the linear protocol on action recognition, untrimmed activity recognition and object tracking.
  arXiv  Detail & Related papers  (2020-08-03T15:51:35Z)
• One-Shot Object Detection without Fine-Tuning [62.39210447209698]
  We introduce a two-stage model consisting of a first stage Matching-FCOS network and a second stage Structure-Aware Relation Module. We also propose novel training strategies that effectively improve detection performance. Our method exceeds the state-of-the-art one-shot performance consistently on multiple datasets.
  arXiv  Detail & Related papers  (2020-05-08T01:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.