Self-Supervised Representation Learning from Temporal Ordering of
Automated Driving Sequences
- URL: http://arxiv.org/abs/2302.09043v3
- Date: Wed, 8 Nov 2023 18:57:10 GMT
- Title: Self-Supervised Representation Learning from Temporal Ordering of
Automated Driving Sequences
- Authors: Christopher Lang, Alexander Braun, Lars Schillingmann, Karsten Haug,
Abhinav Valada
- Abstract summary: We propose TempO, a temporal ordering pretext task for pre-training region-level feature representations for perception tasks.
We embed each frame by an unordered set of proposal feature vectors, a representation that is natural for object detection or tracking systems.
Extensive evaluations on the BDD100K, nuImages, and MOT17 datasets show that our TempO pre-training approach outperforms single-frame self-supervised learning methods.
- Score: 49.91741677556553
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Self-supervised feature learning enables perception systems to benefit from
the vast raw data recorded by vehicle fleets worldwide. While video-level
self-supervised learning approaches have shown strong generalizability on
classification tasks, the potential to learn dense representations from
sequential data has been relatively unexplored. In this work, we propose TempO,
a temporal ordering pretext task for pre-training region-level feature
representations for perception tasks. We embed each frame by an unordered set
of proposal feature vectors, a representation that is natural for object
detection or tracking systems, and formulate the sequential ordering by
predicting frame transition probabilities in a transformer-based multi-frame
architecture whose complexity scales less than quadratic with respect to the
sequence length. Extensive evaluations on the BDD100K, nuImages, and MOT17
datasets show that our TempO pre-training approach outperforms single-frame
self-supervised learning methods as well as supervised transfer learning
initialization strategies, achieving an improvement of +0.7% in mAP for object
detection and +2.0% in the HOTA score for multi-object tracking.
Related papers
- Learning a Neural Association Network for Self-supervised Multi-Object Tracking [34.07776597698471]
This paper introduces a novel framework to learn data association for multi-object tracking in a self-supervised manner.
Motivated by the fact that in real-world scenarios object motion can be usually represented by a Markov process, we present a novel expectation (EM) algorithm that trains a neural network to associate detections for tracking.
We evaluate our approach on the challenging MOT17 and MOT20 datasets and achieve state-of-the-art results in comparison to self-supervised trackers.
arXiv Detail & Related papers (2024-11-18T12:22:29Z) - Skeleton2vec: A Self-supervised Learning Framework with Contextualized
Target Representations for Skeleton Sequence [56.092059713922744]
We show that using high-level contextualized features as prediction targets can achieve superior performance.
Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework.
Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
arXiv Detail & Related papers (2024-01-01T12:08:35Z) - Self-Supervised Multi-Object Tracking For Autonomous Driving From
Consistency Across Timescales [53.55369862746357]
Self-supervised multi-object trackers have tremendous potential as they enable learning from raw domain-specific data.
However, their re-identification accuracy still falls short compared to their supervised counterparts.
We propose a training objective that enables self-supervised learning of re-identification features from multiple sequential frames.
arXiv Detail & Related papers (2023-04-25T20:47:29Z) - Towards Sequence-Level Training for Visual Tracking [60.95799261482857]
This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning.
Four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training.
arXiv Detail & Related papers (2022-08-11T13:15:36Z) - Hierarchically Self-Supervised Transformer for Human Skeleton
Representation Learning [45.13060970066485]
We propose a self-supervised hierarchical pre-training scheme incorporated into a hierarchical Transformer-based skeleton sequence encoder (Hi-TRS)
Under both supervised and semi-supervised evaluation protocols, our method achieves the state-of-the-art performance.
arXiv Detail & Related papers (2022-07-20T04:21:05Z) - SeCo: Exploring Sequence Supervision for Unsupervised Representation
Learning [114.58986229852489]
In this paper, we explore the basic and generic supervision in the sequence from spatial, sequential and temporal perspectives.
We derive a particular form named Contrastive Learning (SeCo)
SeCo shows superior results under the linear protocol on action recognition, untrimmed activity recognition and object tracking.
arXiv Detail & Related papers (2020-08-03T15:51:35Z) - One-Shot Object Detection without Fine-Tuning [62.39210447209698]
We introduce a two-stage model consisting of a first stage Matching-FCOS network and a second stage Structure-Aware Relation Module.
We also propose novel training strategies that effectively improve detection performance.
Our method exceeds the state-of-the-art one-shot performance consistently on multiple datasets.
arXiv Detail & Related papers (2020-05-08T01:59:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.