Self-Supervised Video Similarity Learning
- URL: http://arxiv.org/abs/2304.03378v2
- Date: Fri, 16 Jun 2023 14:11:58 GMT
- Title: Self-Supervised Video Similarity Learning
- Authors: Giorgos Kordopatis-Zilos and Giorgos Tolias and Christos Tzelepis and
Ioannis Kompatsiaris and Ioannis Patras and Symeon Papadopoulos
- Abstract summary: We introduce S$2$VS, a video similarity learning approach with self-supervision.
We learn a single universal model that achieves state-of-the-art performance on all tasks.
- Score: 35.512588398849395
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce S$^2$VS, a video similarity learning approach with
self-supervision. Self-Supervised Learning (SSL) is typically used to train
deep models on a proxy task so as to have strong transferability on target
tasks after fine-tuning. Here, in contrast to prior work, SSL is used to
perform video similarity learning and address multiple retrieval and detection
tasks at once with no use of labeled data. This is achieved by learning via
instance-discrimination with task-tailored augmentations and the widely used
InfoNCE loss together with an additional loss operating jointly on
self-similarity and hard-negative similarity. We benchmark our method on tasks
where video relevance is defined with varying granularity, ranging from video
copies to videos depicting the same incident or event. We learn a single
universal model that achieves state-of-the-art performance on all tasks,
surpassing previously proposed methods that use labeled data. The code and
pretrained models are publicly available at: https://github.com/gkordo/s2vs
Related papers
- iBoot: Image-bootstrapped Self-Supervised Video Representation Learning [45.845595749486215]
Video self-supervised learning (SSL) suffers from added challenges: video datasets are typically not as large as image datasets.
We propose to utilize a strong image-based model, pre-trained with self- or language supervision, in a video representation learning framework.
The proposed algorithm is shown to learn much more efficiently in less epochs and with a smaller batch.
arXiv Detail & Related papers (2022-06-16T17:42:48Z) - VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive
Learning [82.09856883441044]
Video understanding relies on perceiving the global content modeling its internal connections.
We propose a block-wise strategy where we mask neighboring video tokens in both spatial and temporal domains.
We also add an augmentation-free contrastive learning method to further capture global content.
arXiv Detail & Related papers (2021-06-21T16:48:19Z) - ASCNet: Self-supervised Video Representation Learning with
Appearance-Speed Consistency [62.38914747727636]
We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information.
Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other.
In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
arXiv Detail & Related papers (2021-06-04T08:44:50Z) - CoCon: Cooperative-Contrastive Learning [52.342936645996765]
Self-supervised visual representation learning is key for efficient video analysis.
Recent success in learning image representations suggests contrastive learning is a promising framework to tackle this challenge.
We introduce a cooperative variant of contrastive learning to utilize complementary information across views.
arXiv Detail & Related papers (2021-04-30T05:46:02Z) - Composable Augmentation Encoding for Video Representation Learning [94.2358972764708]
We focus on contrastive methods for self-supervised video representation learning.
A common paradigm in contrastive learning is to construct positive pairs by sampling different data views for the same instance, with different data instances as negatives.
We propose an 'augmentation aware' contrastive learning framework, where we explicitly provide a sequence of augmentation parameterisations.
We show that our method encodes valuable information about specified spatial or temporal augmentation, and in doing so also achieve state-of-the-art performance on a number of video benchmarks.
arXiv Detail & Related papers (2021-04-01T16:48:53Z) - Adversarial Training of Variational Auto-encoders for Continual
Zero-shot Learning [1.90365714903665]
We present a hybrid network that consists of a shared VAE module to hold information of all tasks and task-specific private VAE modules for each task.
The model's size grows with each task to prevent catastrophic forgetting of task-specific skills.
We show our method is superior on class sequentially learning with ZSL(Zero-Shot Learning) and GZSL(Generalized Zero-Shot Learning)
arXiv Detail & Related papers (2021-02-07T11:21:24Z) - Unsupervised Learning of Video Representations via Dense Trajectory
Clustering [86.45054867170795]
This paper addresses the task of unsupervised learning of representations for action recognition in videos.
We first propose to adapt two top performing objectives in this class - instance recognition and local aggregation.
We observe promising performance, but qualitative analysis shows that the learned representations fail to capture motion patterns.
arXiv Detail & Related papers (2020-06-28T22:23:03Z) - Rethinking Zero-shot Video Classification: End-to-end Training for
Realistic Applications [26.955001807330497]
Zero-shot learning (ZSL) trains a model once and generalizes to new tasks whose classes are not present in the training dataset.
We propose the first end-to-end algorithm for ZSL in video classification.
Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features.
arXiv Detail & Related papers (2020-03-03T11:09:59Z) - Evolving Losses for Unsupervised Video Representation Learning [91.2683362199263]
We present a new method to learn video representations from large-scale unlabeled video data.
The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods.
arXiv Detail & Related papers (2020-02-26T16:56:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.