Revisiting Feature Prediction for Learning Visual Representations from Video

• URL: http://arxiv.org/abs/2404.08471v1
• Date: Thu, 15 Feb 2024 18:59:11 GMT
• Title: Revisiting Feature Prediction for Learning Visual Representations from Video
• Authors: Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas,
• Abstract summary: V-JEPA is a collection of vision models trained solely using a feature prediction objective. The models are trained on 2 million videos collected from public datasets. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks.
• Score: 62.08833572467379
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision. The models are trained on 2 million videos collected from public datasets and are evaluated on downstream image and video tasks. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks, without adaption of the model's parameters; e.g., using a frozen backbone. Our largest model, a ViT-H/16 trained only on videos, obtains 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet1K.

Related papers

• HVM-1: Large-scale video models pretrained with nearly 5000 hours of human-like video data [10.225358400539722]
  We release two 633M parameter models trained at spatial resolutions of 224x and 448x pixels. We evaluate the performance of these models in downstream few-shot video and image recognition tasks. HVM-1 models learn more accurate and more robust object representations compared to models pretrained with the image-based MAE algorithm.
  arXiv  Detail & Related papers  (2024-07-25T14:21:50Z)
• Advancing Human Action Recognition with Foundation Models trained on Unlabeled Public Videos [2.3247413495885647]
  We use 283,582 unique, unlabeled TikTok video clips, categorized into 386 hashtags, to train a domain-specific foundation model for action recognition. Our model achieves state-of-the-art results: 99.05% on UCF101, 86.08% on HMDB51, 85.51% on Kinetics-400, and 74.27% on Something-Something V2 using the ViT-giant backbone.
  arXiv  Detail & Related papers  (2024-02-14T00:41:10Z)
• ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition [4.36572039512405]
  We present the first pose augmented Vision-language model (VLM) for Video Action Recognition. Notably, our scheme achieves an accuracy of 92.81% and 73.02% on two popular human video action recognition benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-07T20:50:54Z)
• Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning [123.63301596019522]
  Masked video distillation (MVD) is a simple yet effective two-stage masked feature modeling framework for video representation learning. For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks. We design a spatial-temporal co-teaching method for MVD to leverage the advantage of different teachers.
  arXiv  Detail & Related papers  (2022-12-08T18:59:59Z)
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning [52.69422763715118]
  We present general video foundation models, InternVideo, for dynamic and complex video-level understanding tasks. InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives. InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications.
  arXiv  Detail & Related papers  (2022-12-06T18:09:49Z)
• Could Giant Pretrained Image Models Extract Universal Representations? [94.97056702288317]
  We present a study of frozen pretrained models when applied to diverse and representative computer vision tasks. Our work answers the questions of what pretraining task fits best with this frozen setting, how to make the frozen setting more flexible to various downstream tasks, and the effect of larger model sizes.
  arXiv  Detail & Related papers  (2022-11-03T17:57:10Z)
• PreViTS: Contrastive Pretraining with Video Tracking Supervision [53.73237606312024]
  PreViTS is an unsupervised SSL framework for selecting clips containing the same object. PreViTS spatially constrains the frame regions to learn from and trains the model to locate meaningful objects. We train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS.
  arXiv  Detail & Related papers  (2021-12-01T19:49:57Z)
• RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning [100.76672109782815]
  We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only. It is difficult to construct a suitable self-supervised task to well model both motion and appearance features. We propose a new way to perceive the playback speed and exploit the relative speed between two video clips as labels.
  arXiv  Detail & Related papers  (2020-10-27T16:42:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.