HVM-1: Large-scale video models pretrained with nearly 5000 hours of human-like video data

• URL: http://arxiv.org/abs/2407.18067v1
• Date: Thu, 25 Jul 2024 14:21:50 GMT
• Title: HVM-1: Large-scale video models pretrained with nearly 5000 hours of human-like video data
• Authors: A. Emin Orhan,
• Abstract summary: We release two 633M parameter models trained at spatial resolutions of 224x and 448x pixels. We evaluate the performance of these models in downstream few-shot video and image recognition tasks. HVM-1 models learn more accurate and more robust object representations compared to models pretrained with the image-based MAE algorithm.
• Score: 10.225358400539722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Human-like Video Models (HVM-1), large-scale video models pretrained with nearly 5000 hours of curated human-like video data (mostly egocentric, temporally extended, continuous video recordings), using the spatiotemporal masked autoencoder (ST-MAE) algorithm. We release two 633M parameter models trained at spatial resolutions of 224x224 and 448x448 pixels. We evaluate the performance of these models in downstream few-shot video and image recognition tasks and compare them against a model pretrained with 1330 hours of short action-oriented video clips from YouTube (Kinetics-700). HVM-1 models perform competitively against the Kinetics-700 pretrained model in downstream evaluations despite substantial qualitative differences between the spatiotemporal characteristics of the corresponding pretraining datasets. HVM-1 models also learn more accurate and more robust object representations compared to models pretrained with the image-based MAE algorithm on the same data, demonstrating the potential benefits of learning to predict temporal regularities in natural videos for learning better object representations.

Related papers

• Learning Video Representations without Natural Videos [36.0052738021796]
  We show that useful video representations can be learned from synthetic videos and natural images, without incorporating natural videos in the training. A VideoMAE model pre-trained on our synthetic videos closes 97.2% of the performance gap on UCF101 action classification between training from scratch and self-supervised pre-training from natural videos. Introducing crops of static images to the pre-training stage results in similar performance to UCF101 pre-training and outperforms the UCF101 pre-trained model on 11 out of 14 out-of-distribution datasets of UCF101-P.
  arXiv  Detail & Related papers  (2024-10-31T17:59:30Z)
• Revisiting Feature Prediction for Learning Visual Representations from Video [62.08833572467379]
  V-JEPA is a collection of vision models trained solely using a feature prediction objective. The models are trained on 2 million videos collected from public datasets. Our results show that learning by predicting video features leads to versatile visual representations that perform well on both motion and appearance-based tasks.
  arXiv  Detail & Related papers  (2024-02-15T18:59:11Z)
• Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models [52.93036326078229]
  Off-the-shelf billion-scale datasets for image generation are available, but collecting similar video data of the same scale is still challenging. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. Our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks.
  arXiv  Detail & Related papers  (2023-05-17T17:59:16Z)
• VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking [57.552798046137646]
  Video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-03-29T14:28:41Z)
• Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning [123.63301596019522]
  Masked video distillation (MVD) is a simple yet effective two-stage masked feature modeling framework for video representation learning. For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks. We design a spatial-temporal co-teaching method for MVD to leverage the advantage of different teachers.
  arXiv  Detail & Related papers  (2022-12-08T18:59:59Z)
• Revisiting Classifier: Transferring Vision-Language Models for Video Recognition [102.93524173258487]
  Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. In this study, we focus on transferring knowledge for video classification tasks. We utilize the well-pretrained language model to generate good semantic target for efficient transferring learning.
  arXiv  Detail & Related papers  (2022-07-04T10:00:47Z)
• ViViT: A Video Vision Transformer [75.74690759089529]
  We present pure-transformer based models for video classification. Our model extracts-temporal tokens from the input video, which are then encoded by a series of transformer layers. We show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets.
  arXiv  Detail & Related papers  (2021-03-29T15:27:17Z)
• Unified Image and Video Saliency Modeling [21.701431656717112]
  We ask: Can image and video saliency modeling be approached via a unified model? We propose four novel domain adaptation techniques and an improved formulation of learned Gaussian priors. We integrate these techniques into a simple and lightweight encoder-RNN-decoder-style network, UNISAL, and train it jointly with image and video saliency data. We evaluate our method on the video saliency datasets DHF1K, Hollywood-2 and UCF-Sports, and the image saliency datasets SALICON and MIT300.
  arXiv  Detail & Related papers  (2020-03-11T18:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.