Switch-a-View: Few-Shot View Selection Learned from Edited Videos

• URL: http://arxiv.org/abs/2412.18386v1
• Date: Tue, 24 Dec 2024 12:16:43 GMT
• Title: Switch-a-View: Few-Shot View Selection Learned from Edited Videos
• Authors: Sagnik Majumder, Tushar Nagarajan, Ziad Al-Halah, Kristen Grauman,
• Abstract summary: We introduce Switch-a-View, a model that learns to automatically select the viewpoint to display at each timepoint when creating a how-to video. Key insight of our approach is how to train such a model from unlabeled--but human-edited--video samples.
• Score: 71.01549400773197
• License:
• Abstract: We introduce Switch-a-View, a model that learns to automatically select the viewpoint to display at each timepoint when creating a how-to video. The key insight of our approach is how to train such a model from unlabeled--but human-edited--video samples. We pose a pretext task that pseudo-labels segments in the training videos for their primary viewpoint (egocentric or exocentric), and then discovers the patterns between those view-switch moments on the one hand and the visual and spoken content in the how-to video on the other hand. Armed with this predictor, our model then takes an unseen multi-view video as input and orchestrates which viewpoint should be displayed when. We further introduce a few-shot training setting that permits steering the model towards a new data domain. We demonstrate our idea on a variety of real-world video from HowTo100M and Ego-Exo4D and rigorously validate its advantages.

Related papers

• An Empirical Study of Autoregressive Pre-training from Videos [67.15356613065542]
  We treat videos as visual tokens and train transformer models to autoregressively predict future tokens. Our models are pre-trained on a diverse dataset of videos and images comprising over 1 trillion visual tokens. Our results demonstrate that, despite minimal inductive biases, autoregressive pre-training leads to competitive performance.
  arXiv  Detail & Related papers  (2025-01-09T18:59:58Z)
• Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Videos [66.1935609072708]
  Key hypothesis is that the more accurately an individual view can predict a view-agnostic text summary, the more informative it is. We propose a framework that uses the relative accuracy of view-dependent caption predictions as a proxy for best view pseudo-labels. During inference, our model takes as input only a multi-view video--no language or camera poses--and returns the best viewpoint to watch at each timestep.
  arXiv  Detail & Related papers  (2024-11-13T16:31:08Z)
• Video In-context Learning [46.40277880351059]
  In this paper, we study video in-context learning, where the model starts from an existing video clip and generates diverse potential future sequences. To achieve this, we provide a clear definition of the task, and train an autoregressive Transformer on video datasets. We design various evaluation metrics, including both objective and subjective measures, to demonstrate the visual quality and semantic accuracy of generation results.
  arXiv  Detail & Related papers  (2024-07-10T04:27:06Z)
• Self-Supervised Video Representation Learning with Motion-Contrastive Perception [13.860736711747284]
  Motion-Contrastive Perception Network (MCPNet) MCPNet consists of two branches, namely, Motion Information Perception (MIP) and Contrastive Instance Perception (CIP) Our method outperforms current state-of-the-art visual-only self-supervised approaches.
  arXiv  Detail & Related papers  (2022-04-10T05:34:46Z)
• Multiview Pseudo-Labeling for Semi-supervised Learning from Video [102.36355560553402]
  We present a novel framework that uses complementary views in the form of appearance and motion information for semi-supervised learning in video. Our method capitalizes on multiple views, but it nonetheless trains a model that is shared across appearance and motion input. On multiple video recognition datasets, our method substantially outperforms its supervised counterpart, and compares favorably to previous work on standard benchmarks in self-supervised video representation learning.
  arXiv  Detail & Related papers  (2021-04-01T17:59:48Z)
• Broaden Your Views for Self-Supervised Video Learning [97.52216510672251]
  We introduce BraVe, a self-supervised learning framework for video. In BraVe, one of the views has access to a narrow temporal window of the video while the other view has a broad access to the video content. We demonstrate that BraVe achieves state-of-the-art results in self-supervised representation learning on standard video and audio classification benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T17:58:46Z)
• Recognizing Actions in Videos from Unseen Viewpoints [80.6338404141284]
  We show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in training data. We introduce a new dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.
  arXiv  Detail & Related papers  (2021-03-30T17:17:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.