Moving Off-the-Grid: Scene-Grounded Video Representations

• URL: http://arxiv.org/abs/2411.05927v1
• Date: Fri, 08 Nov 2024 19:26:51 GMT
• Title: Moving Off-the-Grid: Scene-Grounded Video Representations
• Authors: Sjoerd van Steenkiste, Daniel Zoran, Yi Yang, Yulia Rubanova, Rishabh Kabra, Carl Doersch, Dilara Gokay, Joseph Heyward, Etienne Pot, Klaus Greff, Drew A. Hudson, Thomas Albert Keck, Joao Carreira, Alexey Dosovitskiy, Mehdi S. M. Sajjadi, Thomas Kipf,
• Abstract summary: We present Moving Off-the-Grid (MooG), a self-supervised video representation model. MooG allows tokens to move "off-the-grid" to better enable them to represent scene elements consistently. We show that MooG provides a strong foundation for different vision tasks when compared to "on-the-grid" baselines.
• Score: 44.13534423774967
• License:
• Abstract: Current vision models typically maintain a fixed correspondence between their representation structure and image space. Each layer comprises a set of tokens arranged "on-the-grid," which biases patches or tokens to encode information at a specific spatio(-temporal) location. In this work we present Moving Off-the-Grid (MooG), a self-supervised video representation model that offers an alternative approach, allowing tokens to move "off-the-grid" to better enable them to represent scene elements consistently, even as they move across the image plane through time. By using a combination of cross-attention and positional embeddings we disentangle the representation structure and image structure. We find that a simple self-supervised objective--next frame prediction--trained on video data, results in a set of latent tokens which bind to specific scene structures and track them as they move. We demonstrate the usefulness of MooG's learned representation both qualitatively and quantitatively by training readouts on top of the learned representation on a variety of downstream tasks. We show that MooG can provide a strong foundation for different vision tasks when compared to "on-the-grid" baselines.

Related papers

• TCFormer: Visual Recognition via Token Clustering Transformer [79.24723479088097]
  We propose the Token Clustering Transformer (TCFormer), which generates dynamic vision tokens based on semantic meaning. Our dynamic tokens possess two crucial characteristics: (1) Representing image regions with similar semantic meanings using the same vision token, even if those regions are not adjacent, and (2) concentrating on regions with valuable details and represent them using fine tokens.
  arXiv  Detail & Related papers  (2024-07-16T02:26:18Z)
• UnLoc: A Unified Framework for Video Localization Tasks [82.59118972890262]
  UnLoc is a new approach for temporal localization in untrimmed videos. It uses pretrained image and text towers, and feeds tokens to a video-text fusion model. We achieve state of the art results on all three different localization tasks with a unified approach.
  arXiv  Detail & Related papers  (2023-08-21T22:15:20Z)
• Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens [93.98605636451806]
  StructureViT shows how utilizing the structure of a small number of images only available during training can improve a video model. SViT shows strong performance improvements on multiple video understanding tasks and datasets.
  arXiv  Detail & Related papers  (2022-06-13T17:45:05Z)
• BlobGAN: Spatially Disentangled Scene Representations [67.60387150586375]
  We propose an unsupervised, mid-level representation for a generative model of scenes. The representation is mid-level in that it is neither per-pixel nor per-image; rather, scenes are modeled as a collection of spatial, depth-ordered "blobs" of features.
  arXiv  Detail & Related papers  (2022-05-05T17:59:55Z)
• Continuous Scene Representations for Embodied AI [33.00565252990522]
  Continuous Scene Representations (CSR) is a scene representation constructed by an embodied agent navigating within a space. Our key insight is to embed pair-wise relationships between objects in a latent space. CSR can track objects as the agent moves in a scene, update the representation accordingly, and detect changes in room configurations.
  arXiv  Detail & Related papers  (2022-03-31T17:55:33Z)
• Leveraging Local Temporal Information for Multimodal Scene Classification [9.548744259567837]
  Video scene classification models should capture the spatial (pixel-wise) and temporal (frame-wise) characteristics of a video effectively. Transformer models with self-attention which are designed to get contextualized representations for individual tokens given a sequence of tokens, are becoming increasingly popular in many computer vision tasks. We propose a novel self-attention block that leverages both local and global temporal relationships between the video frames to obtain better contextualized representations for the individual frames.
  arXiv  Detail & Related papers  (2021-10-26T19:58:32Z)
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning [82.09856883441044]
  Video understanding relies on perceiving the global content modeling its internal connections. We propose a block-wise strategy where we mask neighboring video tokens in both spatial and temporal domains. We also add an augmentation-free contrastive learning method to further capture global content.
  arXiv  Detail & Related papers  (2021-06-21T16:48:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.