VideoMAR: Autoregressive Video Generatio with Continuous Tokens

• URL: http://arxiv.org/abs/2506.14168v2
• Date: Wed, 18 Jun 2025 09:44:09 GMT
• Title: VideoMAR: Autoregressive Video Generatio with Continuous Tokens
• Authors: Hu Yu, Biao Gong, Hangjie Yuan, DanDan Zheng, Weilong Chai, Jingdong Chen, Kecheng Zheng, Feng Zhao,
• Abstract summary: Masked-based autoregressive models have demonstrated promising image generation capability in continuous space.<n>We propose textbfVideoMAR, a decoder-only autoregressive image-to-video model with continuous tokens.<n>VideoMAR surpasses the previous state-of-the-art (Cosmos I2V) while requiring significantly fewer parameters.
• Score: 33.906543515428424
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. In this paper, we propose \textbf{VideoMAR}, a concise and efficient decoder-only autoregressive image-to-video model with continuous tokens, composing temporal frame-by-frame and spatial masked generation. We first identify temporal causality and spatial bi-directionality as the first principle of video AR models, and propose the next-frame diffusion loss for the integration of mask and video generation. Besides, the huge cost and difficulty of long sequence autoregressive modeling is a basic but crucial issue. To this end, we propose the temporal short-to-long curriculum learning and spatial progressive resolution training, and employ progressive temperature strategy at inference time to mitigate the accumulation error. Furthermore, VideoMAR replicates several unique capacities of language models to video generation. It inherently bears high efficiency due to simultaneous temporal-wise KV cache and spatial-wise parallel generation, and presents the capacity of spatial and temporal extrapolation via 3D rotary embeddings. On the VBench-I2V benchmark, VideoMAR surpasses the previous state-of-the-art (Cosmos I2V) while requiring significantly fewer parameters ($9.3\%$), training data ($0.5\%$), and GPU resources ($0.2\%$).

Related papers

• Lumos-1: On Autoregressive Video Generation from a Unified Model Perspective [37.58855048653859]
  Autoregressive large language models (LLMs) have unified a vast range of language tasks, inspiring preliminary efforts of autoregressive video generation.<n>Lumos-1 retains autoregressive video generator architecture with minimal architectural modifications.
  arXiv  Detail & Related papers  (2025-07-11T17:59:42Z)
• RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation [30.252593687028767]
  We address the problem of generating long-horizon videos for robotic manipulation tasks.<n>We propose a novel pipeline that bypasses the need for autoregressive generation.<n>Our approach achieves state-of-the-art results on two benchmarks in video quality and consistency.
  arXiv  Detail & Related papers  (2025-06-27T08:21:55Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [73.25506085339252]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• Mavors: Multi-granularity Video Representation for Multimodal Large Language Model [39.24524388617938]
  $mathbfMavors$ is a novel framework for holistic longvideo modeling.<n>Mavors encodes raw video content into latent representations through two core components.<n>The framework unifies image and video understanding by treating images as single-frame videos.
  arXiv  Detail & Related papers  (2025-04-14T10:14:44Z)
• Token-Efficient Long Video Understanding for Multimodal LLMs [101.70681093383365]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [52.32078428442281]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• When Spatial meets Temporal in Action Recognition [34.53091498930863]
  We introduce the Temporal Integration and Motion Enhancement (TIME) layer, a novel preprocessing technique designed to incorporate temporal information. The TIME layer generates new video frames by rearranging the original sequence, preserving temporal order while embedding $N2$ temporally evolving frames into a single spatial grid. Our experiments show that the TIME layer enhances recognition accuracy, offering valuable insights for video processing tasks.
  arXiv  Detail & Related papers  (2024-11-22T16:39:45Z)
• MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion [3.7270979204213446]
  We present four key contributions to address the challenges of video processing.<n>First, we introduce the 3D Inverted Vector-Quantization Variencoenco Autocoder.<n>Second, we present MotionAura, a text-to-video generation framework.<n>Third, we propose a spectral transformer-based denoising network.<n>Fourth, we introduce a downstream task of Sketch Guided Videopainting.
  arXiv  Detail & Related papers  (2024-10-10T07:07:56Z)
• SIGMA:Sinkhorn-Guided Masked Video Modeling [69.31715194419091]
  Sinkhorn-guided Masked Video Modelling ( SIGMA) is a novel video pretraining method. We distribute features of space-time tubes evenly across a limited number of learnable clusters. Experimental results on ten datasets validate the effectiveness of SIGMA in learning more performant, temporally-aware, and robust video representations.
  arXiv  Detail & Related papers  (2024-07-22T08:04:09Z)
• RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks [93.18404922542702]
  We present a novel video generative model designed to address long-term spatial and temporal dependencies. Our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks. Our model synthesizes high-fidelity video clips at a resolution of $256times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps.
  arXiv  Detail & Related papers  (2024-01-11T16:48:44Z)
• Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation [55.36617538438858]
  We propose a novel approach that strengthens the interaction between spatial and temporal perceptions. We curate a large-scale and open-source video dataset called HD-VG-130M.
  arXiv  Detail & Related papers  (2023-05-18T11:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.