Video-T1: Test-Time Scaling for Video Generation

• URL: http://arxiv.org/abs/2503.18942v2
• Date: Tue, 01 Apr 2025 06:52:58 GMT
• Title: Video-T1: Test-Time Scaling for Video Generation
• Authors: Fangfu Liu, Hanyang Wang, Yimo Cai, Kaiyan Zhang, Xiaohang Zhan, Yueqi Duan,
• Abstract summary: Researchers in Large Language Models (LLMs) have expanded the scaling to test-time.<n>We aim to answer the question: if a video generation model is allowed to use non-trivial amount of inference-time compute, how much can it improve generation quality given a challenging text prompt.<n>Experiments on text-conditioned video generation benchmarks demonstrate that increasing test-time compute consistently leads to significant improvements in the quality of videos.
• Score: 19.089876374170167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the scale capability of increasing training data, model size, and computational cost, video generation has achieved impressive results in digital creation, enabling users to express creativity across various domains. Recently, researchers in Large Language Models (LLMs) have expanded the scaling to test-time, which can significantly improve LLM performance by using more inference-time computation. Instead of scaling up video foundation models through expensive training costs, we explore the power of Test-Time Scaling (TTS) in video generation, aiming to answer the question: if a video generation model is allowed to use non-trivial amount of inference-time compute, how much can it improve generation quality given a challenging text prompt. In this work, we reinterpret the test-time scaling of video generation as a searching problem to sample better trajectories from Gaussian noise space to the target video distribution. Specifically, we build the search space with test-time verifiers to provide feedback and heuristic algorithms to guide searching process. Given a text prompt, we first explore an intuitive linear search strategy by increasing noise candidates at inference time. As full-step denoising all frames simultaneously requires heavy test-time computation costs, we further design a more efficient TTS method for video generation called Tree-of-Frames (ToF) that adaptively expands and prunes video branches in an autoregressive manner. Extensive experiments on text-conditioned video generation benchmarks demonstrate that increasing test-time compute consistently leads to significant improvements in the quality of videos. Project page: https://liuff19.github.io/Video-T1

Related papers

• Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval [56.05621657583251]
  Cross-modal (e.g. image-text, video-text) retrieval is an important task in information retrieval and multimodal vision-language understanding field. We introduce RTime, a novel temporal-emphasized video-text retrieval dataset. Our RTime dataset currently consists of 21k videos with 10 captions per video, totalling about 122 hours.
  arXiv  Detail & Related papers  (2024-12-26T11:32:00Z)
• Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs [56.040198387038025]
  We present a novel prompt-guided visual perception framework (abbreviated as Free Video-LLM) for efficient inference of training-free video LLMs. Our method effectively reduces the number of visual tokens while maintaining high performance across multiple video question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T12:35:12Z)
• SF-V: Single Forward Video Generation Model [57.292575082410785]
  We propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained models. Experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead.
  arXiv  Detail & Related papers  (2024-06-06T17:58:27Z)
• Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation [72.90144343056227]
  We explore the visual representations produced from a pre-trained text-to-video (T2V) diffusion model for video understanding tasks. We introduce a novel framework, termed "VD-IT", tailored with dedicatedly designed components built upon a fixed T2V model. Our VD-IT achieves highly competitive results, surpassing many existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-18T17:59:58Z)
• Text-Conditioned Resampler For Long Form Video Understanding [94.81955667020867]
  We present a text-conditioned video resampler (TCR) module that uses a pre-trained visual encoder and large language model (LLM) TCR can process more than 100 frames at a time with plain attention and without optimised implementations.
  arXiv  Detail & Related papers  (2023-12-19T06:42:47Z)
• FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling [85.60543452539076]
  Existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. This study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. We propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models.
  arXiv  Detail & Related papers  (2023-10-23T17:59:58Z)
• LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation [44.220329202024494]
  We present a few-shot-based tuning framework, LAMP, which enables text-to-image diffusion model Learn A specific Motion Pattern with 816 videos on a single GPU. Specifically, we design a first-frame-conditioned pipeline that uses an off-the-shelf text-to-image model for content generation. To capture the features of temporal dimension, we expand the pretrained 2D convolution layers of the T2I model to our novel temporal-spatial motion learning layers.
  arXiv  Detail & Related papers  (2023-10-16T19:03:19Z)
• A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval [16.548016892117083]
  Text-video retrieval methods have received increased attention over the past few years. Data augmentation techniques were introduced to increase the performance on unseen test examples. We propose a multimodal data augmentation technique which works in the feature space and creates new videos and captions by mixing semantically similar samples.
  arXiv  Detail & Related papers  (2022-08-03T14:05:20Z)
• Capturing Temporal Information in a Single Frame: Channel Sampling Strategies for Action Recognition [19.220288614585147]
  We address the problem of capturing temporal information for video classification in 2D networks, without increasing computational cost. We propose a novel sampling strategy, where we re-order the channels of the input video, to capture short-term frame-to-frame changes. Our sampling strategies do not require training from scratch and do not increase the computational cost of training and testing.
  arXiv  Detail & Related papers  (2022-01-25T15:24:37Z)
• Video Generation from Text Employing Latent Path Construction for Temporal Modeling [70.06508219998778]
  Video generation is one of the most challenging tasks in Machine Learning and Computer Vision fields of study. In this paper, we tackle the text to video generation problem, which is a conditional form of video generation. We believe that video generation from natural language sentences will have an important impact on Artificial Intelligence.
  arXiv  Detail & Related papers  (2021-07-29T06:28:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.