TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation

• URL: http://arxiv.org/abs/2412.10275v2
• Date: Mon, 16 Dec 2024 03:32:09 GMT
• Title: TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation
• Authors: Xingrui Wang, Xin Li, Yaosi Hu, Hanxin Zhu, Chen Hou, Cuiling Lan, Zhibo Chen,
• Abstract summary: Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. We propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment. Our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.
• Score: 31.43081425504501
• License:
• Abstract: Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. The primary challenges of this task lie in two parts: (i) how to identify the target objects and ensure the consistency between the movement trajectory and the textual description. (ii) how to improve the subjective quality of generated videos. To tackle the above challenges, we propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment, intending to achieve precise control and high-quality video generation based on textual-described motion for different objects. Concretely, we enable our TIV-Diffuion model to perceive the textual-described objects and their motion trajectory by incorporating the fused textual and visual knowledge through scale-offset modulation. Moreover, to mitigate the problems of object disappearance and misaligned objects and motion, we introduce an object-centric textual-visual alignment module, which reduces the risk of misaligned objects/motion by decoupling the objects in the reference image and aligning textual features with each object individually. Based on the above innovations, our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.

Related papers

• Instance-Level Moving Object Segmentation from a Single Image with Events [84.12761042512452]
  Moving object segmentation plays a crucial role in understanding dynamic scenes involving multiple moving objects. Previous methods encounter difficulties in distinguishing whether pixel displacements of an object are caused by camera motion or object motion. Recent advances exploit the motion sensitivity of novel event cameras to counter conventional images' inadequate motion modeling capabilities. We propose the first instance-level moving object segmentation framework that integrates complementary texture and motion cues.
  arXiv  Detail & Related papers  (2025-02-18T15:56:46Z)
• Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation [52.337472185022136]
  We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description. We propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation. We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art consistency.
  arXiv  Detail & Related papers  (2025-01-06T14:49:26Z)
• Referring Video Object Segmentation via Language-aligned Track Selection [30.226373787454833]
  Referring Video Object (RVOS) seeks to segment objects throughout a video based on natural language expressions. inconsistent mask tracks can disrupt vision-language alignment, leading to suboptimal performance. We present Selection by Object Language Alignment (SOLA), a novel framework that reformulates RVOS into two sub-problems, track generation and track selection.
  arXiv  Detail & Related papers  (2024-12-02T05:20:35Z)
• Text-Animator: Controllable Visual Text Video Generation [149.940821790235]
  We propose an innovative approach termed Text-Animator for visual text video generation. Text-Animator contains a text embedding injection module to precisely depict the structures of visual text in generated videos. We also develop a camera control module and a text refinement module to improve the stability of generated visual text.
  arXiv  Detail & Related papers  (2024-06-25T17:59:41Z)
• HumanTOMATO: Text-aligned Whole-body Motion Generation [30.729975715600627]
  This work targets a novel text-driven whole-body motion generation task. It aims at generating high-quality, diverse, and coherent facial expressions, hand gestures, and body motions simultaneously.
  arXiv  Detail & Related papers  (2023-10-19T17:59:46Z)
• Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning [50.60891619269651]
  Control-A-Video is a controllable T2V diffusion model that can generate videos conditioned on text prompts and reference control maps like edge and depth maps. We propose novel strategies to incorporate content prior and motion prior into the diffusion-based generation process. Our framework generates higher-quality, more consistent videos compared to existing state-of-the-art methods in controllable text-to-video generation.
  arXiv  Detail & Related papers  (2023-05-23T09:03:19Z)
• Text-driven Video Prediction [83.04845684117835]
  We propose a new task called Text-driven Video Prediction (TVP) Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. To investigate the capability of text in causal inference for progressive motion information, our TVP framework contains a Text Inference Module (TIM)
  arXiv  Detail & Related papers  (2022-10-06T12:43:07Z)
• Make It Move: Controllable Image-to-Video Generation with Text Descriptions [69.52360725356601]
  TI2V task aims at generating videos from a static image and a text description. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor structure. Experiments conducted on datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task.
  arXiv  Detail & Related papers  (2021-12-06T07:00:36Z)
• O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning [41.14313691818424]
  We propose an Object-Oriented Non-Autoregressive approach (O2NA) for video captioning. O2NA performs caption generation in three steps: 1) identify the focused objects and predict their locations in the target caption; 2) generate the related attribute words and relation words of these focused objects to form a draft caption; and 3) combine video information to refine the draft caption to a fluent final caption. Experiments on two benchmark datasets, MSR-VTT and MSVD, demonstrate the effectiveness of O2NA.
  arXiv  Detail & Related papers  (2021-08-05T04:17:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.