LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing

• URL: http://arxiv.org/abs/2606.06042v2
• Date: Fri, 05 Jun 2026 04:03:45 GMT
• Title: LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing
• Abstract summary: We present LoomVideo, a highly efficient 5B- parameter unified architecture for both video generation and editing.<n>By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation.<n>Our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks.
• Score: 28.426681930851604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier field. Existing unified frameworks predominantly rely on massive models (typically 13B parameters or more) and incorporate source video conditions for editing by concatenating sequence tokens. This concatenation inevitably doubles the sequence length, quadrupling the computational complexity of the self-attention mechanism and introducing prohibitive overhead. To address these bottlenecks, we present LoomVideo, a highly efficient 5B-parameter unified architecture for both video generation and editing. LoomVideo replaces the standard text encoder with a Multimodal Large Language Model (MLLM) and employs Deepstack injection mechanism to align multi-layer MLLM features with the Diffusion Transformer (DiT). Crucially, we introduce a zero-overhead Scale-and-Add conditioning approach for video editing. By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation, drastically reducing computational cost while maintaining robust capabilities for complex, non-rigid edits. Furthermore, a Negative Temporal RoPE strategy is seamlessly integrated to handle multiple reference images. Extensive experiments demonstrate that our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, exhibiting exceptional superiority in e-commerce and fashion generation scenarios. Benefiting from the zero-overhead conditioning mechanism, LoomVideo achieves at least a 5.41x acceleration in inference speed compared to models of similar capabilities, paving the way for highly practical and efficient video foundation models.

Related papers

• Sol Video Inference Engine: Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation [50.504140880964336]
  We present Sol Video Inference, an agentic, native, training-free acceleration framework for video diffusion models.<n>It organizes five broadly applicable techniques, cache, sparse attention, token pruning, quantization, and kernel fusion, into an agentic acceleration stack.<n>It achieves more than 2x end-to-end acceleration while maintaining near-lossless VBench quality.
  arXiv  Detail & Related papers  (2026-06-21T17:23:20Z)
• Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing [21.525921468472685]
  We present a scalable and computationally efficient model that connects pretrained multimodal large-language models (MLLMs) with video diffusion models for unified video generation and editing.<n>Our key idea is to exploit the understanding and reasoning capabilities of MLLMs to produce explicit target captions to interpret user instructions.<n>We evaluate the performance of Omni-Video 2 on the FiVE benchmark for fine-grained video editing and the VBench benchmark for text-to-video generation.
  arXiv  Detail & Related papers  (2026-02-09T15:56:05Z)
• LoViC: Efficient Long Video Generation with Context Compression [68.22069741704158]
  We introduce LoViC, a DiT-based framework trained on million-scale open-domain videos.<n>At the core of our approach is FlexFormer, an expressive autoencoder that jointly compresses video and text into unified latent representations.
  arXiv  Detail & Related papers  (2025-07-17T09:46:43Z)
• M4V: Multi-Modal Mamba for Text-to-Video Generation [58.51139515986472]
  Text-to-video generation has enriched content and holds potential to create powerful world simulators.<n>Modeling the vast space remains computationally demanding, particularly when employing quadratic in sequence processing.<n>We introduce a Multi-Modal Mamba framework for text-to-video generation.<n>Experiments on text-to-video benchmarks demonstrate M4V's ability to produce high-quality videos while significantly lowering computational costs.
  arXiv  Detail & Related papers  (2025-06-12T17:29:40Z)
• StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant [27.21599940736722]
  We present StreamBridge, a framework that seamlessly transforms offline Video-LLMs into streaming-capable models.<n>It addresses two fundamental challenges in adapting existing models into online scenarios: (1) limited capability for multi-turn real-time understanding, and (2) lack of proactive response mechanisms.
  arXiv  Detail & Related papers  (2025-05-08T17:57:40Z)
• Mobile-VideoGPT: Fast and Accurate Video Understanding Language Model [60.171601995737646]
  Mobile-VideoGPT is an efficient multimodal framework for video understanding.<n>It consists of lightweight dual visual encoders, efficient projectors, and a small language model (SLM)<n>Our results show that Mobile-VideoGPT-0.5B can generate up to 46 tokens per second.
  arXiv  Detail & Related papers  (2025-03-27T17:59:58Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models [26.866184981409607]
  We present an efficient encoder-free approach for video-language understanding that achieves competitive performance while significantly reducing computational overhead.<n>Our method introduces a novel Spatio-Temporal Alignment Block (STAB) that directly processes video inputs without requiring pre-trained encoders.<n>Our model achieves comparable or superior performance to encoder-based approaches for open-ended video question answering on standard benchmarks.
  arXiv  Detail & Related papers  (2024-12-24T18:59:56Z)
• When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding [118.72266141321647]
  Cross-Modality Video Coding (CMVC) is a pioneering approach to explore multimodality representation and video generative models in video coding.<n>During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes.<n>Experiments indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency.
  arXiv  Detail & Related papers  (2024-08-15T11:36:18Z)
• DreamFrame: Enhancing Video Understanding via Automatically Generated QA and Style-Consistent Keyframes [11.2645921649719]
  Recent vision-language models (LVLMs) for understanding are primarily fine-tuned with various scraped from online platforms.<n>While current LVLMs are primarily trained on existing datasets in broad, general-purpose settings, adapting them to specific downstream scenarios remains challenging.<n>We propose a three-stage framework named DreamFrame for automatically generating style-consistent videos.
  arXiv  Detail & Related papers  (2024-03-03T07:43:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.