MiVE: Multiscale Vision-language features for reference-guided video Editing

• URL: http://arxiv.org/abs/2605.14664v1
• Date: Thu, 14 May 2026 10:19:19 GMT
• Title: MiVE: Multiscale Vision-language features for reference-guided video Editing
• Abstract summary: We present MiVE (Multiscale Vision-language features for reference-guided video Editing), a framework that repurposes VLMs as multiscale feature extractors.<n> experiments demonstrate that MiVE achieves state-of-the-art performance by ranking highest in human preference, outperforming both academic methods and commercial systems.
• Score: 31.68670789147968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reference-guided video editing takes a source video, a text instruction, and a reference image as inputs, requiring the model to faithfully apply the instructed edits while preserving original motion and unedited content. Existing methods fall into two paradigms, each with inherent limitations: decoupled encoders suffer from modality gaps when processing instructions and visual content independently, while unified vision-language encoders lose fine-grained spatial details by relying solely on final-layer representations. We observe that VLM layers encode complementary information hierarchically -- early layers capture localized spatial details essential for precise editing, while deeper layers encode global semantics for instruction comprehension. Building on this insight, we present MiVE (Multiscale Vision-language features for reference-guided video Editing), a framework that repurposes VLMs as multiscale feature extractors. MiVE extracts hierarchical features from Qwen3-VL and integrates them into a unified self-attention Diffusion Transformer, eliminating the modality mismatch inherent in cross-attention designs. Experiments demonstrate that MiVE achieves state-of-the-art performance by ranking highest in human preference, outperforming both academic methods and commercial systems.

Related papers

• VoT: Vision-of-Thought for Unified Multimodal Representation Alignment [38.89764677562606]
  Vision-of-Thought (VoT) is a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs)<n>VoT generates discrete VoT tokens representing high-level visual plans, such as objects and layouts, before rendering pixels.<n>We train a specialized VoT tokenizer in the VLM semantic space with a closed-loop objective that combines VLM alignment, feature reconstruction, and vector-quantization losses.
  arXiv  Detail & Related papers  (2026-09-07T17:53:13Z)
• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuning [5.85033069870214]
  We propose an efficient vision-language fine-tuning method based on dynamic embedding and fusion of hierarchical visual features.<n>By fine-tuning only a small number of parameters, DEHVF achieves precise alignment and complement of cross-modal information.
  arXiv  Detail & Related papers  (2025-08-25T03:57:46Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• CoMemo: LVLMs Need Image Context with Image Memory [51.681858871027345]
  CoMemo is a dual-path architecture that combines a Context image path with an image Memory path for visual processing.<n>We introduce RoPE-DHR, a novel positional encoding mechanism that employs thumbnail-based positional aggregation to maintain 2D spatial awareness.
  arXiv  Detail & Related papers  (2025-06-06T17:59:06Z)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding [79.43306110124875]
  AlignVLM is a vision-text alignment method that maps visual features to a weighted average of text embeddings.<n>Our experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods.
  arXiv  Detail & Related papers  (2025-02-03T13:34:51Z)
• MAKIMA: Tuning-free Multi-Attribute Open-domain Video Editing via Mask-Guided Attention Modulation [55.101611012677616]
  Diffusion-based text-to-image (T2I) models have demonstrated remarkable results in global video editing tasks.<n>We present MAKIMA, a tuning-free MAE framework built upon pretrained T2I models for open-domain video editing.
  arXiv  Detail & Related papers  (2024-12-28T02:36:51Z)
• VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval [8.908777234657046]
  Large-language and vision-language models (LLM/LVLMs) have gained prominence across various domains.<n>Here we propose VideoLights, a novel HD/MR framework addressing these limitations through (i) Convolutional Projection and Feature Refinement modules.<n> Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-02T14:45:53Z)
• BRAVE: Broadening the visual encoding of vision-language models [48.41146184575914]
  Vision-language models (VLMs) are composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders. We introduce BRAVE, which consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen LM.
  arXiv  Detail & Related papers  (2024-04-10T17:59:45Z)
• Deeply Interleaved Two-Stream Encoder for Referring Video Segmentation [87.49579477873196]
  We first design a two-stream encoder to extract CNN-based visual features and transformer-based linguistic features hierarchically. A vision-language mutual guidance (VLMG) module is inserted into the encoder multiple times to promote the hierarchical and progressive fusion of multi-modal features. In order to promote the temporal alignment between frames, we propose a language-guided multi-scale dynamic filtering (LMDF) module.
  arXiv  Detail & Related papers  (2022-03-30T01:06:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.