TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation

• URL: http://arxiv.org/abs/2606.08260v1
• Date: Sat, 06 Jun 2026 17:11:13 GMT
• Title: TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation
• Abstract summary: Diffusion Transformers have driven rapid progress in video generation and editing, yet these capabilities are still handled by task-specific models.<n>We propose TIDE, a unified framework that integrates instruction-based editing, reference-guided editing, and multi-reference generation.
• Score: 30.09687573502619
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Diffusion Transformers have driven rapid progress in video generation and editing, yet these capabilities are still handled by separate, task-specific models. Building a unified framework that supports diverse video tasks remains an open challenge: existing unified attempts either require dedicated auxiliary encoders or lack explicit mechanisms to distinguish heterogeneous conditioning tokens, struggling when the number and type of visual conditions vary across tasks. We propose TIDE, a unified framework that integrates instruction-based editing, reference-guided editing, and multi-reference generation. At its core, we introduce per-token task embeddings that assign each input token a task-specific identifier, enabling the model to explicitly disambiguate target, source, and reference tokens. To simultaneously capture high-level semantic understanding and fine-grained structural fidelity, we design a dual-path conditioning scheme that couples a vision-language model with a VAE latent path for complementary signals. We further devise a multi-task progressive training strategy that incrementally introduces tasks of increasing complexity, effectively harmonizing diverse objectives and enabling smooth generalization across heterogeneous task distributions. Extensive experiments on multiple video editing and generation benchmarks demonstrate that TIDE achieves state-of-the-art performance across all evaluated tasks. Our project page is available at https://LittleWork123.github.io/tide.

Related papers

• Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning [56.17582599802973]
  We introduce the Multi-Instruction Multi-Shot Long-Video Editing (MMLVE) task, which is structured around three core objectives.<n>To tackle these three unique challenges, we introduce an agentic editing framework that leverages the Large Language Models (LLMs) and Vision-Language Models (VLMs)<n>Extensive experiments demonstrate that our MMLVE-Agent outperforms existing closed-source SOTA approaches.
  arXiv  Detail & Related papers  (2026-08-27T08:48:13Z)
• Vorch-Omni: Multi-Task Orchestration of Sight and Sound [51.39284238118125]
  Vorch- Omni is a unified framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-language formulation.<n>It flexibly treats video and audio signals as either conditioning inputs or generation targets.<n>It supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing.
  arXiv  Detail & Related papers  (2026-08-06T09:38:30Z)
• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing [93.8111348452324]
  Tele- Omni is a unified framework for video generation and editing that follows multimodal instructions.<n>It supports text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing.
  arXiv  Detail & Related papers  (2026-02-10T10:01:16Z)
• VINO: A Unified Visual Generator with Interleaved OmniModal Context [36.71641694179164]
  VINO is a unified visual generator that performs image and video generation and editing within a single framework.<n>Instead of relying on task-specific models or independent modules for each modality, VINO uses a shared diffusion backbone.
  arXiv  Detail & Related papers  (2026-01-05T18:56:34Z)
• Tracking and Segmenting Anything in Any Modality [75.32774085793498]
  We propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input.<n> SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.
  arXiv  Detail & Related papers  (2025-11-22T09:09:22Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• UNIC: Unified In-Context Video Editing [76.76077875564526]
  UNified In-Context Video Editing (UNIC) is a framework that unifies diverse video editing tasks within a single model in an in-context manner.<n>We introduce task-aware RoPE to facilitate consistent temporal positional encoding, and condition bias that enables the model to clearly differentiate different editing tasks.<n>Results demonstrate that our unified approach achieves superior performance on each task and exhibits emergent task composition abilities.
  arXiv  Detail & Related papers  (2025-06-04T17:57:43Z)
• FullDiT: Multi-Task Video Generative Foundation Model with Full Attention [37.776430879317765]
  FullDiT is a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms.<n> Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation.
  arXiv  Detail & Related papers  (2025-03-25T17:59:06Z)
• VACE: All-in-One Video Creation and Editing [18.809248697934397]
  VACE enables users to perform Video tasks within an All-in-one framework for Creation and Editing.<n>We introduce VACE, which enables users to perform Video tasks within an All-in-one framework for Creation and Editing.
  arXiv  Detail & Related papers  (2025-03-10T17:57:04Z)
• MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing [25.118495616895597]
  MIGE is a unified framework that standardizes task representations using multimodal instructions.<n>It first treats subject-driven generation as creation on a blank canvas and instruction-based editing as modification of an existing image.<n>MIGE excels in both subject-driven generation and instruction-based editing while setting a SOTA in the new task of instruction-based subject-driven editing.
  arXiv  Detail & Related papers  (2025-02-28T18:21:08Z)
• Video Task Decathlon: Unifying Image and Video Tasks in Autonomous Driving [85.62076860189116]
  Video Task Decathlon (VTD) includes ten representative image and video tasks spanning classification, segmentation, localization, and association of objects and pixels. We develop our unified network, VTDNet, that uses a single structure and a single set of weights for all ten tasks.
  arXiv  Detail & Related papers  (2023-09-08T16:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.