From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creation

• URL: http://arxiv.org/abs/2608.14740v1
• Date: Thu, 13 Aug 2026 18:11:45 GMT
• Title: From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creation
• Abstract summary: Unified image and video creation requires a model to follow diverse instructions while preserving identity, geometry, and temporal structure from visual context.<n>We formulate depth and surface-normal prediction as image-form denoising targets, using these dense tasks as structured visual supervision within the same creation interface.
• Score: 47.1844759979843
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified image and video creation requires a model to follow diverse instructions while preserving identity, geometry, and temporal structure from visual context. However, semantic-only conditioning and creation-only training do not explicitly supervise the local structure needed for precise, temporally consistent editing. We therefore formulate depth and surface-normal prediction as image-form denoising targets, using these dense tasks as structured visual supervision within the same creation interface. Our framework decouples semantic interpretation from spatially aligned visual injection while sharing one multimodal diffusion transformer (MMDiT) backbone across all tasks. Mutual Context Attention (MCA), a paired-video data-construction procedure, and a progressive training curriculum then connect the learned structural cues to temporally localized editing and reference-conditioned creation. A single checkpoint obtains the highest overall score in the reported comparison of unified systems (4.15); adding dense supervision improves OpenVE Overall from 3.98 to 4.06 and Local Add from 3.92 to 4.18. These results support a deliberately bounded conclusion: perception-oriented dense supervision transfers useful structural knowledge to downstream creation, especially editing locality and preservation; we do not claim superiority as a standalone dense predictor.

Related papers

• SenseNova-U1.5: Towards Native Unified Visual Intelligence [100.20348258853939]
  SenseNova-U1.5 is an 8B-MoT native unified multimodal model that understands, reasons about, and generates visual content.<n>We scale its training with carefully curated generation and editing data, improved task formulation, structural prompt enhancement, and native resolutions of up to 4K.<n>We will open-source training code, including supervised fine-tuning, reinforcement learning, and on-policy distillation.
  arXiv  Detail & Related papers  (2026-09-10T17:59:55Z)
• CoT-Edit: Let CoT Guide Instruction Video Editing [13.600094788792633]
  We propose a plan--guide--edit framework that bridges semantic intent and spatial execution.<n>In our framework, a Chain-of-Thought (CoT)-enhanced multimodal large language model (MLLM) serves as a planner.<n>These spatial priors then guide a box-conditioned mask generator, transforming ambiguous global retrieval into localized, context-aware refinement.
  arXiv  Detail & Related papers  (2026-08-02T09:20:15Z)
• IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generation [45.4088603828487]
  Implicit Visual Chain-of-conditioned (IV-CoT) is a latent visual reasoning framework for query-Thought image generation.<n> IV-CoT decomposes the visual conditioning queries into a structural-to-semantic cascade.<n>Visualizations and analyses demonstrate that the learned structural and semantic queries play complementary roles in structure-aware generation.
  arXiv  Detail & Related papers  (2026-06-23T17:28:00Z)
• SceneForge: Structured World Supervision from 3D Interventions [5.973748478214713]
  multimodal learning tasks require supervision that remains consistent across edits, viewpoints, and scene-level interventions.<n>We present SceneForge, an intervention-driven framework that generates structured supervision from editable 3D world states.
  arXiv  Detail & Related papers  (2026-05-14T05:38:00Z)
• 3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing [57.785328880266775]
  We introduce a Structured Reasoning framework that performs text-conditioned spatial layout editing via scene-graph reasoning.<n>By explicitly guiding the reasoning process through structured relational representations, our approach improves both interpretability and control over spatial relationships.
  arXiv  Detail & Related papers  (2026-03-23T17:59:14Z)
• InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning [60.799998743918955]
  We propose a novel text-vision Interleaved Chain-of-Grounding reasoning framework for fine-grained image editing in complex real-world scenes.<n>The key insight of InterCoG is to first perform object position reasoning solely within text.<n>We also propose two auxiliary training modules: multimodal grounding reconstruction supervision and multimodal grounding reasoning alignment.
  arXiv  Detail & Related papers  (2026-03-02T08:13:16Z)
• RT-VLM: Re-Thinking Vision Language Model with 4-Clues for Real-World Object Recognition Robustness [2.9979091009694088]
  Real world deployments often expose modern object recognition models to domain shifts that precipitate a severe drop in accuracy.<n>To mitigate this degradation, we introduce the Re-Thinking Vision Language Model (RT-VLM) framework.
  arXiv  Detail & Related papers  (2025-09-01T02:13:00Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• Collaborative Temporal Consistency Learning for Point-supervised Natural Language Video Localization [129.43937834515688]
  We propose a new COllaborative Temporal consistEncy Learning (COTEL) framework to strengthen the video-language alignment.<n>Specifically, we first design a frame- and a segment-level Temporal Consistency Learning (TCL) module that models semantic alignment across frame saliencies and sentence-moment pairs.
  arXiv  Detail & Related papers  (2025-03-22T05:04:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.