Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusion

• URL: http://arxiv.org/abs/2604.01761v1
• Date: Thu, 02 Apr 2026 08:27:48 GMT
• Title: Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusion
• Abstract summary: We introduce a lightweight architecture and training strategy that decouples appearance from other features.<n>We show that low spatial resolution can be compensated by higher feature dimensionality, improving controllability in generative rendering from explicit spatial representations.
• Score: 4.4853338999399375
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Video models have recently been applied with success to problems in content generation, novel view synthesis, and, more broadly, world simulation. Many applications in generation and transfer rely on conditioning these models, typically through perceptual, geometric, or simple semantic signals, fundamentally using them as generative renderers. At the same time, high-dimensional features obtained from large-scale self-supervised learning on images or point clouds are increasingly used as a general-purpose interface for vision models. The connection between the two has been explored for subject specific editing, aligning and training video diffusion models, but not in the role of a more general conditioning signal for pretrained video diffusion models. Features obtained through self-supervised learning like DINO, contain a lot of entangled information about style, lighting and semantics of the scene. This makes them great at reconstruction tasks but limits their generative capabilities. In this paper, we show how we can use the features for tasks such as video domain transfer and video-from-3D generation. We introduce a lightweight architecture and training strategy that decouples appearance from other features that we wish to preserve, enabling robust control for appearance changes such as stylization and relighting. Furthermore, we show that low spatial resolution can be compensated by higher feature dimensionality, improving controllability in generative rendering from explicit spatial representations.

Related papers

• Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors [61.34273238077091]
  We present a novel method for generating orbital videos from a single image of an object.<n>Our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-14T05:35:46Z)
• Lighting-grounded Video Generation with Renderer-based Agent Reasoning [56.50946217758078]
  LiVER is a diffusion-based framework for scene-controllable video generation.<n>We present a novel framework that conditions video synthesis on explicit 3D scene properties.<n>Our method disentangles these properties by rendering control signals from a unified 3D representation.
  arXiv  Detail & Related papers  (2026-04-09T08:29:29Z)
• Moaw: Unleashing Motion Awareness for Video Diffusion Models [71.34328578845721]
  Moaw is a framework that unleashes motion awareness for video diffusion models.<n>We train a diffusion model for motion perception, shifting its modality from image-to-video generation to video-to-dense-tracking.<n>We then construct a motion-labeled dataset to identify features that encode the strongest motion information, and inject them into a structurally identical video generation model.
  arXiv  Detail & Related papers  (2026-01-19T06:45:46Z)
• DANCER: Dance ANimation via Condition Enhancement and Rendering with diffusion model [5.78710251788825]
  We propose a novel framework, named as DANCER, for realistic single-person dance synthesis based on the most recent stable video diffusion model.<n>We introduce two important modules into our framework to fully benefit from the two inputs.<n>To further improve the generation capability of our model, we also collect a large amount of video data from Internet, and generate a novel datasetTikTok-3K to enhance the model training.
  arXiv  Detail & Related papers  (2025-10-31T04:42:08Z)
• From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models [65.0487600936788]
  Video Diffusion Models (VDMs) have emerged as powerful generative tools capable of synthesizing high-quality content.<n>We argue that VDMs naturally push to probe structured representations and an implicit understanding of the visual world.<n>Our method transforms each task into a visual transition, enabling the training of LoRA weights on short input- sequences.
  arXiv  Detail & Related papers  (2025-06-08T20:52:34Z)
• Dimension-Reduction Attack! Video Generative Models are Experts on Controllable Image Synthesis [12.160537328404622]
  textttDRA-Ctrl provides new insights into reusing resource-intensive video models.<n>textttDRA-Ctrl lays foundation for future unified generative models across visual modalities.
  arXiv  Detail & Related papers  (2025-05-29T10:34:45Z)
• GenLit: Reformulating Single-Image Relighting as Video Generation [42.0880277180892]
  We introduce GenLit, a framework that distills the ability of a graphics engine to perform light manipulation into a video-generation model.<n>We find that a model fine-tuned on only a small synthetic dataset generalizes to real-world scenes.
  arXiv  Detail & Related papers  (2024-12-15T15:40:40Z)
• Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions [94.03133100056372]
  Moonshot is a new video generation model that conditions simultaneously on multimodal inputs of image and text. Model can be easily repurposed for a variety of generative applications, such as personalized video generation, image animation and video editing.
  arXiv  Detail & Related papers  (2024-01-03T16:43:47Z)
• TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models [75.20168902300166]
  We propose TrackDiffusion, a novel video generation framework affording fine-grained trajectory-conditioned motion control. A pivotal component of TrackDiffusion is the instance enhancer, which explicitly ensures inter-frame consistency of multiple objects. generated video sequences by our TrackDiffusion can be used as training data for visual perception models.
  arXiv  Detail & Related papers  (2023-12-01T15:24:38Z)
• StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN [70.31913835035206]
  We present a novel approach to the video synthesis problem that helps to greatly improve visual quality. We make use of a pre-trained StyleGAN network, the latent space of which allows control over the appearance of the objects it was trained for. Our temporal architecture is then trained not on sequences of RGB frames, but on sequences of StyleGAN latent codes.
  arXiv  Detail & Related papers  (2021-07-15T09:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.