Unified Video-Action Joint Denoising for Dexterous Action and Data Generation

• URL: http://arxiv.org/abs/2606.03868v1
• Date: Tue, 02 Jun 2026 16:39:23 GMT
• Title: Unified Video-Action Joint Denoising for Dexterous Action and Data Generation
• Abstract summary: We propose Donk, a unified video-action denoising model for dexterous hands.<n>Donk samples future videos and bimanual MANO trajectories as an action policy.<n>Without the image condition, the same denoising architecture samples video-action rollouts from a text-conditioned distribution.
• Score: 13.23337683310307
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent world action models leverage video foundation models by aligning broad visual-dynamics priors with executable robot actions. We revisit this alignment from a distributional perspective. Existing formulations typically narrow the aligned prior into an observation-conditioned policy distribution over future actions. In contrast, we keep the distribution broader by modeling the joint space of interaction videos and executable hand trajectories under multiple conditioning regimes. We propose Donk, a unified video-action denoising model for dexterous hands. With language, an initial image, and the initial hand state, Donk samples future videos and bimanual MANO trajectories as an action policy. Without the image condition, the same denoising architecture samples paired video-action rollouts from a text-conditioned distribution, turning the aligned video prior into a data engine. Across action, video, and text-only generation evaluations, Donk improves dexterous trajectory accuracy, preserves strong video fidelity, and produces smooth text-conditioned action rollouts under the same unified training recipe.

Related papers

• DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning [28.673173377342593]
  General robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions.<n>DeVA is a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically guidance.<n> Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.
  arXiv  Detail & Related papers  (2026-07-27T08:37:35Z)
• Geo-Align: Video Generation Alignment via Metric Geometry Reward [53.65904111864641]
  Geo-Align is the first Reinforcement Learning framework specifically designed for camera-controlled video re-rendering.<n>Built upon a pretrained model, we optimize the model through a scale-aware perceptual reward mechanism.<n>Experiments demonstrate that Geo-Align consistently outperforms existing supervised learning baselines in both precise camera controllability and visual fidelity.
  arXiv  Detail & Related papers  (2026-05-22T17:59:43Z)
• SWEET: Sparse World Modeling with Image Editing for Embodied Task Execution [61.612676324369595]
  We study whether image editing models can serve as sparse visual world models for robot manipulation.<n>We propose SWEET, a one-shot prediction visual planning framework.<n>Experiments on DROID and RoboMimic show that SWEET improves prediction across seen and unseen scenes.
  arXiv  Detail & Related papers  (2026-05-19T03:54:46Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion [27.567059323636112]
  We present a method to generate video-action pairs that follow text instructions, starting from an initial image observation and the robot's joint states.<n>Our approach automatically provides action labels for video diffusion models, overcoming the common lack of action annotations and enabling their full use for robotic policy learning.
  arXiv  Detail & Related papers  (2025-12-17T23:16:02Z)
• Show Me: Unifying Instructional Image and Video Generation with Diffusion Models [16.324312147741495]
  We propose a unified framework that enables image manipulation and video prediction.<n>We introduce structure and motion consistency rewards to improve structural fidelity and temporal coherence.<n> Experiments on diverse benchmarks demonstrate that our method outperforms expert models in both instructional image and video generation.
  arXiv  Detail & Related papers  (2025-11-21T23:24:28Z)
• Unified Video Action Model [47.88377984526902]
  A unified video and action model holds significant promise for robotics, where videos provide rich scene information for action prediction.<n>We introduce the Unified Video Action model (UVA), which jointly optimize video and action predictions to achieve both high accuracy and efficient action inference.<n>Via an extensive set of experiments, we demonstrate that UVA can serve as a general-purpose solution for a wide range of robotics tasks.
  arXiv  Detail & Related papers  (2025-02-28T21:38:17Z)
• Video Creation by Demonstration [59.389591010842636]
  We present $delta$-Diffusion, a self-supervised training approach that learns from unlabeled videos by conditional future frame prediction.<n>By leveraging a video foundation model with an appearance bottleneck design on top, we extract action latents from demonstration videos for conditioning the generation process.<n> Empirically, $delta$-Diffusion outperforms related baselines in terms of both human preference and large-scale machine evaluations.
  arXiv  Detail & Related papers  (2024-12-12T18:41:20Z)
• Generating Human Interaction Motions in Scenes with Text Control [66.74298145999909]
  We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model. To facilitate training, we embed annotated navigation and interaction motions within scenes.
  arXiv  Detail & Related papers  (2024-04-16T16:04:38Z)
• Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning [50.60891619269651]
  Control-A-Video is a controllable T2V diffusion model that can generate videos conditioned on text prompts and reference control maps like edge and depth maps. We propose novel strategies to incorporate content prior and motion prior into the diffusion-based generation process. Our framework generates higher-quality, more consistent videos compared to existing state-of-the-art methods in controllable text-to-video generation.
  arXiv  Detail & Related papers  (2023-05-23T09:03:19Z)
• Align and Prompt: Video-and-Language Pre-training with Entity Prompts [111.23364631136339]
  Video-and-language pre-training has shown promising improvements on various downstream tasks. We propose Align and Prompt: an efficient and effective video-and-language pre-training framework with better cross-modal alignment. Our code and pre-trained models will be released.
  arXiv  Detail & Related papers  (2021-12-17T15:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.