From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation

• URL: http://arxiv.org/abs/2605.08712v1
• Date: Sat, 09 May 2026 05:48:51 GMT
• Title: From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation
• Abstract summary: Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery.<n>We propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities.<n>Our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines.
• Score: 43.4347713269043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Action-conditioned surgical video generation is a critical yet highly challenging problem for robotic surgery. The core difficulty is that low-dimensional control vectors must precisely govern complex image-space evolution. In this work, we propose a kinematic-to-visual lifting paradigm that converts articulated kinematics into a unified set of five image-aligned control modalities. Building on this representation, we introduce a hierarchically routed visual control framework that selectively activates the most relevant control modalities and motion scales. Instead of uniformly applying all control signals, our model performs hierarchical routing to dynamically allocate conditioning capacity. We further design kinematic-prior-guided routing loss functions to ensure physically meaningful, temporally stable, and efficient expert utilization. To improve efficiency, we propose a budgeted training and inference scheme that leverages routing-induced sparsity. By selectively discarding low-significance control pathways during training and execution, our approach enables adaptive computation that is complementary to standard distillation. We additionally construct a new benchmark with curated articulated annotations, obtained through human-in-the-loop semantic labeling and differentiable pose tracking, providing realistic supervision for action-conditioned surgical video generation. Extensive experiments demonstrate that our method consistently improves action faithfulness, visual fidelity, and cross-domain generalization over diverse baselines. Moreover, our efficient variant achieves substantial reductions in latency while maintaining strong control accuracy.

Related papers

• EmbodiedVAE: Disentangled Video VAE for Efficient and Controllable Embodied Manipulation [35.20482569112933]
  We present EmbodiedVAE, a novel video VAE that provides compact yet controllable latent representations tailored for robotic manipulation world models.<n>Our proposed EmbodiedVAE achieves superior reconstruction quality with high compression rate, while enabling more precise action control in robotic manipulation scenarios with an average of 2dB PSNR improvement over state-of-the-art video VAEs.
  arXiv  Detail & Related papers  (2026-08-04T01:01:18Z)
• Prior-First, Condition-Second: Scalable and Controllable Hand Motion Completion [27.154998447721102]
  We propose a prior-first, condition-second framework for body-conditioned hand motion completion.<n>Our framework generates coherent, kinematically consistent hand motion from body dynamics in real time.<n>Our framework improves kinematic plausibility, robustness, and controllability compared to end-to-end conditioned baselines.
  arXiv  Detail & Related papers  (2026-07-07T07:40:13Z)
• AI-Enabled Image-Based Hybrid Vision/Force Control of Tendon-Driven Aerial Continuum Manipulators [0.6999740786886535]
  This paper presents an AI-enabled cascaded hybrid vision/force control framework for tendon-driven aerial manipulators.<n>The proposed controller is designed to enable autonomous, physical interaction with a static environment while stabilizing the image feature error.
  arXiv  Detail & Related papers  (2026-04-21T01:19:11Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation [50.39748673817223]
  We introduce two training-free, inference-time techniques that fully exploit explicit action parameters in robot video generation.<n>First, action-scaled classifier-free guidance dynamically modulates guidance strength in proportion to action magnitude, enhancing controllability over motion intensity.<n>Second, action-scaled noise truncation adjusts the distribution of initially sampled noise to better align with the desired motion dynamics.
  arXiv  Detail & Related papers  (2025-09-29T03:30:40Z)
• Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control [16.319698848279966]
  We present Diffuse-CLoC, a guided diffusion framework for physics-based look-ahead control.<n>It enables intuitive, steerable, and physically realistic motion generation.
  arXiv  Detail & Related papers  (2025-03-14T18:42:29Z)
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
  In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
  arXiv  Detail & Related papers  (2024-04-05T17:58:37Z)
• TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models [75.20168902300166]
  We propose TrackDiffusion, a novel video generation framework affording fine-grained trajectory-conditioned motion control. A pivotal component of TrackDiffusion is the instance enhancer, which explicitly ensures inter-frame consistency of multiple objects. generated video sequences by our TrackDiffusion can be used as training data for visual perception models.
  arXiv  Detail & Related papers  (2023-12-01T15:24:38Z)
• Interactive Character Control with Auto-Regressive Motion Diffusion Models [18.727066177880708]
  We propose A-MDM (Auto-regressive Motion Diffusion Model) for real-time motion synthesis. Our conditional diffusion model takes an initial pose as input, and auto-regressively generates successive motion frames conditioned on previous frame. We introduce a suite of techniques for incorporating interactive controls into A-MDM, such as task-oriented sampling, in-painting, and hierarchical reinforcement learning.
  arXiv  Detail & Related papers  (2023-06-01T07:48:34Z)
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives [89.34229413345541]
  We propose a conditioning scheme which avoids pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion. We report significant improvements in task success over representative MPC and IL baselines.
  arXiv  Detail & Related papers  (2020-03-19T15:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.