EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation

• URL: http://arxiv.org/abs/2603.05757v1
• Date: Thu, 05 Mar 2026 23:31:56 GMT
• Title: EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation
• Abstract summary: Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos.<n>We present method, a data-free framework that aligns VGM outputs with insight generated by vision-language models at inference time.<n>We evaluate method on six real-robot manipulation tasks requiring precise, constraint-sensitive execution.
• Score: 17.037238350207513
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos that capture rich object dynamics, offering a compelling foundation for zero-shot robotic manipulation. However, VGMs often produce physically implausible rollouts, and converting their pixel-space motion into robot actions through geometric retargeting further introduces cumulative errors from imperfect depth estimation and keypoint tracking. To address these challenges, we present \method{}, a data-free framework that aligns VGM outputs with compositional constraints generated by vision-language models (VLMs) at inference time. The key insight is that VLMs offer a capability complementary to VGMs: structured spatial reasoning that can identify the physical constraints critical to the success and safety of manipulation execution. Given a language instruction, \method{} uses a VLM to automatically extract a set of compositional constraints capturing task-specific requirements, which are then applied at two stages: (1) constraint-guided rollout selection, which scores and filters a batch of VGM rollouts to retain the most physically plausible candidate, and (2) constraint-based trajectory optimization, which uses the selected rollout as initialization and refines the robot trajectory under the same constraint set to correct retargeting errors. We evaluate \method{} on six real-robot manipulation tasks requiring precise, constraint-sensitive execution, improving the overall success rate by 43.3\% points over the strongest baseline without any task-specific training data.

Related papers

• PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation [17.70834422785338]
  Video-based manipulation provides object-centric motion priors from human demonstrations, generated videos, or RGB-D observations.<n>PhysV2A is a framework for converting video-derived 6D object motion into robot-executable manipulation trajectories.
  arXiv  Detail & Related papers  (2026-07-10T12:43:00Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models [9.25715767290436]
  We propose DRIFT, a framework for adapting pretrained vision-language models to continuous decoding tasks.<n>DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching.<n>We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control.
  arXiv  Detail & Related papers  (2026-06-04T06:37:10Z)
• CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL [56.969946199335716]
  We introduce a compositional constraint-based reward model for post-training embodied video generation models.<n>We propose CreFlow, a novel online RL framework with two key designs.
  arXiv  Detail & Related papers  (2026-05-14T02:18:58Z)
• V-CAGE: Vision-Closed-Loop Agentic Generation Engine for Robotic Manipulation [6.820118518027692]
  We present V-CAGE, an agentic framework for autonomous robotic data synthesis.<n>Unlike traditional scripted pipelines, V-CAGE operates as an embodied agentic system.<n>To overcome the storage bottleneck of massive video datasets, we implement a perceptually-driven compression algorithm.
  arXiv  Detail & Related papers  (2026-04-10T06:56:17Z)
• RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset [48.645870795753105]
  We introduce Robust Autonomous Data Acquisition for Robotics (RADAR)<n>RADAR is a fully autonomous, closed-loop data generation engine that completely removes human intervention from the collection cycle.<n>In simulation, our framework achieves up to 90% success rates on complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-03-12T11:18:52Z)
• Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance [92.72779885657373]
  We propose a framework that grounds model selection in the internal functional dynamics of the visual encoder.<n>Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment.<n>Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks.
  arXiv  Detail & Related papers  (2026-02-01T17:29:43Z)
• Plan Verification for LLM-Based Embodied Task Completion Agents [10.439882851477162]
  Large language model (LLM) based task plans and corresponding human demonstrations for embodied AI may be noisy.<n>We propose an iterative verification framework in which a Judge LLM critiques action sequences and a Planner LLM applies the revisions.
  arXiv  Detail & Related papers  (2025-09-02T19:06:56Z)
• Constrained Decoding for Robotics Foundation Models [12.916330118607918]
  We introduce SafeDec, a constrained decoding framework for autoregressive robot foundation models.<n>Task-specific safety rules are expressed as Signal Temporal Logic (STL) formulas and are enforced at inference time with minimal overhead.<n>Our method ensures that generated actions provably satisfy STL specifications under assumed dynamics at runtime without retraining.
  arXiv  Detail & Related papers  (2025-09-01T19:17:40Z)
• Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulation [2.434849352801735]
  Vision-Language Models (VLMs) demonstrate remarkable potential in robotic manipulation.<n>But challenges persist in executing complex fine manipulation tasks with high speed and precision.<n>We introduce a progressive VLM planning algorithm that empowers robots to perform fast, precise, and error-correctable fine manipulation.
  arXiv  Detail & Related papers  (2025-03-07T00:55:42Z)
• GRAPE: Generalizing Robot Policy via Preference Alignment [58.419992317452376]
  We present GRAPE: Generalizing Robot Policy via Preference Alignment.<n>We show GRAPE increases success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively.<n> GRAPE can be aligned with various objectives, such as safety and efficiency, reducing collision rates by 37.44% and rollout step-length by 11.15%, respectively.
  arXiv  Detail & Related papers  (2024-11-28T18:30:10Z)
• Vision Language Models are In-Context Value Learners [89.29486557646624]
  We present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks.
  arXiv  Detail & Related papers  (2024-11-07T09:17:50Z)
• RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents [105.13169239919272]
  We propose RH20T-P, a primitive-level robotic manipulation dataset.<n>It contains about 38k video clips covering 67 diverse manipulation tasks in real-world scenarios.<n>We standardize a plan-execute CGA paradigm and implement an exemplar baseline called RA-P on our RH20T-P.
  arXiv  Detail & Related papers  (2024-03-28T17:42:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.