SWEET: Sparse World Modeling with Image Editing for Embodied Task Execution

• URL: http://arxiv.org/abs/2605.19319v1
• Date: Tue, 19 May 2026 03:54:46 GMT
• Title: SWEET: Sparse World Modeling with Image Editing for Embodied Task Execution
• Abstract summary: We study whether image editing models can serve as sparse visual world models for robot manipulation.<n>We propose SWEET, a one-shot prediction visual planning framework.<n>Experiments on DROID and RoboMimic show that SWEET improves prediction across seen and unseen scenes.
• Score: 61.612676324369595
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual prediction has emerged as a promising paradigm for embodied control, where future observations are generated and then translated into actions. However, dense video generation is computationally expensive and often unnecessary for many manipulation tasks, whose progress can be summarized by a small number of task-relevant visual states. In this work, we study whether image editing models can serve as sparse visual world models for robot manipulation by predicting task-level future states without dense video rollout. We first conduct a controlled comparison between the video generation model Wan2.2 and the image editing model FLUX-Kontext under the same robotic data setting, and find that image editing produces more reliable task-level keyframes with better visual fidelity and substantially lower inference cost. Motivated by this observation, we propose SWEET, a one-shot sparse visual planning framework that progressively generates a sequence of task-relevant manipulation keyframes through successive image editing, conditioned on language instructions and optional arrow-based spatial guidance. A goal-conditioned diffusion action predictor then converts adjacent imagined keyframes into executable action chunks. To reduce the mismatch between real and edited visual subgoals, we further introduce a mixed-training strategy with filtered edited targets. Experiments on DROID and RoboMimic show that SWEET improves keyframe prediction across seen and unseen scenes and enables a full pipeline from sequential keyframe planning to executable robot actions, suggesting that image editing is a promising and underexplored direction for embodied visual prediction.

Related papers

• ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing? [70.07849531480132]
  We propose ImageWAM, a framework that repurposes pretrained image editing models for robot action prediction.<n>ImageWAM does not decode the target frame at inference time; instead, it conditions a flow-matching action expert on the KV caches.<n>It also reduces FLOPs to 1/6 and latency to 1/4 of video-based WAMs.
  arXiv  Detail & Related papers  (2026-06-17T19:25:28Z)
• STRIPS-WM: Learning Grounded Propositional STRIPS-style World Models from Images [1.3535770763481905]
  STRIPS-WM is a framework for learning STRIPS-style world models directly from visual transitions.<n>It learns latent binary predicates and one grounded propositional operator per action label.<n>It forms a symbolic action model with sparse preconditions and add/delete effects.
  arXiv  Detail & Related papers  (2026-06-05T02:16:50Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection [40.21337735524356]
  IMAGAgent is a multi-turn image editing agent framework based on a "plan-execute-reflect" closed-loop mechanism.<n>It achieves deep synergy among instruction parsing, tool scheduling, and adaptive correction within a unified pipeline.<n>Experiments on our constructed textbfMTEditBench and the MagicBrush dataset demonstrate that IMAGAgent achieves performance significantly superior to existing methods.
  arXiv  Detail & Related papers  (2026-02-12T02:37:38Z)
• Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion [61.63215708592008]
  Embodied visual planning aims to enable manipulation tasks by imagining how a scene evolves toward a desired goal.<n>Video diffusion models provide a promising foundation for such visual imagination.<n>We propose Envision, a diffusion-based framework that performs visual planning for embodied agents.
  arXiv  Detail & Related papers  (2025-12-27T15:46:41Z)
• Image Generation as a Visual Planner for Robotic Manipulation [0.0]
  Generating realistic robotic manipulation videos is an important step toward unifying perception, planning, and action in embodied agents.<n>We propose a two-part framework that includes: (1) text-conditioned generation, which uses a language instruction and the first frame, and (2) trajectory-conditioned generation, which uses a 2D trajectory overlay and the same initial frame.<n>Our findings indicate that pretrained image generators encode transferable temporal priors and can function as video-like robotic planners under minimal supervision.
  arXiv  Detail & Related papers  (2025-11-29T15:54:16Z)
• Show Me: Unifying Instructional Image and Video Generation with Diffusion Models [16.324312147741495]
  We propose a unified framework that enables image manipulation and video prediction.<n>We introduce structure and motion consistency rewards to improve structural fidelity and temporal coherence.<n> Experiments on diverse benchmarks demonstrate that our method outperforms expert models in both instructional image and video generation.
  arXiv  Detail & Related papers  (2025-11-21T23:24:28Z)
• PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control [67.17998939712326]
  We present PoseDiff, a conditional diffusion model that unifies robot state estimation and control within a single framework.<n>At its core, PoseDiff maps raw visual observations into structured robot states-such as 3D keypoints or joint angles-from a single RGB image.<n>Building upon this foundation, PoseDiff extends naturally to video-to-action inverse dynamics.
  arXiv  Detail & Related papers  (2025-09-29T10:55:48Z)
• VeriGraph: Scene Graphs for Execution Verifiable Robot Planning [33.8868315479384]
  We propose VeriGraph, a framework that integrates vision-language models (VLMs) for robotic planning while verifying action feasibility. VeriGraph employs scene graphs as an intermediate representation, capturing key objects and spatial relationships to improve plan verification and refinement. Our approach significantly enhances task completion rates across diverse manipulation scenarios, outperforming baseline methods by 58% for language-based tasks and 30% for image-based tasks.
  arXiv  Detail & Related papers  (2024-11-15T18:59:51Z)
• Text-driven Video Prediction [83.04845684117835]
  We propose a new task called Text-driven Video Prediction (TVP) Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. To investigate the capability of text in causal inference for progressive motion information, our TVP framework contains a Text Inference Module (TIM)
  arXiv  Detail & Related papers  (2022-10-06T12:43:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.