AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction

• URL: http://arxiv.org/abs/2608.28491v1
• Date: Fri, 28 Aug 2026 16:19:24 GMT
• Title: AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction
• Abstract summary: We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance.<n>A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last frame.<n>On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce error by 21.0% over persistence.
• Score: 14.556446786016787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting robot videos requires both precise motion reasoning and preservation of high-frequency appearance, yet monolithic pixel models entangle these objectives and often conceal their progress behind a strong last-frame baseline. We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance. A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent. A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM. A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last observed frame; a residual refiner and learned delivery mask produce five future frames without access to future appearance. On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0\% over persistence. Across three delivery-mask seeds, AcrossVAM1.0 improves future-frame PSNR/SSIM from 19.97/0.796 to 20.573/0.8004, while raw particle generation improves motion-region PSNR from 11.89 to 13.23. The delivered model does not yet beat persistence in LPIPS, and correct-versus- shuffled language changes trajectory error by only 2.8--3.1%. We report these limitations alongside oracle, negative-control, multi-seed, and per-robot analyses. The results show that explicit particle dynamics are a promising low-dimensional interface for robot video prediction, while robust language grounding and appearance delivery remain the principal open challenges.

Related papers

• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation [37.9748770367306]
  Video generation models can produce physically implausible manipulations, including inconsistent robot-object interactions.<n>We propose PhysisForcing, a training framework that strengthens physical consistency by focusing supervision on physics-informative regions.<n>Our experiments show that PhysisForcing consistently improves embodied video generation over strong baselines.
  arXiv  Detail & Related papers  (2026-06-26T14:30:18Z)
• DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control [16.562259973551786]
  We introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer.<n>Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process.<n>It achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1.
  arXiv  Detail & Related papers  (2026-03-11T06:03:53Z)
• Masked Modeling for Human Motion Recovery Under Occlusions [21.05382087890133]
  MoRo is an end-to-end generative framework that formulates motion reconstruction as a video-conditioned task.<n>MoRo achieves real-time inference at 70 FPS on a single H200 GPU.
  arXiv  Detail & Related papers  (2026-01-22T16:22:20Z)
• InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation [77.07565723756119]
  We present InternVLA-A1, a vision-language-action model with dynamic predictive capabilities.<n>We pre-train these models on heterogeneous data sources over real-world robot data, synthetic simulation data, and human videos.<n>We evaluate InternVLA-A1 on 12 real-world robotic tasks and a simulation benchmark.
  arXiv  Detail & Related papers  (2026-01-05T18:54:29Z)
• M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation [65.48046909056468]
  We reformulate talking head generation into a unified framework comprising video preprocessing, motion representation, and rendering reconstruction.<n>M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness.
  arXiv  Detail & Related papers  (2025-07-11T04:48:12Z)
• EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation [55.26713167507132]
  We introduce a generative robotics foundation model that constructs and interprets embodied spaces.<n>EnerVerse employs an autoregressive video diffusion framework to predict future embodied spaces from instructions, enhanced by a sparse context memory for long-term reasoning.<n>We present EnerVerse-D, a data engine pipeline combining the generative model with 4D Gaussian Splatting, forming a self-reinforcing data loop to reduce the sim-to-real gap.
  arXiv  Detail & Related papers  (2025-01-03T17:00:33Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• Robust Human Motion Forecasting using Transformer-based Model [14.088942546585068]
  We propose a new model based on Transformer that deals with the real time 3D human motion forecasting in short and long term. Our model is tested in conditions where the human motion is severely occluded, demonstrating its robustness in reconstructing and predicting 3D human motion in a highly noisy environment. Our model reduces in 8.89% the mean squared error of ST-Transformer in short-term prediction, and 2.57% in long-term prediction in Humanrere3.6M dataset with 400ms input prefix.
  arXiv  Detail & Related papers  (2023-02-16T13:06:39Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.