Diffusion Transformer World-Action Model for AV Scene Prediction

• URL: http://arxiv.org/abs/2606.12987v1
• Date: Thu, 11 Jun 2026 07:24:12 GMT
• Title: Diffusion Transformer World-Action Model for AV Scene Prediction
• Abstract summary: Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls.<n>But at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead.<n>We confront this with a compact latent world model that predicts future scene latents a frozen decoder renders to $256 times 256$ frames up to 8 seconds ahead.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls, enabling planning and simulation without real-world rollouts, but at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead: they reward a blurry regression mean over a realistic prediction. We confront this with a compact latent world model that, given the present front-camera latent and a sequence of ego-actions, predicts future scene latents a frozen decoder renders to $256 \times 256$ frames up to 8 seconds ahead, evaluated on 150 held-out nuScenes scenes. We first benchmark where to predict: across six frozen encoders spanning four representation families, V-JEPA2 with temporal context reduces steering RMSE by 40% over the best single-frame encoder. We then train a latent Diffusion Transformer (DiT) and, through a controlled diagnosis, identify the four ingredients it needs: spatial tokens, the $x_0$ objective, residual anchoring, and sampling matched to target uncertainty. In a Stable-Diffusion-VAE encode-predict-decode pipeline we expose the central tension: distortion metrics (cosine similarity, SSIM) favor the blurry mean, masking that the diffusion model is far closer to the real frame distribution. Inception-based FID and KID reveal a clean perception-distortion frontier: diffusion attains KID 0.078 versus 0.375 for regression ($4.8\times$ better), and a deployable train-derived calibration makes this practical without test-time ground truth. The model is genuinely action-controllable (steering drives scene displacement, Spearman $ρ= 0.81$, vs $-0.18$ for regression). We trace limited single-pass motion to a shared-present anchor and engineer a compact 1.7M-parameter "jump" model that recovers full ground-truth motion magnitude ($1.02\times$ GT), where single-pass models capture less than half.

Related papers

• Disparity Has a Sign: Stereo Matching Beyond the Zero-Disparity Plane [52.210414985125816]
  We show how a stereoscopic content contains objects behind the zero-disparity plane corresponding to negative disparities.<n>We propose textitZDPShift, a benchmark of $21,495$ stereo pairs from seven cinematographer-authored open movies.<n>On identical scene content, Foundation goes from $2.24$ EPE to $75.33$px, with every backbone leaving roughly half of all pixels exceeding a three-pixel disparity error.
  arXiv  Detail & Related papers  (2026-09-06T19:58:35Z)
• AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction [14.556446786016787]
  We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance.<n>A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last frame.<n>On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce error by 21.0% over persistence.
  arXiv  Detail & Related papers  (2026-08-28T16:19:24Z)
• LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics [50.45297537821907]
  We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective.<n>A single encoder is trained with an invariance loss over global and local views of a clip.<n>LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute.
  arXiv  Detail & Related papers  (2026-08-27T17:26:24Z)
• Interpretable Fuzzy Inference for UAV Target Tracking Using Bounding-Box Geometry [0.528933036850801]
  We present an interpretable fuzzy-inference framework that generates continuous yaw commands from low-dimensional features extracted from YOLO boxes.<n>Results show that the framework is transparent, data-efficient, computationally lightweight, and suitable for real-time vision-based UAV guidance toward mobile ground targets.
  arXiv  Detail & Related papers  (2026-08-04T18:18:29Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision [78.04385989413382]
  Video diffusion models generate visually compelling content but routinely violate elementary physics when the subject involves fluids.<n>We build a physics-simulation fluid dataset combining 1,638 MPM-simulated pouring/sloshing videos with 2,320 keyword-filtered real pouring videos.<n>We introduce a dual-stream image-to-video architecture built on a pretrained diffusion-filtereder video generator.
  arXiv  Detail & Related papers  (2026-07-28T06:13:18Z)
• Toward Goal-Agnostic Joint-Embedding Predictive Control of Partial Differential Equations [0.0]
  We present a goal-agnostic control framework for partial differential equations (PDEs) built around a joint-embedding predictive architecture (JEPA)<n>We find that when available, the control objective is better applied to an explicit physical observable (provided injectivity) than to minimizing raw Euclidean distance ($L2$) in the learned latent space.
  arXiv  Detail & Related papers  (2026-07-21T20:49:07Z)
• Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots [7.140499031711496]
  Force and contact leave almost no trace in a single frame, and raw vision features read force at $R2$ at or below $0.10$ on every robot we test.<n>We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that treats robot state as a modality and fuses vision, proprioception, and force/torque into a single shared latent.<n>On held-out scenes, the vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched vision-only control on every sensored robot.
  arXiv  Detail & Related papers  (2026-07-15T07:21:00Z)
• SKIP: Sparse Keyframe Interpolation Paradigm for Efficient Embodied World Models [34.49867155987541]
  We propose Sparse Keyframe Interpolation Paradigm (SKIP), an event-preserving sparse-to-dense framework that avoids dense frame-by-frame generation.<n>SKIP identifies task-relevants by leveraging robot-aware multimodal features.<n>It then synthesizes only these Ems with a sparse video diffusion model.<n>A learned gap predictor videos and an action-conditioned interpolator subsequently reconstruct the missing intervals according to the robot actions.
  arXiv  Detail & Related papers  (2026-05-30T10:41:34Z)
• Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning [18.397872306430006]
  Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence.<n>These models are optimized for short-term prediction and break down when deployed autoregressively.<n>We introduce a reinforcement learning scheme that trains the world model on its own autoregressive rollouts.
  arXiv  Detail & Related papers  (2026-03-26T17:36:08Z)
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels [49.35636088613484]
  Joint Embedding Predictive Architectures (JEPAs) offer a compelling framework for learning world models in compact latent spaces.<n>We introduce LeModelWorld, the first JEPA that trains stably end-to-end from raw pixels using only two loss terms.<n>With 15M parameters trainable on a single GPU in a few hours, LeWM plans up to 48x faster than foundation-model-based world models.
  arXiv  Detail & Related papers  (2026-03-13T19:48:14Z)
• DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control [16.562259973551786]
  We introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer.<n>Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process.<n>It achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1.
  arXiv  Detail & Related papers  (2026-03-11T06:03:53Z)
• Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation [76.04880323498598]
  We introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX)<n>Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains.
  arXiv  Detail & Related papers  (2025-12-12T18:56:35Z)
• Delta-Triplane Transformers as Occupancy World Models [57.16979927973973]
  Occupancy World Models (OWMs) aim to predict future scenes via 3D voxelized representations of the environment to support intelligent motion planning.<n>We propose Delta-Triplane Transformers (DTT), a novel 4D OWM for autonomous driving, that introduces two key innovations.<n>DTT delivers a 1.44$times$ speedup (26 FPS) over the state of the art, improves mean IoU to 30.85, and reduces the mean absolute planning error to 1.0 meters.
  arXiv  Detail & Related papers  (2025-03-10T13:50:23Z)
• A Fair Ranking and New Model for Panoptic Scene Graph Generation [51.78798765130832]
  Decoupled SceneFormer (DSFormer) is a novel two-stage model that outperforms all existing scene graph models. As a core design principle, DSFormer encodes subject and object masks directly into feature space.
  arXiv  Detail & Related papers  (2024-07-12T12:28:08Z)
• Model Predictive Simulation Using Structured Graphical Models and Transformers [4.229560419171488]
  We propose an approach to simulating trajectories of multiple interacting agents (road users) based on transformers and probabilistic graphical models (PGMs) We then improve upon these generated trajectories using a PGM, which contains factors which encode prior knowledge. We show that MPS improves upon the MTR baseline, especially in safety critical metrics such as collision rate.
  arXiv  Detail & Related papers  (2024-06-28T03:46:53Z)
• Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost [53.746169882193456]
  Recent works have proposed various sparse attention modules to overcome the quadratic cost of self-attention. We propose a model that resolves both problems by endowing each attention head with a mixed-membership Block Model. Our model outperforms previous efficient variants as well as the original Transformer with full attention.
  arXiv  Detail & Related papers  (2022-10-27T15:30:52Z)
• It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-training [76.69480467101143]
  Self-supervised video transformer pre-training has recently benefited from the mask-and-predict pipeline. We explicitly investigate motion cues in videos as extra prediction target and propose our Masked Appearance-Motion Modeling framework. Our method learns generalized video representations and achieves 82.3% on Kinects-400, 71.3% on Something-Something V2, 91.5% on UCF101, and 62.5% on HMDB51.
  arXiv  Detail & Related papers  (2022-10-11T08:05:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.