TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy

• URL: http://arxiv.org/abs/2506.11302v3
• Date: Thu, 19 Jun 2025 15:10:29 GMT
• Title: TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy
• Abstract summary: We show how to permuting 360-degree panoramic imagery into rich interconnected observation, state and action nodes.<n>We benchmark this dataset via TARDIS, a transformer-based generative world model.<n>We demonstrate robust performance across a range of agentic tasks such as controllable image synthesis, instruction following, autonomous self-control, and state-of-the-art georeferencing.
• Score: 44.85881816317044
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: World models aim to simulate environments and enable effective agent behavior. However, modeling real-world environments presents unique challenges as they dynamically change across both space and, crucially, time. To capture these composed dynamics, we introduce a Spatio-Temporal Road Image Dataset for Exploration (STRIDE) permuting 360-degree panoramic imagery into rich interconnected observation, state and action nodes. Leveraging this structure, we can simultaneously model the relationship between egocentric views, positional coordinates, and movement commands across both space and time. We benchmark this dataset via TARDIS, a transformer-based generative world model that integrates spatial and temporal dynamics through a unified autoregressive framework trained on STRIDE. We demonstrate robust performance across a range of agentic tasks such as controllable photorealistic image synthesis, instruction following, autonomous self-control, and state-of-the-art georeferencing. These results suggest a promising direction towards sophisticated generalist agents--capable of understanding and manipulating the spatial and temporal aspects of their material environments--with enhanced embodied reasoning capabilities. Training code, datasets, and model checkpoints are made available at https://huggingface.co/datasets/Tera-AI/STRIDE.

Related papers

• DyG$^2$T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer [86.4618122245946]
  DyG$2$T is a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations.<n>Experiments on both synthetic and real-world datasets demonstrate that DyG$2$T achieves accurate dynamics modeling and reasoning.
  arXiv  Detail & Related papers  (2026-08-19T03:44:28Z)
• WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory [106.28234880163723]
  WorldDirector is a highly controllable video world model framework.<n>Our framework explicitly decouples semantic motion orchestration from visual generation.<n>We successfully preserve the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view.
  arXiv  Detail & Related papers  (2026-07-02T17:59:59Z)
• ViPSim: Collaborating Visual and Parameter Spaces for Consistent Long-Horizon Embodied World Models [11.15585863355844]
  Embodied World Models (EWMs) have emerged as a scalable and risk-free paradigm for advancing embodied intelligence.<n>EWMs are often hindered by the representation gap between low-dimensional actions and high-dimensional video synthesis.<n>We propose ViPSim, a framework that achieves consistent long-horizon generation.
  arXiv  Detail & Related papers  (2026-06-27T08:18:09Z)
• IOI: Decoupling Kinematics and Physics for Interactive World Models [46.3330122411516]
  We propose IOI, a hybrid interactive world model integrating kinematics with learned physical dynamics.<n> IOI introduces explicit kinematic guidance, computing kinematics sequences for accurate motion trajectories.<n> experiments on the RoboTwin benchmark validate IOI across kinematic fidelity, out-of-distribution, and policy evaluation.
  arXiv  Detail & Related papers  (2026-06-22T13:09:34Z)
• AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization [79.26935895370191]
  AnchorWorld is a framework that advances egocentric simulation through enhanced interaction integrity.<n> auxiliary training supervision incorporates viewpoints decoupled from the agent's first-person sensorium.<n>We propose a simple yet effective mechanism for customizing self-evolving worlds.
  arXiv  Detail & Related papers  (2026-06-05T14:43:13Z)
• Earth-o1: A Grid-free Observation-native Atmospheric World Model [81.55899748753434]
  We present Earth-o1, an observation-native atmospheric world model that overcomes structural limitations.<n>By integrating diverse sensor inputs into a unified, grid-free dynamical field, the model autonomously advances the atmospheric state in space and time.<n>We show that this fundamentally distinct paradigm enables direct, real-time forecasting and cross-sensor inference without the overhead of explicit numerical solvers.
  arXiv  Detail & Related papers  (2026-05-07T14:27:48Z)
• INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling [44.09983529522167]
  INSPATIO-WORLD is a novel real-time framework capable of recovering and generating high-fidelity interactive scenes from a single video.<n>Implicit Spatiotemporal Cache aggregates reference and historical observations into a latent world representation.<n>Explicit Spatial Constraint Module enforces geometric structure and translates user interactions into precise and physically plausible camera trajectories.
  arXiv  Detail & Related papers  (2026-04-08T15:31:22Z)
• Kinematics-Aware Latent World Models for Data-Efficient Autonomous Driving [8.391716372682316]
  Vehicle kinematic information is incorporated into the observation encoder to ground latent transitions in physically meaningful motion dynamics.<n> geometry-aware supervision regularizes the RSSM latent state to capture task-relevant spatial structure beyond pixel reconstruction.
  arXiv  Detail & Related papers  (2026-03-07T15:47:54Z)
• RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space [51.441415833480505]
  RAYNOVA is a multiview world model for driving scenarios that employs a dual-causal autoregressive framework.<n>It constructs an isotropic-temporal representation across views, frames, and scales based on relative Plcker-ray positional encoding.
  arXiv  Detail & Related papers  (2026-02-24T08:41:40Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment [16.343768407636322]
  We introduce Reinforcement Learning with World Grounding (RLWG), a self-supervised post-training framework that aligns pretrained world models with a physically verifiable structure through geometric and perceptual rewards.<n>We instantiate this framework with GrndCtrl, a reward-aligned adaptation method based on Group Relative Policy Optimization (GRPO), yielding world models that maintain stable trajectories, consistent geometry, and reliable rollouts for embodied navigation.
  arXiv  Detail & Related papers  (2025-12-01T18:03:29Z)
• SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world models [42.814012901180774]
  textbfSAMPO is a hybrid framework that combines visual autoregressive modeling for intra-frame generation with causal modeling for next-frame generation.<n>We show that SAMPO achieves competitive performance in action-conditioned video prediction and model-based control.<n>We also evaluate SAMPO's zero-shot generalization and scaling behavior, demonstrating its ability to generalize to unseen tasks.
  arXiv  Detail & Related papers  (2025-09-19T02:41:37Z)
• ArmGS: Composite Gaussian Appearance Refinement for Modeling Dynamic Urban Environments [22.371417505012566]
  This work focuses on modeling dynamic urban environments for autonomous driving simulation.<n>We propose a new approach named ArmGS that exploits composite driving Gaussian splatting with multi-granularity appearance refinement.<n>This not only models global scene appearance variations between frames and camera viewpoints, but also models local fine-grained photorealistic changes of background and objects.
  arXiv  Detail & Related papers  (2025-07-05T03:54:40Z)
• Dyn-O: Building Structured World Models with Object-Centric Representations [42.65409148846005]
  We introduce Dyn-O, an enhanced structured world model built upon object-centric representations.<n>Compared to prior work in object-centric representations, Dyn-O improves in both learning representations and modeling dynamics.<n>We find that our method can learn object-centric world models directly from pixel observations, outperforming DreamerV3 in rollout prediction accuracy.
  arXiv  Detail & Related papers  (2025-07-04T05:06:15Z)
• GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control [50.67481583744243]
  We introduce GeoDrive, which explicitly integrates robust 3D geometry conditions into driving world models.<n>We propose a dynamic editing module during training to enhance the renderings by editing the positions of the vehicles.<n>Our method significantly outperforms existing models in both action accuracy and 3D spatial awareness.
  arXiv  Detail & Related papers  (2025-05-28T14:46:51Z)
• TiMo: Spatiotemporal Foundation Model for Satellite Image Time Series [39.22426645737932]
  TiMo is a novel hierarchical vision transformer foundation model tailored for SITS analysis.<n>At its core, we introduce atemporal attention mechanism that dynamically captures multiscale patterns across both time and space.<n>Extensive experiments across multipletemporal tasks-including deforestation monitoring-demonstrate TiMo's superiority over state-of-theart methods.
  arXiv  Detail & Related papers  (2025-05-13T16:35:11Z)
• LaM-SLidE: Latent Space Modeling of Spatial Dynamical Systems via Linked Entities [11.76748620770499]
  We present LaM-SLidE (Latent Space Modeling of Spatial Dynamical Systems via Linked Entities)<n>LaM-SLidE bridges the gap between: (1) keeping the traceability of individual entities in a latent system representation, and (2) leveraging the efficiency and scalability of recent advances in image and video generation.<n>We show that LaM-SLidE performs favorably in terms of speed, accuracy, and generalizability.
  arXiv  Detail & Related papers  (2025-02-17T18:49:13Z)
• TriHuman : A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis [76.73338151115253]
  TriHuman is a novel human-tailored, deformable, and efficient tri-plane representation. We non-rigidly warp global ray samples into our undeformed tri-plane texture space. We show how such a tri-plane feature representation can be conditioned on the skeletal motion to account for dynamic appearance and geometry changes.
  arXiv  Detail & Related papers  (2023-12-08T16:40:38Z)
• Spatio-temporal Tendency Reasoning for Human Body Pose and Shape Estimation from Videos [10.50306784245168]
  We present atemporal tendency reasoning (STR) network for recovering human body pose shape from videos. Our STR aims to learn accurate and spatial motion sequences in an unconstrained environment. Our STR remains competitive with the state-of-the-art on three datasets.
  arXiv  Detail & Related papers  (2022-10-07T16:09:07Z)
• A Spatio-Temporal Multilayer Perceptron for Gesture Recognition [70.34489104710366]
  We propose a multilayer state-weighted perceptron for gesture recognition in the context of autonomous vehicles. An evaluation of TCG and Drive&Act datasets is provided to showcase the promising performance of our approach. We deploy our model to our autonomous vehicle to show its real-time capability and stable execution.
  arXiv  Detail & Related papers  (2022-04-25T08:42:47Z)
• Tracking and Planning with Spatial World Models [17.698319441265223]
  We introduce a method for real-time navigation and tracking with differentiably rendered world models. We achieve up to 92% navigation success rate at a frequency of 15 Hz using only image and depth observations.
  arXiv  Detail & Related papers  (2022-01-25T14:16:46Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• CLOUD: Contrastive Learning of Unsupervised Dynamics [19.091886595825947]
  We propose to learn forward and inverse dynamics in a fully unsupervised manner via contrastive estimation. We demonstrate the efficacy of our approach across a variety of tasks including goal-directed planning and imitation from observations.
  arXiv  Detail & Related papers  (2020-10-23T15:42:57Z)
• A Spatial-Temporal Attentive Network with Spatial Continuity for Trajectory Prediction [74.00750936752418]
  We propose a novel model named spatial-temporal attentive network with spatial continuity (STAN-SC) First, spatial-temporal attention mechanism is presented to explore the most useful and important information. Second, we conduct a joint feature sequence based on the sequence and instant state information to make the generative trajectories keep spatial continuity.
  arXiv  Detail & Related papers  (2020-03-13T04:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.