Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models

• URL: http://arxiv.org/abs/2507.13162v1
• Date: Thu, 17 Jul 2025 14:29:34 GMT
• Title: Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
• Abstract summary: Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios.<n>We develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras.<n>We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data.
• Score: 28.127179660781838
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios. In this work, we develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras. We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data. It particularly stands out in difficult scenarios like turning maneuvers and urban traffic. We test whether discrete token models possibly have advantages over continuous models based on flow matching. To this end, we set up a hybrid tokenizer that is compatible with both approaches and allows for a side-by-side comparison. Our study concludes in favor of the continuous autoregressive model, which is less brittle on individual design choices and more powerful than the model built on discrete tokens. Code, models and qualitative results are publicly available at https://lmb-freiburg.github.io/orbis.github.io/.

Related papers

• Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens [53.99177152562075]
  Scaling up autoregressive models in vision has not proven as beneficial as in large language models. We focus on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed order using BERT- or GPT-like transformer architectures. Our results show that while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends.
  arXiv  Detail & Related papers  (2024-10-17T17:59:59Z)
• Exploring Model Kinship for Merging Large Language Models [52.01652098827454]
  We introduce model kinship, the degree of similarity or relatedness between Large Language Models.<n>We find that there is a certain relationship between model kinship and the performance gains after model merging.<n>We propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-16T14:29:29Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Trajeglish: Traffic Modeling as Next-Token Prediction [67.28197954427638]
  A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs. We apply tools from discrete sequence modeling to model how vehicles, pedestrians and cyclists interact in driving scenarios. Our model tops the Sim Agents Benchmark, surpassing prior work along the realism meta metric by 3.3% and along the interaction metric by 9.9%.
  arXiv  Detail & Related papers  (2023-12-07T18:53:27Z)
• Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts [14.610244867640471]
  Recent vision-language models are driven by large-scale pretrained models. We introduce a parameter-efficient method to address challenges such as overfitting, catastrophic forgetting, and the cross-modal gap between vision and language. Our experiments on several video question answering benchmarks demonstrate the superiority of our approach in terms of performance and parameter efficiency.
  arXiv  Detail & Related papers  (2023-09-27T18:00:09Z)
• Constraining Generative Models for Engineering Design with Negative Data [11.432911164773488]
  We introduce a novel training method to guide a generative model toward constraint-satisfying outputs.<n>Our negative-data generative model (NDGM) formulation easily outperforms classic models.
  arXiv  Detail & Related papers  (2023-06-27T02:47:59Z)
• Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion [54.33764537135906]
  VideoQA Transformer models demonstrate competitive performance on standard benchmarks. Do these models capture the rich multimodal structures and dynamics from video and text jointly? Are they achieving high scores by exploiting biases and spurious features?
  arXiv  Detail & Related papers  (2023-06-15T06:45:46Z)
• Machine Learning Model Attribution Challenge [2.6532805035238747]
  Fine-tuned machine learning models may derive from other trained models without obvious attribution characteristics. In this challenge, participants identify the publicly-available base models that underlie a set of anonymous, fine-tuned large language models.
  arXiv  Detail & Related papers  (2023-02-13T22:05:27Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.