Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness

• URL: http://arxiv.org/abs/2501.15842v1
• Date: Mon, 27 Jan 2025 08:08:17 GMT
• Title: Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness
• Authors: Yue Yao, Daniel Goehring, Joerg Reichardt,
• Abstract summary: We study the Out-of-Distribution (OoD) generalization ability of three SotA trajectory prediction models with comparable In-Distribution (ID) performance but different model designs. We find that the smallest model with highest inductive bias exhibits the best OoD generalization across different augmentation strategies.
• Score: 2.0965639599405366
• License:
• Abstract: We study the Out-of-Distribution (OoD) generalization ability of three SotA trajectory prediction models with comparable In-Distribution (ID) performance but different model designs. We investigate the influence of inductive bias, size of training data and data augmentation strategy by training the models on Argoverse 2 (A2) and testing on Waymo Open Motion (WO) and vice versa. We find that the smallest model with highest inductive bias exhibits the best OoD generalization across different augmentation strategies when trained on the smaller A2 dataset and tested on the large WO dataset. In the converse setting, training all models on the larger WO dataset and testing on the smaller A2 dataset, we find that all models generalize poorly, even though the model with the highest inductive bias still exhibits the best generalization ability. We discuss possible reasons for this surprising finding and draw conclusions about the design and test of trajectory prediction models and benchmarks.

Related papers

• Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoders [13.474737752636608]
  We present the largest comprehensive analysis to-date of how the upstream pre-training factors and downstream performance of CLIP models relate to intrinsic biases. We study 131 unique CLIP models, trained on 26 datasets, using 55 architectures, and in a variety of sizes. We find that the choice of pre-training dataset is the most significant upstream predictor of bias, whereas architectural variations have minimal impact.
  arXiv  Detail & Related papers  (2025-02-11T21:11:47Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• Improving Out-of-Distribution Generalization of Trajectory Prediction for Autonomous Driving via Polynomial Representations [16.856874154363588]
  We present an OoD testing protocol that homogenizes datasets and prediction tasks across two large-scale motion datasets. With a much smaller model size, training effort, and inference time, we reach near SotA performance for ID testing and significantly improve robustness in OoD testing.
  arXiv  Detail & Related papers  (2024-07-18T12:00:32Z)
• GeoBench: Benchmarking and Analyzing Monocular Geometry Estimation Models [41.76935689355034]
  Discriminative and generative pretraining have yielded geometry estimation models with strong generalization capabilities. We build fair and strong baselines for evaluating and analyzing the geometry estimation models. We evaluate monocular geometry estimators on more challenging benchmarks for geometry estimation task with diverse scenes and high-quality annotations.
  arXiv  Detail & Related papers  (2024-06-18T14:44:12Z)
• Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning [0.2812395851874055]
  This paper proposes a comprehensive approach using multiple methods to remove bias in AI models. We train multiple models with the counter-bias of the pre-trained model through data splitting, local training, and regularized fine-tuning. We conclude our solution with knowledge distillation that results in a single unbiased neural network.
  arXiv  Detail & Related papers  (2024-02-01T09:24:36Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases [17.431381376675432]
  In this paper we explore the performance of large language models based on instruction tuning across different scales of instruction data. With Bloomz-7B1-mt as the base model, the results show that merely increasing the amount of instruction data leads to continuous improvement in tasks such as open-ended generation. We propose potential future research directions such as effectively selecting high-quality training data, scaling base models and training methods specialized for hard tasks.
  arXiv  Detail & Related papers  (2023-03-26T14:49:37Z)
• Comparing Test Sets with Item Response Theory [53.755064720563]
  We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
  arXiv  Detail & Related papers  (2021-06-01T22:33:53Z)
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning [109.74041512359476]
  We study a number of design decisions for the predictive model in visual MBRL algorithms. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks will perform the same as the best-performing models when trained on the same training data.
  arXiv  Detail & Related papers  (2020-12-08T18:03:21Z)
• Improving QA Generalization by Concurrent Modeling of Multiple Biases [61.597362592536896]
  Existing NLP datasets contain various biases that models can easily exploit to achieve high performances on the corresponding evaluation sets. We propose a general framework for improving the performance on both in-domain and out-of-domain datasets by concurrent modeling of multiple biases in the training data. We extensively evaluate our framework on extractive question answering with training data from various domains with multiple biases of different strengths.
  arXiv  Detail & Related papers  (2020-10-07T11:18:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.