Building a Foundation Model for Trajectory from Scratch

• URL: http://arxiv.org/abs/2511.20610v1
• Date: Tue, 25 Nov 2025 18:37:55 GMT
• Title: Building a Foundation Model for Trajectory from Scratch
• Abstract summary: Foundation models are transformative in artificial intelligence, but building them from scratch, especially for mobility trajectories, is not yet clear or documented.<n>This tutorial bridges this gap by demonstrating the steps and code of a minimal implementation of a trajectory-focused foundation model starting from GPT-2.
• Score: 0.764671395172401
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Foundation models are transformative in artificial intelligence, but building them from scratch, especially for mobility trajectories, is not yet clear or documented. This tutorial bridges this gap by demonstrating the steps and code of a minimal implementation of a trajectory-focused foundation model starting from GPT-2. Through a concise, step-by-step, code-driven process, we demonstrate adapting GPT-2 for spatiotemporal data. We then review and compare representative trajectory foundation models, such as TrajFM and TrajGPT, highlighting their architectural innovations and differences. Additionally, we introduce complementary techniques from related domains, like TimesFM's patching approach. Targeted at researchers and practitioners, this tutorial aims to explain the concepts and terminology of foundation models, at the implementation level. We find it timely and indispensable to create this educational material in order to support the SIGSPATIAL community in building and evaluating mobility foundation models, enhancing both research clarity and peer-review effectiveness in mobility AI.

Related papers

• The Emerging Paradigm of Geospatial Foundation Models: From Pre-Training to Agentic Reasoning [0.0]
  The analysis of satellite and aerial imagery has entered a new era with the advent of foundation models.<n>This paper describes the concept of Geospatial Foundation Models (GeoFMs), which are artificial intelligence/machine learning (AI/ML) models pre-trained on massive geospatial datasets.
  arXiv  Detail & Related papers  (2026-07-13T21:50:50Z)
• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges [53.47232506143113]
  Multi-modal foundation models have transformed the technology for autonomous driving.<n>We provide a comprehensive examination of such methods through a unifying taxonomy.<n>We assess these approaches with respect to the openness of their source code and datasets.
  arXiv  Detail & Related papers  (2025-10-31T18:05:02Z)
• A Survey on Efficient Vision-Language-Action Models [153.11669266922993]
  Vision-Language-Action models (VLAs) represent a significant frontier in embodied intelligence, aiming to bridge digital knowledge with physical-world interaction.<n>Motivated by the urgent need to address these challenges, this survey presents the first comprehensive review of Efficient Vision-Language-Action models.
  arXiv  Detail & Related papers  (2025-10-27T17:57:33Z)
• Foundation Models and Transformers for Anomaly Detection: A Survey [2.3264194695971656]
  The survey categorizes VAD methods into reconstruction-based, feature-based and zero/few-shot approaches.<n>Transformers and foundation models enable more robust, interpretable, and scalable anomaly detection solutions.
  arXiv  Detail & Related papers  (2025-07-21T12:01:04Z)
• A Survey of Model Architectures in Information Retrieval [59.61734783818073]
  The period from 2019 to the present has represented one of the biggest paradigm shifts in information retrieval (IR) and natural language processing (NLP)<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude with a forward-looking discussion of emerging challenges and future directions.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• Vision Foundation Models in Remote Sensing: A Survey [6.036426846159163]
  Foundation models are large-scale, pre-trained AI models capable of performing a wide array of tasks with unprecedented accuracy and efficiency.<n>This survey aims to serve as a resource for researchers and practitioners by providing a panorama of advances and promising pathways for continued development and application of foundation models in remote sensing.
  arXiv  Detail & Related papers  (2024-08-06T22:39:34Z)
• ZhiJian: A Unifying and Rapidly Deployable Toolbox for Pre-trained Model Reuse [59.500060790983994]
  This paper introduces ZhiJian, a comprehensive and user-friendly toolbox for model reuse, utilizing the PyTorch backend. ZhiJian presents a novel paradigm that unifies diverse perspectives on model reuse, encompassing target architecture construction with PTM, tuning target model with PTM, and PTM-based inference.
  arXiv  Detail & Related papers  (2023-08-17T19:12:13Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Towards Efficient Task-Driven Model Reprogramming with Foundation Models [52.411508216448716]
  Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task.
  arXiv  Detail & Related papers  (2023-04-05T07:28:33Z)
• Assemble Foundation Models for Automatic Code Summarization [9.53949558569201]
  We propose a flexible and robust approach for automatic code summarization based on neural networks. We assemble available foundation models, such as CodeBERT and GPT-2, into a single model named AdaMo. We introduce two adaptive schemes from the perspective of knowledge transfer, namely continuous pretraining and intermediate finetuning.
  arXiv  Detail & Related papers  (2022-01-13T21:38:33Z)
• Understanding the Mechanics of SPIGOT: Surrogate Gradients for Latent Structure Learning [20.506232306308977]
  Latent structure models are a powerful tool for modeling language data. One challenge with end-to-end training of these models is the argmax operation, which has null gradient. We explore latent structure learning through the angle of pulling back the downstream learning objective.
  arXiv  Detail & Related papers  (2020-10-05T21:56:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.