AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models

• URL: http://arxiv.org/abs/2603.07989v1
• Date: Mon, 09 Mar 2026 05:53:06 GMT
• Title: AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models
• Abstract summary: AutoTraces is an autoregressive vision-trajectory model for robot trajectory forecasting in humam-language environments.<n>Key innovation lies in a trajectory tokenization scheme, which represents waypoints with point tokens as categorical markers while encoding waypoint numerical values as corresponding point embeddings.
• Score: 5.085937845977223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present AutoTraces, an autoregressive vision-language-trajectory model for robot trajectory forecasting in humam-populated environments, which harnesses the inherent reasoning capabilities of large language models (LLMs) to model complex human behaviors. In contrast to prior works that rely solely on textual representations, our key innovation lies in a novel trajectory tokenization scheme, which represents waypoints with point tokens as categorical and positional markers while encoding waypoint numerical values as corresponding point embeddings, seamlessly integrated into the LLM's space through a lightweight encoder-decoder architecture. This design preserves the LLM's native autoregressive generation mechanism while extending it to physical coordinate spaces, facilitates modeling of long-term interactions in trajectory data. We further introduce an automated chain-of-thought (CoT) generation mechanism that leverages a multimodal LLM to infer spatio-temporal relationships from visual observations and trajectory data, eliminating reliance on manual annotation. Through a two-stage training strategy, our AutoTraces achieves SOTA forecasting accuracy, particularly in long-horizon prediction, while exhibiting strong cross-scene generalization and supporting flexible-length forecasting.

Related papers

• U-STS-LLM A Unified Spatio-Temporal Steered Large Language Model for Traffic Prediction and Imputation [6.194596351746244]
  We present U-STS-LLM, a unified framework built on a-temporal-ly steered LLM.<n>Our core innovation is a Dynamic S-Lo Concurrent Attention Bias Generator that synthesizes a persistent functional graph with transient nodal states.<n>Experiments on real-world cellular datasets demonstrate that U-STS-LLM establishes new state-of-the-art performance in both long-horizon forecasting and high-missing-rate imputation.
  arXiv  Detail & Related papers  (2026-05-12T08:16:22Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• Wireless Traffic Prediction with Large Language Model [54.07581399989292]
  TIDES is a novel framework that captures spatial-temporal correlations for wireless traffic prediction.<n> TIDES achieves efficient adaptation to domain-specific patterns without incurring excessive training overhead.<n>Our results indicate that integrating spatial awareness into LLM-based predictors is the key to unlocking scalable and intelligent network management in future 6G systems.
  arXiv  Detail & Related papers  (2025-12-19T04:47:40Z)
• FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling [5.609178055761294]
  We present FSDAM, a framework that achieves joint attention prediction and caption generation with 100 annotated examples.<n> FSDAM achieves competitive performance on attention prediction, generates coherent, and context-aware explanations.<n>This work shows that effective attention-conditioned generation is achievable with limited supervision, opening new possibilities for practical deployment of explainable driver attention systems.
  arXiv  Detail & Related papers  (2025-11-16T17:50:30Z)
• Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method [54.461213497603154]
  Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities.<n>Nuplan-Occ is the largest occupancy dataset to date, constructed from the widely used Nuplan benchmark.<n>We develop a unified framework that jointly synthesizes high-quality occupancy, multi-view videos, and LiDAR point clouds.
  arXiv  Detail & Related papers  (2025-10-27T03:52:45Z)
• Vision-LLMs for Spatiotemporal Traffic Forecasting [14.700408329373998]
  Large Language Models (LLMs) inherently struggle to model the complex spatial dependencies of grid-based traffic data.<n>We propose ST-Vision-LLM, a novel framework reframe thatstemporal forecasting as a vision-language fusion problem.<n>We show that ST-Vision-LLM outperforms existing methods by 15.6% in long-term prediction accuracy and exceeds the second-best baseline by over 30.04% in cross-domain scenarios.
  arXiv  Detail & Related papers  (2025-10-13T11:15:56Z)
• Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction [20.1863553357121]
  Current deep learning architectures for remote sensing are fundamentally rigid.<n>We introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling.<n> STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands.<n>It unifies various dense prediction tasks and diverse semantic class predictions.
  arXiv  Detail & Related papers  (2025-05-18T07:39:17Z)
• Spatial-Temporal Large Language Model for Traffic Prediction [21.69991612610926]
  We propose a Spatial-Temporal Large Language Model (ST-LLM) for traffic prediction. In the ST-LLM, we define timesteps at each location as tokens and design a spatial-temporal embedding to learn the spatial location and global temporal patterns of these tokens. In experiments on real traffic datasets, ST-LLM is a powerful spatial-temporal learner that outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2024-01-18T17:03:59Z)
• Trajeglish: Traffic Modeling as Next-Token Prediction [67.28197954427638]
  A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs. We apply tools from discrete sequence modeling to model how vehicles, pedestrians and cyclists interact in driving scenarios. Our model tops the Sim Agents Benchmark, surpassing prior work along the realism meta metric by 3.3% and along the interaction metric by 9.9%.
  arXiv  Detail & Related papers  (2023-12-07T18:53:27Z)
• JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds [79.00975648564483]
  Trajectory forecasting models, employed in fields such as robotics, autonomous vehicles, and navigation, face challenges in real-world scenarios. This dataset provides comprehensive data, including the locations of all agents, scene images, and point clouds, all from the robot's perspective. The objective is to predict the future positions of agents relative to the robot using raw sensory input data.
  arXiv  Detail & Related papers  (2023-11-05T18:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.