Beyond Words: Evaluating Large Language Models in Transportation Planning

• URL: http://arxiv.org/abs/2409.14516v1
• Date: Sun, 22 Sep 2024 16:20:00 GMT
• Title: Beyond Words: Evaluating Large Language Models in Transportation Planning
• Authors: Shaowei Ying, Zhenlong Li, Manzhu Yu,
• Abstract summary: This study investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, to enhance transportation planning. The findings underscore the transformative potential of GenAI technologies in urban transportation planning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The resurgence and rapid advancement of Generative Artificial Intelligence (GenAI) in 2023 has catalyzed transformative shifts across numerous industry sectors, including urban transportation and logistics. This study investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, to enhance transportation planning. The study assesses the performance and spatial comprehension of these models through a transportation-informed evaluation framework that includes general geospatial skills, general transportation domain skills, and real-world transportation problem-solving. Utilizing a mixed-methods approach, the research encompasses an evaluation of the LLMs' general Geographic Information System (GIS) skills, general transportation domain knowledge as well as abilities to support human decision-making in the real-world transportation planning scenarios of congestion pricing. Results indicate that GPT-4 demonstrates superior accuracy and reliability across various GIS and transportation-specific tasks compared to Phi-3-mini, highlighting its potential as a robust tool for transportation planners. Nonetheless, Phi-3-mini exhibits competence in specific analytical scenarios, suggesting its utility in resource-constrained environments. The findings underscore the transformative potential of GenAI technologies in urban transportation planning. Future work could explore the application of newer LLMs and the impact of Retrieval-Augmented Generation (RAG) techniques, on a broader set of real-world transportation planning and operations challenges, to deepen the integration of advanced AI models in transportation management practices.

Related papers

• A Survey of World Models for Autonomous Driving [63.33363128964687]
  Recent breakthroughs in autonomous driving have been propelled by advances in robust world modeling. This paper systematically reviews recent advances in world models for autonomous driving.
  arXiv  Detail & Related papers  (2025-01-20T04:00:02Z)
• Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions [1.6121249557846946]
  This paper comprehensively reviews the transformative potential of Large Language Models (LLMs) in optimizing ITS. Our analysis reveals how these advanced models can significantly enhance traffic management and safety. This paper aims to guide researchers and practitioners through the complexities and opportunities of integrating LLMs in ITS.
  arXiv  Detail & Related papers  (2025-01-08T11:37:35Z)
• Exploring the Potential of Large Language Models in Public Transportation: San Antonio Case Study [1.7740414468805545]
  This study explores the potential of large language models (LLMs) to revolutionize public transportation management within the context of San Antonio's transit system. We investigate their capabilities to optimize route planning, reduce wait times, and provide personalized travel assistance.
  arXiv  Detail & Related papers  (2025-01-07T16:18:55Z)
• Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviors [17.20186037322538]
  We introduce TransportBench, a benchmark dataset that includes a sample of transportation engineering problems on a wide range of subjects. This dataset is used by human experts to evaluate the capabilities of various commercial and open-sourced large language models (LLMs) Our study marks a thrilling first step toward harnessing artificial general intelligence for complex transportation challenges.
  arXiv  Detail & Related papers  (2024-08-15T17:55:45Z)
• AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation [81.32722475387364]
  Large Language Model-based agents have garnered significant attention and are becoming increasingly popular. Planning ability is a crucial component of an LLM-based agent, which generally entails achieving a desired goal from an initial state. Recent studies have demonstrated that utilizing expert-level trajectory for instruction-tuning LLMs effectively enhances their planning capabilities.
  arXiv  Detail & Related papers  (2024-08-01T17:59:46Z)
• Large Language Models for Mobility in Transportation Systems: A Survey on Forecasting Tasks [8.548422411704218]
  Machine learning and deep learning methods are favored for their flexibility and accuracy. With the advent of large language models (LLMs), many researchers have combined these models with previous techniques or applied LLMs to directly predict future traffic information and human travel behaviors.
  arXiv  Detail & Related papers  (2024-05-03T02:54:43Z)
• TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models [46.862519898969325]
  TransportationGames is an evaluation benchmark for assessing (M)LLMs in the transportation domain. We test the performance of various (M)LLMs in memorizing, understanding, and applying transportation knowledge by the selected tasks.
  arXiv  Detail & Related papers  (2024-01-09T10:20:29Z)
• A Study of Situational Reasoning for Traffic Understanding [63.45021731775964]
  We devise three novel text-based tasks for situational reasoning in the traffic domain. We adopt four knowledge-enhanced methods that have shown generalization capability across language reasoning tasks in prior work. We provide in-depth analyses of model performance on data partitions and examine model predictions categorically.
  arXiv  Detail & Related papers  (2023-06-05T01:01:12Z)
• A Bibliometric Analysis and Review on Reinforcement Learning for Transportation Applications [43.356096302298056]
  Transportation is the backbone of the economy and urban development. Reinforcement Learning (RL) that enables autonomous decision-makers to interact with the complex environment. This paper conducts a bibliometric analysis to identify the development of RL-based methods for transportation applications.
  arXiv  Detail & Related papers  (2022-10-26T07:34:51Z)
• End-to-end Interpretable Neural Motion Planner [78.69295676456085]
  We propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios. We design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America.
  arXiv  Detail & Related papers  (2021-01-17T14:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.