Beyond Words: Evaluating Large Language Models in Transportation Planning

• URL: http://arxiv.org/abs/2409.14516v1
• Date: Sun, 22 Sep 2024 16:20:00 GMT
• Title: Beyond Words: Evaluating Large Language Models in Transportation Planning
• Authors: Shaowei Ying, Zhenlong Li, Manzhu Yu,
• Abstract summary: This study investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, to enhance transportation planning. The findings underscore the transformative potential of GenAI technologies in urban transportation planning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The resurgence and rapid advancement of Generative Artificial Intelligence (GenAI) in 2023 has catalyzed transformative shifts across numerous industry sectors, including urban transportation and logistics. This study investigates the evaluation of Large Language Models (LLMs), specifically GPT-4 and Phi-3-mini, to enhance transportation planning. The study assesses the performance and spatial comprehension of these models through a transportation-informed evaluation framework that includes general geospatial skills, general transportation domain skills, and real-world transportation problem-solving. Utilizing a mixed-methods approach, the research encompasses an evaluation of the LLMs' general Geographic Information System (GIS) skills, general transportation domain knowledge as well as abilities to support human decision-making in the real-world transportation planning scenarios of congestion pricing. Results indicate that GPT-4 demonstrates superior accuracy and reliability across various GIS and transportation-specific tasks compared to Phi-3-mini, highlighting its potential as a robust tool for transportation planners. Nonetheless, Phi-3-mini exhibits competence in specific analytical scenarios, suggesting its utility in resource-constrained environments. The findings underscore the transformative potential of GenAI technologies in urban transportation planning. Future work could explore the application of newer LLMs and the impact of Retrieval-Augmented Generation (RAG) techniques, on a broader set of real-world transportation planning and operations challenges, to deepen the integration of advanced AI models in transportation management practices.

Related papers

• Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviors [17.20186037322538]
  We introduce TransportBench, a benchmark dataset that includes a sample of transportation engineering problems on a wide range of subjects. This dataset is used by human experts to evaluate the capabilities of various commercial and open-sourced large language models (LLMs) Our study marks a thrilling first step toward harnessing artificial general intelligence for complex transportation challenges.
  arXiv  Detail & Related papers  (2024-08-15T17:55:45Z)
• AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation [89.68433168477227]
  Large Language Model (LLM) based agents have garnered significant attention and are becoming increasingly popular. This paper investigates enhancing the planning abilities of LLMs through instruction tuning. To address this limitation, this paper explores the automated synthesis of diverse environments and a gradual range of planning tasks.
  arXiv  Detail & Related papers  (2024-08-01T17:59:46Z)
• Towards Next-Generation Urban Decision Support Systems through AI-Powered Construction of Scientific Ontology using Large Language Models -- A Case in Optimizing Intermodal Freight Transportation [1.6230958216521798]
  This study investigates the potential of leveraging the pre-trained Large Language Models (LLMs) By adopting ChatGPT API as the reasoning core, we outline an integrated workflow that encompasses natural language processing, methontology-based prompt tuning, and transformers. The outcomes of our methodology are knowledge graphs in widely adopted ontology languages (e.g., OWL, RDF, SPARQL)
  arXiv  Detail & Related papers  (2024-05-29T16:40:31Z)
• Agent Planning with World Knowledge Model [88.4897773735576]
  We introduce parametric World Knowledge Model (WKM) to facilitate agent planning. We develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning. Our method can achieve superior performance compared to various strong baselines.
  arXiv  Detail & Related papers  (2024-05-23T06:03:19Z)
• Large Language Models for Mobility in Transportation Systems: A Survey on Forecasting Tasks [8.548422411704218]
  Machine learning and deep learning methods are favored for their flexibility and accuracy. With the advent of large language models (LLMs), many researchers have combined these models with previous techniques or applied LLMs to directly predict future traffic information and human travel behaviors.
  arXiv  Detail & Related papers  (2024-05-03T02:54:43Z)
• TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models [46.862519898969325]
  TransportationGames is an evaluation benchmark for assessing (M)LLMs in the transportation domain. We test the performance of various (M)LLMs in memorizing, understanding, and applying transportation knowledge by the selected tasks.
  arXiv  Detail & Related papers  (2024-01-09T10:20:29Z)
• A Study of Situational Reasoning for Traffic Understanding [63.45021731775964]
  We devise three novel text-based tasks for situational reasoning in the traffic domain. We adopt four knowledge-enhanced methods that have shown generalization capability across language reasoning tasks in prior work. We provide in-depth analyses of model performance on data partitions and examine model predictions categorically.
  arXiv  Detail & Related papers  (2023-06-05T01:01:12Z)
• Simulating the Integration of Urban Air Mobility into Existing Transportation Systems: A Survey [37.20137308650717]
  Urban air mobility (UAM) has the potential to revolutionize transportation in metropolitan areas. We conduct a survey to investigate the current state of research on UAM in metropolitan-scale traffic using simulation techniques.
  arXiv  Detail & Related papers  (2023-01-25T16:43:28Z)
• A Bibliometric Analysis and Review on Reinforcement Learning for Transportation Applications [43.356096302298056]
  Transportation is the backbone of the economy and urban development. Reinforcement Learning (RL) that enables autonomous decision-makers to interact with the complex environment. This paper conducts a bibliometric analysis to identify the development of RL-based methods for transportation applications.
  arXiv  Detail & Related papers  (2022-10-26T07:34:51Z)
• End-to-end Interpretable Neural Motion Planner [78.69295676456085]
  We propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios. We design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America.
  arXiv  Detail & Related papers  (2021-01-17T14:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.