Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing

• URL: http://arxiv.org/abs/2409.18286v1
• Date: Thu, 26 Sep 2024 20:58:11 GMT
• Title: Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing
• Authors: Huthaifa I. Ashqar, Ahmed Jaber, Taqwa I. Alhadidi, Mohammed Elhenawy,
• Abstract summary: This study aims to comprehensively review and empirically evaluate the application of multimodal large language models (MLLMs) and Large Vision Models (VLMs) in object detection for transportation systems.
• Score: 4.79071544824946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study aims to comprehensively review and empirically evaluate the application of multimodal large language models (MLLMs) and Large Vision Models (VLMs) in object detection for transportation systems. In the first fold, we provide a background about the potential benefits of MLLMs in transportation applications and conduct a comprehensive review of current MLLM technologies in previous studies. We highlight their effectiveness and limitations in object detection within various transportation scenarios. The second fold involves providing an overview of the taxonomy of end-to-end object detection in transportation applications and future directions. Building on this, we proposed empirical analysis for testing MLLMs on three real-world transportation problems that include object detection tasks namely, road safety attributes extraction, safety-critical event detection, and visual reasoning of thermal images. Our findings provide a detailed assessment of MLLM performance, uncovering both strengths and areas for improvement. Finally, we discuss practical limitations and challenges of MLLMs in enhancing object detection in transportation, thereby offering a roadmap for future research and development in this critical area.

Related papers

• Surveying the MLLM Landscape: A Meta-Review of Current Surveys [17.372501468675303]
  Multimodal Large Language Models (MLLMs) have become a transformative force in the field of artificial intelligence. This survey aims to provide a systematic review of benchmark tests and evaluation methods for MLLMs.
  arXiv  Detail & Related papers  (2024-09-17T14:35:38Z)
• A Survey on Benchmarks of Multimodal Large Language Models [65.87641718350639]
  This paper presents a comprehensive review of 200 benchmarks and evaluations for Multimodal Large Language Models (MLLMs) We focus on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better.
  arXiv  Detail & Related papers  (2024-08-16T09:52:02Z)
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks [74.52259252807191]
  Multimodal Large Language Models (MLLMs) address the complexities of real-world applications far beyond the capabilities of single-modality systems. This paper systematically sorts out the applications of MLLM in multimodal tasks such as natural language, vision, and audio.
  arXiv  Detail & Related papers  (2024-08-02T15:14:53Z)
• Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events [5.233512464561313]
  Multimodal Large Language Models (MLLMs) offer a novel approach by integrating textual, visual, and audio modalities. Our framework leverages the reasoning power of MLLMs, directing their output through context-specific prompts. Preliminary results demonstrate the framework's potential in zero-shot learning and accurate scenario analysis.
  arXiv  Detail & Related papers  (2024-06-19T23:50:41Z)
• Efficient Multimodal Large Language Models: A Survey [60.7614299984182]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. The extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. This survey provides a comprehensive and systematic review of the current state of efficient MLLMs.
  arXiv  Detail & Related papers  (2024-05-17T12:37:10Z)
• Large Language Models for Mobility in Transportation Systems: A Survey on Forecasting Tasks [8.548422411704218]
  Machine learning and deep learning methods are favored for their flexibility and accuracy. With the advent of large language models (LLMs), many researchers have combined these models with previous techniques or applied LLMs to directly predict future traffic information and human travel behaviors.
  arXiv  Detail & Related papers  (2024-05-03T02:54:43Z)
• Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection [34.135738700682055]
  This paper conducts a comprehensive evaluation of large language models (LLMs) represented by ChatGPT. We find that LLMs exhibit strong zero-shot and few-shot capabilities, but is still at a disadvantage compared to models fine-tuned with full resource.
  arXiv  Detail & Related papers  (2024-02-27T07:02:10Z)
• Exploring Perceptual Limitation of Multimodal Large Language Models [57.567868157293994]
  We quantitatively study the perception of small visual objects in several state-of-the-art MLLMs. We identify four independent factors that can contribute to this limitation. Lower object quality and smaller object size can both independently reduce MLLMs' ability to answer visual questions.
  arXiv  Detail & Related papers  (2024-02-12T03:04:42Z)
• TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models [46.862519898969325]
  TransportationGames is an evaluation benchmark for assessing (M)LLMs in the transportation domain. We test the performance of various (M)LLMs in memorizing, understanding, and applying transportation knowledge by the selected tasks.
  arXiv  Detail & Related papers  (2024-01-09T10:20:29Z)
• Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey [71.10448142010422]
  Multi-object tracking (MOT) aims to associate target objects across video frames in order to obtain entire moving trajectories. Embedding methods play an essential role in object location estimation and temporal identity association in MOT. We first conduct a comprehensive overview with in-depth analysis for embedding methods in MOT from seven different perspectives.
  arXiv  Detail & Related papers  (2022-05-22T06:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.