GPT-4V Explorations: Mining Autonomous Driving

• URL: http://arxiv.org/abs/2406.16817v1
• Date: Mon, 24 Jun 2024 17:26:06 GMT
• Title: GPT-4V Explorations: Mining Autonomous Driving
• Authors: Zixuan Li,
• Abstract summary: GPT-4V introduces capabilities for visual question answering and complex scene comprehension. Our evaluation focuses on its proficiency in scene understanding, reasoning, and driving functions.
• Score: 7.955756422680219
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores the application of the GPT-4V(ision) large visual language model to autonomous driving in mining environments, where traditional systems often falter in understanding intentions and making accurate decisions during emergencies. GPT-4V introduces capabilities for visual question answering and complex scene comprehension, addressing challenges in these specialized settings.Our evaluation focuses on its proficiency in scene understanding, reasoning, and driving functions, with specific tests on its ability to recognize and interpret elements such as pedestrians, various vehicles, and traffic devices. While GPT-4V showed robust comprehension and decision-making skills, it faced difficulties in accurately identifying specific vehicle types and managing dynamic interactions. Despite these challenges, its effective navigation and strategic decision-making demonstrate its potential as a reliable agent for autonomous driving in the complex conditions of mining environments, highlighting its adaptability and operational viability in industrial settings.

Related papers

• Vision-Integrated LLMs for Autonomous Driving Assistance : Human Performance Comparison and Trust Evaluation [2.322929119892535]
  This study introduces a Large Language Model (LLM)-based Autonomous Driving (AD) assistance system. The vision adapter, combining YOLOv4 and Vision Transformer (ViT), extracts comprehensive visual features. The system closely mirrors human performance in describing situations and moderately aligns with human decisions in generating appropriate responses.
  arXiv  Detail & Related papers  (2025-02-06T19:19:28Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs) Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding. We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Driving [65.04643267731122]
  General MLLMs combined with CLIP often struggle to represent driving-specific scenarios accurately. We propose the Hints of Prompt (HoP) framework, which introduces three key enhancements. These hints are fused through a Hint Fusion module, enriching visual representations and enhancing multimodal reasoning.
  arXiv  Detail & Related papers  (2024-11-20T06:58:33Z)
• The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition [136.32656319458158]
  The 2024 RoboDrive Challenge was crafted to propel the development of driving perception technologies. This year's challenge consisted of five distinct tracks and attracted 140 registered teams from 93 institutes across 11 countries. The competition culminated in 15 top-performing solutions.
  arXiv  Detail & Related papers  (2024-05-14T17:59:57Z)
• GPT-4V as Traffic Assistant: An In-depth Look at Vision Language Model on Complex Traffic Events [25.51232964290688]
  The recognition and understanding of traffic incidents, particularly traffic accidents, is a topic of paramount importance in the realm of intelligent transportation systems and vehicles. The advent of large vision-language models (VLMs) such as GPT-4V has introduced innovative approaches to addressing this issue. We observe that GPT-4V demonstrates remarkable cognitive, reasoning, and decision-making ability in certain classic traffic events.
  arXiv  Detail & Related papers  (2024-02-03T16:38:25Z)
• On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving [37.617793990547625]
  This report provides an exhaustive evaluation of the latest state-of-the-art VLM, GPT-4V. We explore the model's abilities to understand and reason about driving scenes, make decisions, and ultimately act in the capacity of a driver. Our findings reveal that GPT-4V demonstrates superior performance in scene understanding and causal reasoning compared to existing autonomous systems.
  arXiv  Detail & Related papers  (2023-11-09T12:58:37Z)
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model [84.29836263441136]
  This study introduces DriveGPT4, a novel interpretable end-to-end autonomous driving system based on multimodal large language models (MLLMs) DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users.
  arXiv  Detail & Related papers  (2023-10-02T17:59:52Z)
• Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics [77.34726150561087]
  This work aims to carry out a study on the current scenario of camera and radar-based perception for ADAS and autonomous vehicles. Concepts and characteristics related to both sensors, as well as to their fusion, are presented. We give an overview of the Deep Learning-based detection and segmentation tasks, and the main datasets, metrics, challenges, and open questions in vehicle perception.
  arXiv  Detail & Related papers  (2023-03-08T00:48:32Z)
• Learning energy-efficient driving behaviors by imitating experts [75.12960180185105]
  This paper examines the role of imitation learning in bridging the gap between control strategies and realistic limitations in communication and sensing. We show that imitation learning can succeed in deriving policies that, if adopted by 5% of vehicles, may boost the energy-efficiency of networks with varying traffic conditions by 15% using only local observations.
  arXiv  Detail & Related papers  (2022-06-28T17:08:31Z)
• Probabilistic End-to-End Vehicle Navigation in Complex Dynamic Environments with Multimodal Sensor Fusion [16.018962965273495]
  All-day and all-weather navigation is a critical capability for autonomous driving. We propose a probabilistic driving model with ultiperception capability utilizing the information from the camera, lidar and radar. The results suggest that our proposed model outperforms baselines and achieves excellent generalization performance in unseen environments.
  arXiv  Detail & Related papers  (2020-05-05T03:48:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.