Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems

• URL: http://arxiv.org/abs/2409.02278v1
• Date: Tue, 3 Sep 2024 20:24:37 GMT
• Title: Evaluation and Comparison of Visual Language Models for Transportation Engineering Problems
• Authors: Sanjita Prajapati, Tanu Singh, Chinmay Hegde, Pranamesh Chakraborty,
• Abstract summary: We have explored state-of-the-art vision language models (VLM) for vision-based transportation engineering tasks. The image classification task involves congestion detection and crack identification, whereas, for object detection, helmet violations were identified. We have applied open-source models such as CLIP, BLIP, OWL-ViT, Llava-Next, and closed-source GPT-4o to evaluate the performance of these VLM models.
• Score: 16.49637074299509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent developments in vision language models (VLM) have shown great potential for diverse applications related to image understanding. In this study, we have explored state-of-the-art VLM models for vision-based transportation engineering tasks such as image classification and object detection. The image classification task involves congestion detection and crack identification, whereas, for object detection, helmet violations were identified. We have applied open-source models such as CLIP, BLIP, OWL-ViT, Llava-Next, and closed-source GPT-4o to evaluate the performance of these state-of-the-art VLM models to harness the capabilities of language understanding for vision-based transportation tasks. These tasks were performed by applying zero-shot prompting to the VLM models, as zero-shot prompting involves performing tasks without any training on those tasks. It eliminates the need for annotated datasets or fine-tuning for specific tasks. Though these models gave comparative results with benchmark Convolutional Neural Networks (CNN) models in the image classification tasks, for object localization tasks, it still needs improvement. Therefore, this study provides a comprehensive evaluation of the state-of-the-art VLM models highlighting the advantages and limitations of the models, which can be taken as the baseline for future improvement and wide-scale implementation.

Related papers

• Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities [30.176918208200604]
  Vision-Language Models (VLMs) have emerged as general purpose tools for addressing a variety of complex computer vision problems. These models have been shown to be highly capable, but also lacking some basic visual understanding skills. This paper sets out to understand the limitations of SoTA VLMs on fundamental visual tasks.
  arXiv  Detail & Related papers  (2024-08-13T08:26:32Z)
• VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs [102.36953558562436]
  Vision language models (VLMs) are an exciting emerging class of language models (LMs) One understudied capability inVLMs is visual spatial planning. Our study introduces a benchmark that evaluates the spatial planning capability in these models in general.
  arXiv  Detail & Related papers  (2024-07-02T00:24:01Z)
• Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM [3.2688425993442696]
  Many probing studies have revealed that even the best-performing Vision and Language Models (VLMs) struggle to capture aspects of compositional scene understanding. Recent VLM advancements include scaling up both model and dataset sizes, additional training objectives and levels of supervision. This paper introduces a novel suite of quantitative metrics that utilize GradCAM activations to rigorously evaluate the grounding capabilities of pre-trained VLMs.
  arXiv  Detail & Related papers  (2024-04-29T22:06:17Z)
• Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
  We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
  arXiv  Detail & Related papers  (2024-02-07T13:41:53Z)
• Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes [1.76602679361245]
  We present two interactive visualizations in the context of Sprite, a system for creating Computer Vision (CV) models. We study how these visualizations help Sprite's users identify (evaluate) and select (plan) images where a model is struggling and can lead to improved performance.
  arXiv  Detail & Related papers  (2023-05-19T14:43:00Z)
• Vision-Language Models for Vision Tasks: A Survey [62.543250338410836]
  Vision-Language Models (VLMs) learn rich vision-language correlation from web-scale image-text pairs. This paper provides a systematic review of visual language models for various visual recognition tasks.
  arXiv  Detail & Related papers  (2023-04-03T02:17:05Z)
• Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
  We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
  arXiv  Detail & Related papers  (2023-03-13T16:00:31Z)
• Vision Checklist: Towards Testable Error Analysis of Image Models to Help System Designers Interrogate Model Capabilities [26.177391265710362]
  Vision Checklist is a framework aimed at interrogating the capabilities of a model in order to produce a report that can be used by a system designer for robustness evaluations. Our framework is evaluated on multiple datasets like Tinyimagenet, CIFAR10, CIFAR100 and Camelyon17 and for models like ViT and Resnet.
  arXiv  Detail & Related papers  (2022-01-27T17:20:16Z)
• VinVL: Revisiting Visual Representations in Vision-Language Models [96.39332942534368]
  We develop an improved object detection model to provide object-centric representations of images. New visual features significantly improve the performance across all vision language (VL) tasks. We will release the new object detection model to public.
  arXiv  Detail & Related papers  (2021-01-02T23:35:27Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.