Revolutionizing Urban Safety Perception Assessments: Integrating Multimodal Large Language Models with Street View Images

• URL: http://arxiv.org/abs/2407.19719v2
• Date: Mon, 5 Aug 2024 12:29:47 GMT
• Title: Revolutionizing Urban Safety Perception Assessments: Integrating Multimodal Large Language Models with Street View Images
• Authors: Jiaxin Zhang, Yunqin Li, Tomohiro Fukuda, Bowen Wang,
• Abstract summary: Measuring urban safety perception is an important and complex task that traditionally relies heavily on human resources. Recent advances in multimodal large language models (MLLMs) have demonstrated powerful reasoning and analytical capabilities. We propose a method based on the pre-trained Contrastive Language-Image Pre-training (CLIP) feature and K-Nearest Neighbors (K-NN) retrieval to quickly assess the safety index of the entire city.
• Score: 5.799322786332704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measuring urban safety perception is an important and complex task that traditionally relies heavily on human resources. This process often involves extensive field surveys, manual data collection, and subjective assessments, which can be time-consuming, costly, and sometimes inconsistent. Street View Images (SVIs), along with deep learning methods, provide a way to realize large-scale urban safety detection. However, achieving this goal often requires extensive human annotation to train safety ranking models, and the architectural differences between cities hinder the transferability of these models. Thus, a fully automated method for conducting safety evaluations is essential. Recent advances in multimodal large language models (MLLMs) have demonstrated powerful reasoning and analytical capabilities. Cutting-edge models, e.g., GPT-4 have shown surprising performance in many tasks. We employed these models for urban safety ranking on a human-annotated anchor set and validated that the results from MLLMs align closely with human perceptions. Additionally, we proposed a method based on the pre-trained Contrastive Language-Image Pre-training (CLIP) feature and K-Nearest Neighbors (K-NN) retrieval to quickly assess the safety index of the entire city. Experimental results show that our method outperforms existing training needed deep learning approaches, achieving efficient and accurate urban safety evaluations. The proposed automation for urban safety perception assessment is a valuable tool for city planners, policymakers, and researchers aiming to improve urban environments.

Related papers

• Machine Learning for Public Good: Predicting Urban Crime Patterns to Enhance Community Safety [0.0]
  This paper explores the effectiveness of ML techniques to predict spatial and temporal patterns of crimes in urban areas. Research goal is to achieve a high degree of accuracy in categorizing calls into priority levels.
  arXiv  Detail & Related papers  (2024-09-17T02:07:14Z)
• Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing [8.468153670795443]
  This paper introduces an innovative approach that leverages large multimodal models (LMMs) to interpret complex street crossing scenes. By generating a safety score and scene description in natural language, our method supports safe decision-making for the blind and low-vision individuals.
  arXiv  Detail & Related papers  (2024-02-09T21:37:13Z)
• The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness [56.174255970895466]
  Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark.
  arXiv  Detail & Related papers  (2023-12-30T17:37:06Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• Safety Assessment of Chinese Large Language Models [51.83369778259149]
  Large language models (LLMs) may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes. To promote the deployment of safe, responsible, and ethical AI, we release SafetyPrompts including 100k augmented prompts and responses by LLMs.
  arXiv  Detail & Related papers  (2023-04-20T16:27:35Z)
• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements [76.80453043969209]
  This survey presents a framework for safety research pertaining to large models. We begin by introducing safety issues of wide concern, then delve into safety evaluation methods for large models. We explore the strategies for enhancing large model safety from training to deployment.
  arXiv  Detail & Related papers  (2023-02-18T09:32:55Z)
• Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks [70.76757529955577]
  This paper revisits prior work in this scope from the perspective of state-wise safe RL. We propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. To facilitate further research in this area, we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit.
  arXiv  Detail & Related papers  (2022-12-12T06:30:17Z)
• Evaluating the Perceived Safety of Urban City via Maximum Entropy Deep Inverse Reinforcement Learning [10.605168966435981]
  inverse reinforcement learning (IRL) based framework for predicting urban safety and recovering the corresponding reward function. We presented a scalable state representation method to model the prediction problem as a Markov decision process (MDP) and use reinforcement learning (RL) to solve the problem. We will later open-source the crowdsourcing data collection site and the model proposed in this paper.
  arXiv  Detail & Related papers  (2022-11-19T11:01:08Z)
• Evaluating the Safety of Deep Reinforcement Learning Models using Semi-Formal Verification [81.32981236437395]
  We present a semi-formal verification approach for decision-making tasks based on interval analysis. Our method obtains comparable results over standard benchmarks with respect to formal verifiers. Our approach allows to efficiently evaluate safety properties for decision-making models in practical applications.
  arXiv  Detail & Related papers  (2020-10-19T11:18:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.