AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery

• URL: http://arxiv.org/abs/2607.14756v1
• Date: Thu, 16 Jul 2026 09:41:33 GMT
• Title: AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery
• Abstract summary: This research investigates the potential of Vision-Language Models (VLMs) to infer building typologies from Google Street View images.<n>We evaluate several state-of-the-art VLMs, including GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash.<n>We find that AI tends to focus on visual indicators, whereas human experts place greater emphasis on broader contextual cues and domain knowledge.
• Score: 4.997351634293919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This research investigates the potential of Vision-Language Models (VLMs) to infer building typologies: Construction, Current Use, and Storeys from Google Street View (GSV) images. Predictions generated by VLMs are compared with inference by human experts (civil engineers and architects) as a source of manually labelled ground-truth data. We evaluate several state-of-the-art VLMs, including GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash. By applying different scaling strategies and prompting techniques, we found that Chain-of-Thought prompts provide an overall more stable model performance. We also investigate the reasoning behind VLMs' building-typology predictions by examining the probabilities of keywords appearing in AI explanations. This enabled us to analyse patterns in these reasonings and identify key themes driving both agreements and disagreements between VLM and expert labels. We find that AI tends to focus on visual indicators, whereas human experts place greater emphasis on broader contextual cues and domain knowledge, in addition to visual cues. Overall, VLM can approximate experts' capability in building-typology classification at scale, with an average accuracy of approximately 70%. The study demonstrates the VLM's potential for AI automation in tasks that require pattern recognition and object identification in an urban context. AI have the potential to serve as complementary and collaborative tools for urban analysis, leveraging their strengths in understanding visual patterns. This study contributes to the exploration of the efficiency and scalability of AI visual prediction and provides insights into the reasoning processes that could support automation processes in urban analysis and prediction.

Related papers

• Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models [20.954224027029625]
  We present EyeVLM, a systematic evaluation framework for gaze understanding in vision-language models (VLMs)<n>To assess gaze understanding capabilities, we focus on two core tasks. The first, gaze following, predicts the 2D location where a person is looking.<n>The second, social gaze prediction, requires social and relational reasoning over multi-person interactions.
  arXiv  Detail & Related papers  (2026-05-19T13:50:40Z)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models [63.69856480318313]
  AGILE formulates jigsaw solving as an interactive process, enabling the model to progressively engage with the environment.<n>We show that AGILE substantially boosts performance on jigsaw tasks of varying complexity.<n>We also demonstrate strong generalization across 9 general vision tasks, achieving an average improvement of 3.1%.
  arXiv  Detail & Related papers  (2025-10-01T17:58:05Z)
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective [103.44502230776352]
  We present a systematic investigation of Visual Spatial Reasoning (VSR) in Vision-Language Models (VLMs)<n>We categorize spatial intelligence into three levels of capability, ie, basic perception, spatial understanding, spatial planning, and curate SIBench, a spatial intelligence benchmark encompassing nearly 20 open-source datasets across 23 task settings.
  arXiv  Detail & Related papers  (2025-09-23T12:00:14Z)
• Explain Before You Answer: A Survey on Compositional Visual Reasoning [74.27548620675748]
  Compositional visual reasoning has emerged as a key research frontier in multimodal AI.<n>This survey systematically reviews 260+ papers from top venues (CVPR, ICCV, NeurIPS, ICML, ACL, etc.)<n>We then catalog 60+ benchmarks and corresponding metrics that probe compositional visual reasoning along dimensions such as grounding accuracy, chain-of-thought faithfulness, and high-resolution perception.
  arXiv  Detail & Related papers  (2025-08-24T11:01:51Z)
• A Comprehensive Perspective on Explainable AI across the Machine Learning Workflow [1.269939585263915]
  Holistic Explainable Artificial Intelligence (HXAI) is a user-centric framework that embeds explanation into every stage of the data-analysis workflow.<n>HXAI unifies six components (data, analysis set-up, learning process, model output, model quality, communication channel) into a single taxonomy.<n>A 112-item question bank covers these needs; our survey of contemporary tools highlights critical coverage gaps.
  arXiv  Detail & Related papers  (2025-08-15T15:15:25Z)
• AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models [78.08374249341514]
  The rapid development of AI-generated content (AIGC) has led to the misuse of AI-generated images (AIGI) in spreading misinformation.<n>We introduce a large-scale and comprehensive dataset, Holmes-Set, which includes an instruction-tuning dataset with explanations on whether images are AI-generated.<n>Our work introduces an efficient data annotation method called the Multi-Expert Jury, enhancing data generation through structured MLLM explanations and quality control.<n>In addition, we propose Holmes Pipeline, a meticulously designed three-stage training framework comprising visual expert pre-training, supervised fine-tuning, and direct preference optimization
  arXiv  Detail & Related papers  (2025-07-03T14:26:31Z)
• StreetLens: Enabling Human-Centered AI Agents for Neighborhood Assessment from Street View Imagery [8.188916140324139]
  We present StreetLens, a user-configurable human-centered workflow that integrates relevant social science expertise into a vision language model.<n>StreetLens places domain knowledge at the core of the analysis process.<n>It also supports the integration of prior survey data to enhance robustness and expand the range of characteristics assessed.
  arXiv  Detail & Related papers  (2025-06-17T16:06:03Z)
• AI-in-the-loop: The future of biomedical visual analytics applications in the era of AI [3.0942901747200975]
  How will massive developments of AI in data analytics shape future data visualizations and visual analytics?<n>What are opportunities, open challenges and threats in the context of an increasingly powerful AI?<n>We highlight the potential of AI to transform biomedical visualization as a research field.
  arXiv  Detail & Related papers  (2024-12-20T13:27:24Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting [97.52388851329667]
  We introduce Marking Open-world Keypoint Affordances (MOKA) to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.
  arXiv  Detail & Related papers  (2024-03-05T18:08:45Z)
• Towards Fair and Explainable AI using a Human-Centered AI Approach [5.888646114353372]
  We present 5 research projects that aim to enhance explainability and fairness in classification systems and word embeddings. The first project explores the utility/downsides of introducing local model explanations as interfaces for machine teachers. The second project presents D-BIAS, a causality-based human-in-the-loop visual tool for identifying and mitigating social biases in datasets. The third project presents WordBias, a visual interactive tool that helps audit pre-trained static word embeddings for biases against groups. The fourth project presents DramatVis Personae, a visual analytics tool that helps identify social
  arXiv  Detail & Related papers  (2023-06-12T21:08:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.