Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning

• URL: http://arxiv.org/abs/2603.20148v1
• Date: Fri, 20 Mar 2026 17:24:46 GMT
• Title: Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning
• Abstract summary: Large Multimodal Models (LMMs) promise a paradigm shift toward active reasoning.<n>We present textitDefectBench, the first benchmark designed to interrogate LMMs beyond basic semantic recognition.<n>This work provides both a rigorous benchmarking standard and a high-quality open-source database.
• Score: 7.005450139695288
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated building facade inspection is a critical component of urban resilience and smart city maintenance. Traditionally, this field has relied on specialized discriminative models (e.g., YOLO, Mask R-CNN) that excel at pixel-level localization but are constrained to passive perception and worse generization without the visual understandng to interpret structural topology. Large Multimodal Models (LMMs) promise a paradigm shift toward active reasoning, yet their application in such high-stakes engineering domains lacks rigorous evaluation standards. To bridge this gap, we introduce a human-in-the-loop semi-automated annotation framework, leveraging expert-proposal verification to unify 12 fragmented datasets into a standardized, hierarchical ontology. Building on this foundation, we present \textit{DefectBench}, the first multi-dimensional benchmark designed to interrogate LMMs beyond basic semantic recognition. \textit{DefectBench} evaluates 18 state-of-the-art (SOTA) LMMs across three escalating cognitive dimensions: Semantic Perception, Spatial Localization, and Generative Geometry Segmentation. Extensive experiments reveal that while current LMMs demonstrate exceptional topological awareness and semantic understanding (effectively diagnosing "what" and "how"), they exhibit significant deficiencies in metric localization precision ("where"). Crucially, however, we validate the viability of zero-shot generative segmentation, showing that general-purpose foundation models can rival specialized supervised networks without domain-specific training. This work provides both a rigorous benchmarking standard and a high-quality open-source database, establishing a new baseline for the advancement of autonomous AI agents in civil engineering.

Related papers

• The Illusion of Multi-Agent Advantage [63.92466143571383]
  We show that expert-architected Multi-Agent Systems (MAS) consistently outperforms automatically generated architectures in raw performance and cost-efficiency.<n>Existing evaluation frameworks mask critical architectural gaps and inefficiencies by failing to account for the marginal utility of increased computational cost.
  arXiv  Detail & Related papers  (2026-06-11T07:39:24Z)
• Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach [1.029029220360574]
  Large language models (LLMs) show promise, but it remains unclear which architectural design choices drive generation quality.<n>We present a controlled experimental study using domain-specific insurance contracts to investigate these questions.<n>Results show that the multi-agent approach significantly improves structural quality and modestly enhances queryability.
  arXiv  Detail & Related papers  (2026-04-25T00:58:17Z)
• FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios [58.34124792457706]
  The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
  arXiv  Detail & Related papers  (2026-04-08T12:23:27Z)
• Beyond the Academic Monoculture: A Unified Framework and Industrial Perspective for Attributed Graph Clustering [75.50670592447219]
  Attributed Graph Clustering (AGC) is a fundamental unsupervised task that partitions nodes into cohesive groups by jointly modeling structural topology and node attributes.<n>This survey provides a comprehensive, industrially grounded review of AGC from three complementary perspectives.
  arXiv  Detail & Related papers  (2026-03-21T14:15:34Z)
• Topo-R1: Detecting Topological Anomalies via Vision-Language Models [42.86510112477789]
  Topological correctness is crucial for structures such as blood vessels, nerve fibers, and road networks.<n>We develop an automated data-curation pipeline that synthesizes diverse topological anomalies with verifiable annotations.<n>We then introduce Topo-R1, a framework that endows Vision-Language Models with topology-aware perception.
  arXiv  Detail & Related papers  (2026-03-13T15:05:04Z)
• Diagnosing Generalization Failures from Representational Geometry Markers [8.403001493770427]
  We study generalization failures inspired by medical biomarkers.<n>We design and test network markers to probe structure, function links, identify prognostic indicators, and validate predictions in real-world settings.<n>This work demonstrates that representational geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and AI interpretability.
  arXiv  Detail & Related papers  (2026-03-02T13:59:19Z)
• Agentic Adversarial QA for Improving Domain-Specific LLMs [53.00642389531106]
  Large Language Models (LLMs) often struggle to adapt effectively to specialized domains.<n>We propose an adversarial question-generation framework that produces a compact set of semantically challenging questions.
  arXiv  Detail & Related papers  (2026-02-20T10:53:09Z)
• LLM-Driven Ontology Construction for Enterprise Knowledge Graphs [0.0]
  This paper introduces OntoEKG, a pipeline designed to accelerate the generation of domain-specific unstructured from enterprise data.<n>Our approach decomposes the modelling task into two distinct phases: an extraction module that identifies core classes and properties, and an entailment module that logically these elements into a hierarchy before serialising them into standard RDF.<n>Addressing the significant lack of comprehensive benchmarks for end-to-end construction, we adopt a new evaluation dataset derived from documents across the Data, Finance, and Logistics sectors.
  arXiv  Detail & Related papers  (2026-02-01T15:13:30Z)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs [65.04549036809557]
  We introduce a benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors.<n>This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions.<n> Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings.
  arXiv  Detail & Related papers  (2025-12-22T18:58:12Z)
• Beyond Binary Classification: A Semi-supervised Approach to Generalized AI-generated Image Detection [1.189955933770711]
  A critical vulnerability in current forensics is the failure of detectors to achieve cross-generator generalization.<n>We propose a semi-supervised approach that enhances binary classification by discovering latent architectural patterns within the "fake" class.
  arXiv  Detail & Related papers  (2025-11-23T16:02:27Z)
• LTD-Bench: Evaluating Large Language Models by Letting Them Draw [57.237152905238084]
  LTD-Bench is a breakthrough benchmark for large language models (LLMs)<n>It transforms LLM evaluation from abstract scores to directly observable visual outputs by requiring models to generate drawings through dot matrices or executable code.<n> LTD-Bench's visual outputs enable powerful diagnostic analysis, offering a potential approach to investigate model similarity.
  arXiv  Detail & Related papers  (2025-11-04T08:11:23Z)
• RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark [71.3555284685426]
  We introduce RealUnify, a benchmark designed to evaluate bidirectional capability synergy.<n>RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks.<n>We find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient.
  arXiv  Detail & Related papers  (2025-09-29T15:07:28Z)
• LLM-as-classifier: Semi-Supervised, Iterative Framework for Hierarchical Text Classification using Large Language Models [0.0]
  Large Language Models (LLMs) have provided unprecedented capabilities for analyzing unstructured text data.<n>Standard fine-tuning approaches can be resource-intensive and often struggle with the dynamic nature of real-world data distributions.
  arXiv  Detail & Related papers  (2025-08-22T15:47:17Z)
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [93.8285345915925]
  Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making.<n>With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems.<n>We categorize existing methods along two dimensions: (1) Regimes, which define the stage at which reasoning is achieved; and (2) Architectures, which determine the components involved in the reasoning process.
  arXiv  Detail & Related papers  (2025-04-12T01:27:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.