LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios

• URL: http://arxiv.org/abs/2609.09790v1
• Date: Wed, 09 Sep 2026 06:42:08 GMT
• Title: LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios
• Abstract summary: LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks.<n>Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning.<n>Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance.
• Score: 6.531830438566794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs' practical deployment in real-world settings. The industrial dataset is publicly available under the CC BY-NC-SA 4.0 license.

Related papers

• Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines [50.471211036005286]
  We introduce a Large-Scale Multi-Modal Industrial Open-Closed benchmark (MMIOC-1M)<n> MMIOC-1M is the first unified largest benchmark supporting both open-vocabulary and closed-set industrial detection.<n>We propose a Refined Text-Visual Prompt Network (RTVPNet) that incorporates three key innovations.
  arXiv  Detail & Related papers  (2026-06-06T03:06:10Z)
• IndustryAssetEQA: A Neurosymbolic Operational Intelligence System for Embodied Question Answering in Industrial Asset Maintenance [2.4078554988694183]
  We present IndustryAssetEQA, a neurosymbolic operational intelligence system that combines episodic telemetry representations with a Failure Mode Effects Analysis Knowledge Graph (FMEA-KG)<n>We evaluate on four datasets covering four industrial asset types, including rotating machinery, turbofan engines, hydraulic systems, and cyber-physical production systems.<n>IndustryAssetEQA improves structural validity by up to 0.51, counterfactual accuracy by up to 0.47, and explanation entailment by 0.64, while reducing severe expert-rated overclaims from 28% to 2%.
  arXiv  Detail & Related papers  (2026-04-25T21:11:41Z)
• FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios [58.34124792457706]
  The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
  arXiv  Detail & Related papers  (2026-04-08T12:23:27Z)
• An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs [20.181876038751156]
  INS-S1 is an insurance-specific Large Language Models family trained via a novel end-to-end alignment paradigm.<n> INS-S1 achieves SOTA performance on domain tasks, significantly outperforming DeepSeek-R1 and Gemini-2.5-Pro.<n>Our results demonstrate that rigorous domain specialization can be achieved without compromising general intelligence.
  arXiv  Detail & Related papers  (2026-03-15T16:13:37Z)
• IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation [56.43007596544299]
  IndustryNav is the first dynamic industrial navigation benchmark for active spatial reasoning.<n>A study of nine state-of-the-art Visual Large Language Models reveals that closed-source models maintain a consistent advantage.
  arXiv  Detail & Related papers  (2025-11-21T16:48:49Z)
• Real-IAD Variety: Pushing Industrial Anomaly Detection Dataset to a Modern Era [110.83702639978469]
  Real-IAD Variety is the largest and most diverse IAD benchmark, comprising 198,960 high-resolution images across 160 distinct object categories.<n>Its diversity is ensured through comprehensive coverage of 28 industries, 24 material types, and 22 color variations.<n>Real-IAD Variety will be made publicly available to facilitate innovation in this critical field.
  arXiv  Detail & Related papers  (2025-11-01T12:58:02Z)
• IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios [46.421243185923814]
  Existing Embodied Question Answering (EQA) benchmarks primarily focus on household environments.<n>We introduce IndustryEQA, the first benchmark dedicated to evaluating embodied agent capabilities within safety-critical warehouse scenarios.<n>The benchmark includes rich annotations covering six categories: equipment safety, human safety, object recognition, attribute recognition, temporal understanding, and spatial understanding.
  arXiv  Detail & Related papers  (2025-05-27T02:36:17Z)
• Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind [16.96145027280737]
  We introduce AgroMind, a benchmark for agricultural remote sensing (RS)<n>AgroMind covers four task dimensions: spatial perception, object understanding, scene understanding, and scene reasoning.<n>We evaluate 20 open-source LMMs and 4 closed-source models on AgroMind.
  arXiv  Detail & Related papers  (2025-05-18T02:45:19Z)
• Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection? [5.979778557940213]
  Traditional industrial anomaly detection models often struggle with flexibility and adaptability.<n>Recent advancements in Multimodal Large Language Models (MLLMs) hold promise for overcoming these limitations.<n>We propose Echo, a novel multi-expert framework designed to enhance MLLM performance for IAD.
  arXiv  Detail & Related papers  (2025-01-27T05:41:10Z)
• MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection [66.05200339481115]
  We present MMAD, the first-ever full-spectrum MLLMs benchmark in industrial anomaly detection.<n>We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset.<n>With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs.
  arXiv  Detail & Related papers  (2024-10-12T09:16:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.