Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment

• URL: http://arxiv.org/abs/2607.20476v1
• Date: Mon, 25 May 2026 05:43:37 GMT
• Title: Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment
• Abstract summary: Five large language models are tested to assess multisensor physical hazard data.<n>All tested models consistently produced no precautionary warning signal across tested scenarios.<n>These findings have direct implications for practitioners deploying the tested models in physical safety monitoring systems.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present an empirical benchmark evaluating how five large language models assess multisensor physical hazard data. Testing 60 scenarios across three categories - multi-sensor joint assessment, response proportionality, and pattern disambiguation - with 1,800 API calls at temperature 0.0, we find that all tested models consistently produced no precautionary warning signal across the tested scenarios where multiple sensors are simultaneously elevated below their individual safety limits, while achieving near-perfect accuracy on single-sensor threshold violations. All five models (ChatGPT-4o, Gemini 2.5 Flash, DeepSeek, Kimi, Llama 3.1 8B) score near zero on Category A multi-sensor scenarios (Q2: 0.000-0.208; Q3: 0.000-0.592) compared to strong performance on single-sensor scenarios (Category B Q1: 0.975-1.000). Structured tabular formatting shows no consistent advantage over plain prose; ChatGPT-4o performs significantly better under prose (p = 0.001). These findings have direct implications for practitioners deploying the tested models in physical safety monitoring systems.

Related papers

• SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models [73.04773649437375]
  Multimodal safety moderation requires distinguishing risks arising from visual content, user intent, and assistant behavior.<n>We introduce SafeAtlas-VL, a dataset of 1.5M training instances that places image-, request-, and response-level judgments on a five-level ordered scale.<n>We also construct SafeAtlas-Bench, a set of 5,000 instances for evaluating five-level predictions and continuous risk scores.
  arXiv  Detail & Related papers  (2026-08-29T07:08:01Z)
• TestifAI: Tomography-Based Testing for Deep Learning Systems [42.86570387250456]
  TestifAI is a framework for efficient and accurate estimation of robustness against combinations of perturbations.<n>It reconstructs model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations.<n>It can predict higher-order (3 and 4 perturbations) test outcomes from low-order (1 and 2 perturbations) observations with an aggregate robustness estimation error of less than 7%.
  arXiv  Detail & Related papers  (2026-08-19T13:25:50Z)
• Vision-Language Models for Analog Gauge Reading: An Empirical Study of Specialization, Transfer and Reliability [3.112410411428948]
  This study is a systematic evaluation of specialization, transfer, robustness and reliability for a general-purpose vision-language model (VLM)<n>The Qwen2.5-VL-7B-Instruct model is evaluated using zero-shot prompting, in-context learning (ICL) and parameter-efficient fine-tuning with Quantized Low-Rank Adaptation (QLoRA)<n>The best fine-tuned MPE values are 2.39% on the synthetic dataset, with a 95% confidence interval (CI) of 1.43-3.90%; 2.61% on the Pressure Gauge dataset, with a CI of 1.66-3.80%; and 4.4
  arXiv  Detail & Related papers  (2026-08-18T12:47:14Z)
• Evaluation Awareness Is Not One Capability: Evidence from Open Language Models [1.758143872501506]
  Safety benchmarks assume that test-condition behavior predicts deployment behavior, an assumption that fails if models detect evaluation cues and adapt.<n>This opens a gap between benchmark performance and deployment behavior.<n>We characterize this evaluation awareness through eight experiments across 37 open-weight models and seven families.
  arXiv  Detail & Related papers  (2026-06-22T16:48:43Z)
• REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection [0.2529563359433233]
  We present REDACT, a PII benchmark with 13,427 records, 324,078 entity annotations, 51 entity types, 4,127 surface-form patterns, and 25 languages across 9 scripts.<n>A strength-2 covering-array sampler controls nine generation axes: domain, format, difficulty, length, density, code-switching, language, adjacency, and co-occurrence.<n>From the full benchmark, we evaluate five detectors (Presidio, GLiNER, OpenAI Privacy Filter, GPT-4.1, and Claude Sonnet 4.6) on a locked, language-stratified sample of 1,000 records
  arXiv  Detail & Related papers  (2026-06-18T07:38:15Z)
• Sparse Autoencoder Decomposition of Clinical Sequence Model Representations: Feature Complexity, Task Specialisation, and Mortality Prediction [1.7865154997539017]
  Sparse autoencoders (SAEs) have been applied to large language models and protein language models, but not systematically to electronic health record (EHR) foundation models.<n>We train TopK SAEs on FlatASCEND, a 14.5-million- parameter autoregressive clinical sequence model, at all 10 residual stream extraction points on INSPECT (outpatient) and MIMIC-IV (ICU)<n>SAE decomposition reveals progressive abstraction across transformer depth: layer-0 features are near-perfect token detectors (45.7% singleton), while layer-6 features span approximately 30 token types across multiple clinical categories (0.5% singleton)
  arXiv  Detail & Related papers  (2026-04-13T12:08:32Z)
• Brittlebench: Quantifying LLM robustness via prompt sensitivity [44.950999933205985]
  We introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness.<n>We design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models.<n>We find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model.
  arXiv  Detail & Related papers  (2026-02-27T21:12:13Z)
• When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models [54.08784776767683]
  We present Physical-Conditioned World Model Attack (PhysCond-WMA), the first white-box world model attack that perturbs physical-condition channels.<n>PhysCond-WMA induces semantic, logic, or decision-level distortion while preserving perceptual fidelity.
  arXiv  Detail & Related papers  (2026-02-21T07:22:37Z)
• How well are open sourced AI-generated image detection models out-of-the-box: A comprehensive benchmark study [5.740397289924559]
  No universal winner exists, with detector rankings exhibiting substantial instability.<n>Our findings challenge the one-size-fits-all'' detector paradigm.
  arXiv  Detail & Related papers  (2026-02-08T04:36:13Z)
• A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5 [101.4233736714284]
  Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have driven major gains in reasoning, perception, and generation across language and vision.<n>We present an integrated safety evaluation of six frontier models--GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5--assessing each across language, vision-language, and image generation.
  arXiv  Detail & Related papers  (2026-01-15T15:52:52Z)
• Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection [1.2802720336459552]
  We introduce a benchmark for Multi-Vulnerability Detection across four major languages: C, C++, Python, and JavaScript.<n>We construct a dataset of 40,000 files by injecting controlled counts of vulnerabilities into long-context code samples.<n>Our results reveal a sharp degradation in performance as vulnerability density increases.
  arXiv  Detail & Related papers  (2025-12-26T05:43:35Z)
• CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes [56.52618054240197]
  We propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes.<n>Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token.<n>Our model significantly improves robustness and accuracy, especially in adverse-condition scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:56:20Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• Neural Network Virtual Sensors for Fuel Injection Quantities with Provable Performance Specifications [71.1911136637719]
  We show how provable guarantees can be naturally applied to other real world settings. We show how specific intervals of fuel injection quantities can be targeted to maximize robustness for certain ranges.
  arXiv  Detail & Related papers  (2020-06-30T23:33:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.