Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding

• URL: http://arxiv.org/abs/2608.24621v2
• Date: Mon, 31 Aug 2026 12:39:56 GMT
• Title: Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding
• Abstract summary: We study the problem in air traffic control, where controller-pilot communication demands near-zero error tolerance.<n>We use consequence-aware evaluation to test whether semantic scores misstate operational reliabil- ity.
• Score: 6.5835891807589775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Can language models be trusted in safety- critical operations? In such settings, strong per- formance on semantic metrics does not guaran- tee operational reliability: a misread altitude, a dropped execution condition, or a confused call- sign may score well under standard F1 yet carry sharply asymmetric operational consequences. We study this problem in air traffic control (ATC), where controller-pilot communication demands near-zero error tolerance, and use consequence-aware evaluation to test whether semantic scores misstate operational reliabil- ity. The framework is instantiated in a con- trolled diagnostic ATC benchmark grounded in aviation standards and feedback from 40 air traffic controllers across three countries. Evaluating 8 models, we uncover a system- atic semantic-safety gap: conventional scores give substantially higher performance estimates than consequence-aware evaluation, even for models that appear reliable under standard met- rics. Risk-aware fine-tuning narrows but does not close this gap, showing that consequence- aware evaluation is a necessary complement to standard NLP metrics before any real safety- critical deployment claim

Related papers

• FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models [45.90905049052029]
  We propose FLY-EVAL++, an evidence-driven evaluation protocol for large language models in safety-critical environments.<n>We instantiate FLY-EVAL++ for Flight Trajectory and Attitude Prediction (FTAP) by extending the PilotBench setting with history-conditioned and multi-step prediction tasks.<n>Our results show that evaluation in safety-critical domains should measure constraint satisfaction and structured validity explicitly rather than rely on accuracy-centric reporting alone.
  arXiv  Detail & Related papers  (2026-09-03T16:00:02Z)
• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observability [79.08785366532287]
  We propose a lightweight risk-gated reinforcement learning approximation for risk-sensitive control under partial observability.<n>We evaluate the approach in two safety-critical partially observable domains: automated glucose regulation and safety-constrained navigation.
  arXiv  Detail & Related papers  (2026-05-14T01:23:09Z)
• Safety-Oriented Evaluation of Language Understanding Systems for Air Traffic Control [7.102251210849638]
  We propose a safety-oriented, consequence-aware evaluation framework tailored to Air Traffic Control (ATC) operations.<n>Our results reveal that while current LLMs achieve reasonable aggregate accuracy, their operational reliability is severely limited.<n>These findings highlight the necessity of consequence-aware evaluation protocols for the responsible deployment of AI-assisted ATC systems.
  arXiv  Detail & Related papers  (2026-05-12T08:39:54Z)
• ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control [7.018221049093963]
  Vision-language-action (VLA) models have emerged as generalist robotic controllers capable of mapping visual observations and natural language instructions to continuous action sequences.<n>We introduce ReconVLA, a reliable conformal model that produces uncertainty-guided and failure-aware control signals.<n>Our results show that conformalized action predictions consistently improve failure anticipation, reduce catastrophic errors, and provide a calibrated measure of confidence without retraining or modifying the underlying VLA.
  arXiv  Detail & Related papers  (2026-04-17T20:20:43Z)
• Towards a Science of AI Agent Reliability [9.570634569436535]
  AI agents are increasingly deployed to execute important tasks.<n>While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice.<n>We propose twelve metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety.
  arXiv  Detail & Related papers  (2026-02-18T18:05:44Z)
• Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know" [47.930782177987446]
  Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations.<n>We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks.<n>Because factual knowledge is stable while hallucinations are agreement, cross-regime provides a precise signal of internal uncertainty.
  arXiv  Detail & Related papers  (2026-02-04T18:39:58Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning [7.0366838632617705]
  We develop a data-driven approach for safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters.<n>We use offline trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots.<n>We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios.
  arXiv  Detail & Related papers  (2025-11-25T19:57:07Z)
• Pointwise Feasibility of Gaussian Process-based Safety-Critical Control under Model Uncertainty [77.18483084440182]
  Control Barrier Functions (CBFs) and Control Lyapunov Functions (CLFs) are popular tools for enforcing safety and stability of a controlled system, respectively. We present a Gaussian Process (GP)-based approach to tackle the problem of model uncertainty in safety-critical controllers that use CBFs and CLFs.
  arXiv  Detail & Related papers  (2021-06-13T23:08:49Z)
• Approaching Neural Network Uncertainty Realism [53.308409014122816]
  Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles. We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test. We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
  arXiv  Detail & Related papers  (2021-01-08T11:56:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.