PilotBench: A Benchmark for General Aviation Agents with Safety Constraints

• URL: http://arxiv.org/abs/2604.08987v1
• Date: Fri, 10 Apr 2026 05:48:38 GMT
• Title: PilotBench: A Benchmark for General Aviation Agents with Safety Constraints
• Abstract summary: PilotBench is a benchmark evaluating Large Language Models (LLMs) on safety-critical flight trajectory and attitude prediction.<n>Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction.<n>We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance.
• Score: 39.268198343472434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As Large Language Models (LLMs) advance toward embodied AI agents operating in physical environments, a fundamental question emerges: can models trained on text corpora reliably reason about complex physics while adhering to safety constraints? We address this through PilotBench, a benchmark evaluating LLMs on safety-critical flight trajectory and attitude prediction. Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction through comparative analysis of LLMs and traditional forecasters. We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance. Comparative evaluation across 41 models uncovers a Precision-Controllability Dichotomy: traditional forecasters achieve superior MAE of 7.01 but lack semantic reasoning capabilities, while LLMs gain controllability with 86--89% instruction-following at the cost of 11--14 MAE precision. Phase-stratified analysis further exposes a Dynamic Complexity Gap-LLM performance degrades sharply in high-workload phases such as Climb and Approach, suggesting brittle implicit physics models. These empirical discoveries motivate hybrid architectures combining LLMs' symbolic reasoning with specialized forecasters' numerical precision. PilotBench provides a rigorous foundation for advancing embodied AI in safety-constrained domains.

Related papers

• FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models [45.90905049052029]
  We propose FLY-EVAL++, an evidence-driven evaluation protocol for large language models in safety-critical environments.<n>We instantiate FLY-EVAL++ for Flight Trajectory and Attitude Prediction (FTAP) by extending the PilotBench setting with history-conditioned and multi-step prediction tasks.<n>Our results show that evaluation in safety-critical domains should measure constraint satisfaction and structured validity explicitly rather than rely on accuracy-centric reporting alone.
  arXiv  Detail & Related papers  (2026-09-03T16:00:02Z)
• Can Large Language Models Explain Flight Safety Events? A Prior-Guided Semantic LLM-based Approach [14.220005377497259]
  We propose FlightLLM, a prior-guided semantic LLM-based approach for interpretable flight safety analysis.<n>We first perform feature engineering to address modal inconsistency, combining statistical descriptors with physically meaningful flight indicators.<n>This representation is processed by a Semantic Discretization module, which converts abstract numerical patterns into qualitative descriptions.
  arXiv  Detail & Related papers  (2026-08-18T17:11:40Z)
• LAPF: LLM-Agent-Based Path Finder Using the UAVScenes Dataset [8.27333944529512]
  This paper proposes an LLM-Agent-Based Path Finder (LAPF) framework for autonomous UAV navigation in town-scale outdoor environments.<n>LAPF extends LLM-assisted navigation by integrating perception, memory, planning, and action modules into a closed-loop cognitive architecture.<n>LAPF achieves mean path lengths of 512.83 m and 506.37 m, compared to the straight-line optimum of 497.33 m.
  arXiv  Detail & Related papers  (2026-08-15T11:28:21Z)
• Safety-Oriented Evaluation of Language Understanding Systems for Air Traffic Control [7.102251210849638]
  We propose a safety-oriented, consequence-aware evaluation framework tailored to Air Traffic Control (ATC) operations.<n>Our results reveal that while current LLMs achieve reasonable aggregate accuracy, their operational reliability is severely limited.<n>These findings highlight the necessity of consequence-aware evaluation protocols for the responsible deployment of AI-assisted ATC systems.
  arXiv  Detail & Related papers  (2026-05-12T08:39:54Z)
• Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization [51.11398557420066]
  We introduce Metis, a framework that reformulates jailbreaking as inference-time policy optimization.<n>Metis achieves the strongest average Attack Success Rate (ASR) among compared methods at 89.2%.<n>By replacing redundant exploration with directed optimization, Metis reduces token costs by an average of 8.2x and up to 11.4x.
  arXiv  Detail & Related papers  (2026-05-11T06:45:00Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• PHMForge: A Scenario-Driven Agentic Benchmark for Industrial Asset Lifecycle Maintenance [2.532228247215641]
  PHMForge is a benchmark specifically designed to evaluate large language model (LLM) agents on Prognostics and Health Management (PHM) tasks.<n>Our benchmark encompasses 75 scenarios spanning 7 industrial asset classes (fan engines, bearings, electric motors, gearboxes, aero-engines) across 5 core task categories.<n>We construct 65 specialized tools across two MCP servers and implement execution-based evaluators with task-commensurate metrics.
  arXiv  Detail & Related papers  (2026-04-02T02:09:27Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features [0.0]
  Recent deep trajectory predictors have achieved strong average accuracy but remain unreliable in complex long-tail driving scenarios.<n>We propose a dynamic multi-expert gating framework that adaptively selects the most reliable trajectory predictor among a physics-informed LSTM, a Transformer, and a fine-tuned GameFormer on a per-sample basis.
  arXiv  Detail & Related papers  (2025-10-31T10:01:01Z)
• Improving Aviation Safety Analysis: Automated HFACS Classification Using Reinforcement Learning with Group Relative Policy Optimization [0.29494468099506904]
  We introduce an automated HFACS classification framework for aviation safety analysis.<n>Our approach incorporates a multi-component reward system tailored for aviation safety analysis.<n>The resulting GRPO-optimized model achieved noticeable performance gains.
  arXiv  Detail & Related papers  (2025-08-28T20:35:03Z)
• Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour [35.19786322586909]
  We propose Agentic eXplanations via Interrogative Simulation (AXIS)<n>AXIS generates human-centred action explanations for multi-agent policies.<n>We evaluate AXIS on autonomous driving across ten scenarios for five LLMs.
  arXiv  Detail & Related papers  (2025-05-23T12:19:18Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.931194824519935]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism.<n>Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance.<n>We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)
• Phase of Flight Classification in Aviation Safety using LSTM, GRU, and BiLSTM: A Case Study with ASN Dataset [0.0]
  The research aims to determine whether the phase of flight can be inferred from narratives of post-accident events using NLP techniques.<n>The classification performance of various deep learning models was evaluated.
  arXiv  Detail & Related papers  (2025-01-14T08:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.