TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering

• URL: http://arxiv.org/abs/2604.17966v1
• Date: Mon, 20 Apr 2026 08:46:49 GMT
• Title: TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering
• Abstract summary: In hypersonic thermal protection system (TPS) design, inaccurate stagnation-point heat flux or boundary-layer calculations may cause catastrophic design margin violations.<n>Current scientific benchmarks only test abstract math and basic physics, evaluate final answers solely, ignore engineering reasoning processes, and cannot detect such critical failures.<n>We propose TPS-CalcBench, the first diagnostic benchmark for closed-form analytical calculations in hypersonic aerodynamics and high-temperature gas dynamics.
• Score: 0.8921166277011348
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deploying LLMs as reasoning assistants in safety-critical aerospace engineering requires stricter evaluation criteria than general scientific benchmarks. In hypersonic thermal protection system (TPS) design, inaccurate stagnation-point heat flux or boundary-layer calculations may cause catastrophic design margin violations. Models with numerically reasonable but physically invalid answers are more dangerous than those declining to respond. Current scientific benchmarks only test abstract math and basic physics, evaluate final answers solely, ignore engineering reasoning processes, and cannot detect such critical failures. We propose TPS-CalcBench, the first diagnostic benchmark for closed-form analytical calculations in hypersonic aerodynamics and high-temperature gas dynamics that experienced TPS engineers conduct without simulations. Our contributions include domain-oriented task taxonomy with 4 difficulty levels and 8 categories from Anderson's textbook, dual-track evaluation measuring result accuracy and reasoning quality via an 8-dimension rubric and calibrated judge with human audit to identify right answer wrong reasoning issues, human-AI data pipeline producing 420 high-confidence core items and 810 noise-controlled pre-gating items from 4560 raw data, noise-sensitivity analysis measuring data quality impacts on model ranking, and three diagnostic intervention methods: DFA-TPS fine-tuning, RAG-EQ retrieval grounding and PA-CoT process-aware prompting. Tests on 13 models from 7 groups show wide performance differences (KPI 12.6-87.9), hidden formula selection defects, data-driven rank changes and effective intervention improvements, establishing a complete diagnose-evaluate-intervene framework for safety-critical engineering LLM deployment assessment.

Related papers

• How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks [60.853503196898906]
  Low reported scores on leading physics benchmarks suggest frontier language models struggle with advanced physics.<n>We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts.
  arXiv  Detail & Related papers  (2026-09-11T16:06:50Z)
• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation [11.290047096794082]
  We introduce TRACE, a testing framework that models temporal reasoning as constraint satisfaction problems via Allen's Interval Algebra.<n>TRACE provides a robust, automated platform for benchmarking the true temporal reasoning capabilities of LRMs.
  arXiv  Detail & Related papers  (2026-07-06T08:14:53Z)
• MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning [7.000170880015254]
  We propose MedPRMBench, the first process-level reward model benchmark for the medical domain.<n>It is built through a three-phase pipeline based on Clinical Reasoning Blueprints (CRBs)<n>The benchmark comprises 6,500 questions with 13,000 reasoning chains and 113,910 step-level labels, plus 6,879 questions for training.
  arXiv  Detail & Related papers  (2026-04-19T06:44:07Z)
• PHMForge: A Scenario-Driven Agentic Benchmark for Industrial Asset Lifecycle Maintenance [2.532228247215641]
  PHMForge is a benchmark specifically designed to evaluate large language model (LLM) agents on Prognostics and Health Management (PHM) tasks.<n>Our benchmark encompasses 75 scenarios spanning 7 industrial asset classes (fan engines, bearings, electric motors, gearboxes, aero-engines) across 5 core task categories.<n>We construct 65 specialized tools across two MCP servers and implement execution-based evaluators with task-commensurate metrics.
  arXiv  Detail & Related papers  (2026-04-02T02:09:27Z)
• Benchmarking Scientific Machine Learning Models for Air Quality Data [0.0]
  This research study presents an explainable and comprehensive benchmark that enables a guideline and proposed physics-guided best model.<n>Using publicly available U.S. Environmental Protection Agency (EPA) daily observations of air quality data from 2022 to 2024, we curate city-level time series for PM2.5 and O3.<n>For benchmarking the best model, linear regression (LR), SARIMAX, multilayer perceptrons (MLP), and LSTM networks are evaluated.
  arXiv  Detail & Related papers  (2026-03-22T03:31:24Z)
• PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering [71.15346406323827]
  We introduce PRIME, a benchmark for evaluating verifiers on Process-Outcome Alignment verification.<n>We find that current verifiers frequently fail to detect derivation flaws.<n>We propose a process-aware RLVR training paradigm utilizing verifiers selected via PRIME.
  arXiv  Detail & Related papers  (2026-02-12T04:45:01Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning [57.868248683256574]
  PRISM-Physics is a process-level evaluation framework and benchmark for complex physics reasoning problems.<n> Solutions are represented as directed acyclic graphs (DAGs) of formulas.<n>Results show that our evaluation framework is aligned with human experts' scoring.
  arXiv  Detail & Related papers  (2025-10-03T17:09:03Z)
• AI-Powered Dynamic Fault Detection and Performance Assessment in Photovoltaic Systems [44.99833362998488]
  intermittent nature of photovoltaic (PV) solar energy leads to power losses of 10-70% and an average energy production decrease of 25%. Current fault detection strategies are costly and often yield unreliable results due to complex data signal profiles. This research presents a computational model using the PVlib library in Python, incorporating a dynamic loss quantification algorithm.
  arXiv  Detail & Related papers  (2024-08-19T23:52:06Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Practical Recommendations for the Design of Automatic Fault Detection Algorithms Based on Experiments with Field Monitoring Data [0.0]
  Automatic fault detection (AFD) is a key technology to optimize the Operation and Maintenance of photovoltaic (PV) systems portfolios. In this study, a series of AFD algorithms have been tested under real operating conditions, using monitoring data collected over 58 months on 80 rooftop-type PV systems installed in Germany. The results shown that this type of AFD algorithm have the potential to detect up to 82.8% of the energy losses with specificity above 90%.
  arXiv  Detail & Related papers  (2022-03-02T13:43:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.