NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models

• URL: http://arxiv.org/abs/2606.27047v1
• Date: Thu, 25 Jun 2026 13:52:16 GMT
• Title: NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
• Abstract summary: Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge.<n>We introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge.<n>The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: numeric, and verbal.
• Score: 0.9895431386093138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual understanding. To address the need for systematic evaluation in this domain, we introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge. The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: boolean, numeric, and verbal. NuclearQAv2 is constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora. By leveraging structured prompting for both automated question generation and response evaluation, the proposed framework enables scalable benchmark construction and evaluation. We evaluate a diverse set of LLMs using NuclearQAv2 and observe substantial performance differences across task types. While the models generally perform well on factual questions, quantitative reasoning and conceptual understanding remain considerably more challenging. These results highlight the importance of multi-faceted evaluation frameworks and establish NuclearQAv2 as a scalable benchmark for assessing LLM capabilities in technical domains.

Related papers

• CANDI: Contextual Alignment for Niche Domains Question Answering [5.739840108858911]
  We introduce CANDI-QA (Contextual Alignment for Niche Domains Question Answering), a novel dataset evaluating large language models (LLMs)<n>We evaluate over ten diverse language models, from compact open-source to state-of-the-art proprietary systems.<n>As a robust baseline, we present MTSS-Net, a lightweight neuro-symbolic framework combining neural retrieval with rule-based reasoning.
  arXiv  Detail & Related papers  (2026-05-06T06:40:13Z)
• ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning [118.46980291324148]
  ATLAS is a large-scale, high-difficulty, and cross-disciplinary evaluation suite composed of approximately 800 original problems.<n>Its key features include: High Originality and Contamination Resistance, with all questions newly created or substantially adapted to prevent test data leakage.<n>Preliminary results on leading models demonstrate ATLAS's effectiveness in differentiating their advanced scientific reasoning capabilities.
  arXiv  Detail & Related papers  (2025-11-18T11:13:06Z)
• TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework [37.14734285161928]
  We present an evaluation framework that provides a holistic assessment of model performance in code intelligence tasks.<n>Our evaluation framework addresses key limitations in existing approaches with four main improvements.<n>Based on this evaluation framework, we assess 26 state-of-the-art models and uncover both their strengths and limitations.
  arXiv  Detail & Related papers  (2025-10-20T05:05:00Z)
• AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials [40.06511294882352]
  We introduce the AtomWorld benchmark to evaluate Large Language Models (LLMs) on tasks based in Crystallographic Information Files (CIFs)<n>Our experiments show that these models make frequent errors on structure modification tasks, and even in the basic CIF format understandings.<n>By defining these standardized tasks, AtomWorld lays the ground for advancing LLMs toward robust atomic-scale modeling.
  arXiv  Detail & Related papers  (2025-10-06T11:17:56Z)
• AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field [12.465017512854475]
  Large language models (LLMs) are seeing increasing adoption in the Architecture, Engineering, and Construction (AEC) field.<n>This paper establishes AECBench, a benchmark designed to quantify the strengths and limitations of current LLMs in the AEC domain.<n>The benchmark defines 23 representative tasks within a five-level cognition-oriented evaluation framework.
  arXiv  Detail & Related papers  (2025-09-23T08:09:58Z)
• RefactorCoderQA: Benchmarking LLMs for Multi-Domain Coding Question Solutions in Cloud and Edge Deployment [20.416910591388618]
  We introduce RefactorCoderQA, a benchmark designed to evaluate and enhance the performance of Large Language Models (LLMs) across coding tasks.<n>Our fine-tuned model, RefactorCoder-MoE, achieves state-of-the-art performance, significantly outperforming leading open-source and commercial baselines with an overall accuracy of 76.84%.
  arXiv  Detail & Related papers  (2025-09-12T17:44:22Z)
• Evaluating Large Language Models for Real-World Engineering Tasks [75.97299249823972]
  This paper introduces a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios.<n>Using this dataset, we evaluate four state-of-the-art Large Language Models (LLMs)<n>Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.
  arXiv  Detail & Related papers  (2025-05-12T14:05:23Z)
• Computational Reasoning of Large Language Models [51.629694188014064]
  We introduce textbfTuring Machine Bench, a benchmark to assess the ability of Large Language Models (LLMs) to execute reasoning processes.<n> TMBench incorporates four key features: self-contained and knowledge-agnostic reasoning, a minimalistic multi-step structure, controllable difficulty, and a theoretical foundation based on Turing machine.
  arXiv  Detail & Related papers  (2025-04-29T13:52:47Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges [57.88520765782177]
  Large Language Models (LLMs) have opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
  arXiv  Detail & Related papers  (2024-01-13T15:59:09Z)
• NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domain [0.0]
  NuclearQA is a human-made benchmark of 100 questions to evaluate language models in the nuclear domain. We show how the mix of several types of questions makes our benchmark uniquely capable of evaluating models in the nuclear domain.
  arXiv  Detail & Related papers  (2023-10-17T01:27:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.