The NordDRG AI Benchmark for Large Language Models

• URL: http://arxiv.org/abs/2506.13790v3
• Date: Wed, 20 Aug 2025 13:47:47 GMT
• Title: The NordDRG AI Benchmark for Large Language Models
• Abstract summary: NordDRG-AI-Benchmark is the first public, rule-complete test bed for DRG reasoning.<n>Grouper software routes a substantial share of multi-trillion-dollar health spending through Diagnosis-Related Groups (DRGs)<n>This is the first public report of an emulating the complete NordDRG grouper logic with governance-grade traceability.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are being piloted for clinical coding and decision support, yet no open benchmark targets the hospital-funding layer where Diagnosis-Related Groups (DRGs) determine reimbursement. In most OECD systems, DRGs route a substantial share of multi-trillion-dollar health spending through governed grouper software, making transparency and auditability first-order concerns. We release NordDRG-AI-Benchmark, the first public, rule-complete test bed for DRG reasoning. The package includes (i) machine-readable approximately 20-sheet NordDRG definition tables and (ii) expert manuals and change-log templates that capture governance workflows. It exposes two suites: a 13-task Logic benchmark (code lookup, cross-table inference, grouping features, multilingual terminology, and CC/MCC validity checks) and a 13-task Grouper benchmark that requires full DRG grouper emulation with strict exact-match scoring on both the DRG and the triggering drg_logic.id. Lightweight reference agents (LogicAgent, GrouperAgent) enable artefact-only evaluation. Under an artefact-only (no web) setting, on the 13 Logic tasks GPT-5 Thinking and Opus 4.1 score 13/13, o3 scores 12/13; mid-tier models (GPT-5 Thinking Mini, o4-mini, GPT-5 Fast) achieve 6-8/13, and remaining models score 5/13 or below. On full grouper emulation across 13 tasks, GPT-5 Thinking solves 7/13, o3 6/13, o4-mini 3/13; GPT-5 Thinking Mini solves 1/13, and all other tested endpoints score 0/13. To our knowledge, this is the first public report of an LLM partially emulating the complete NordDRG grouper logic with governance-grade traceability. Coupling a rule-complete release with exact-match tasks and open scoring provides a reproducible yardstick for head-to-head and longitudinal evaluation in hospital funding. Benchmark materials available in Github.

Related papers

• SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction [12.71956248550879]
  LLM agents can generate reproduction code, yet often produce scientifically unfaithful implementations.<n>We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications.<n>We introduce SemanticAlign-Bench, a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025.
  arXiv  Detail & Related papers  (2026-08-25T08:48:48Z)
• ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents [9.98905103274257]
  We present a benchmark and diagnostic pipeline-level evaluation of computeruse agents on modern web UIs.<n> ComponentBench is organized around a libraryagnostic of 97 canonical UI components instantiated as 2,910.<n>Changing only the observation and action space shifts task success by more than 30% for the same model.
  arXiv  Detail & Related papers  (2026-08-18T20:38:26Z)
• FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search [88.16262636915975]
  FineVerify is a fine-grained self-verification framework for agentic search.<n>It decomposes each question into checkable sub-questions, verifies sampled candidates, and selects the candidate with the highest aggregated score.<n>FineVerify consistently outperforms standard scaling baselines.
  arXiv  Detail & Related papers  (2026-05-30T10:21:20Z)
• Logic-Regularized Verifier Elicits Reasoning from LLMs [63.65875399266337]
  We propose LOVER, an unsupervised verifier regularized by logical rules.<n>Lover treats theverifier as a binary latent variable, utilizinginternal activations and enforcing three logical constraints.<n>Lover significantly outperforms unsupervised baselines.
  arXiv  Detail & Related papers  (2026-05-07T09:03:49Z)
• RAG over Thinking Traces Can Improve Reasoning Tasks [45.57562898423325]
  Retrieval-augmented generation (RAG) has proven effective for knowledge-intensive tasks, but is widely believed to offer limited benefit for reasoning-intensive problems.<n>We propose retrieving thinking traces, i.e., intermediate thinking trajectories generated during problem solving attempts.<n>Using these traces as a corpus, a simple retrieve-then-generate pipeline consistently improves reasoning performance.
  arXiv  Detail & Related papers  (2026-05-05T04:03:28Z)
• OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models [0.0]
  Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding.<n>We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task.
  arXiv  Detail & Related papers  (2026-03-08T20:06:45Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces [34.59674580962045]
  We introduce A-RAG, an Agentic RAG framework that exposes hierarchical retrieval interfaces directly to the model.<n>A-RAG provides three retrieval tools: keyword search, semantic search, and chunk read, enabling the agent to adaptively search and retrieve information across multiple granularities.<n> Experiments on multiple open-domain QA benchmarks show that A-RAG consistently outperforms existing approaches with comparable or lower retrieved tokens.
  arXiv  Detail & Related papers  (2026-02-03T12:07:21Z)
• ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systems [25.131570054560353]
  Current agentic frameworks underperform on long-horizon tasks.<n>We introduce ROMA, a domain-agnostic framework that addresses these limitations.<n>We show that ROMA, combined with GEPA+, delivers leading system-level performance on reasoning and long-form generation benchmarks.
  arXiv  Detail & Related papers  (2026-02-02T09:20:59Z)
• OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding [57.39403818250357]
  We introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding.<n> OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items.<n>Experiments reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation.
  arXiv  Detail & Related papers  (2026-01-15T12:36:08Z)
• Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning [50.27838512822097]
  We introduce GlobalQA, the first benchmark specifically designed to evaluate global RAG capabilities.<n>We propose GlobalRAG, a multi-tool collaborative framework that preserves structural coherence through chunk-level retrieval.<n>On the Qwen2.5-14B model, GlobalRAG achieves 6.63 F1 compared to the strongest baseline's 1.51 F1.
  arXiv  Detail & Related papers  (2025-10-30T07:29:14Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Large Language Models as Medical Codes Selectors: a benchmark using the International Classification of Primary Care [3.969022748080135]
  This study assesses the potential of large language models to assign ICPC-2 codes using a domain-specific search engine.<n>A dataset of 437 Brazilian Portuguese clinical expressions, each annotated with ICPC-2 codes, was used.
  arXiv  Detail & Related papers  (2025-07-19T16:11:10Z)
• Evaluating the Use of LLMs for Documentation to Code Traceability [3.076436880934678]
  Large Language Models can establish trace links between various software documentation and source code.<n>We create two novel datasets from two open-source projects (Unity Catalog and Crawl4AI)<n>Results show that the best-performing LLM achieves F1-scores of 79.4% and 80.4% across the two datasets.
  arXiv  Detail & Related papers  (2025-06-19T16:18:53Z)
• AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP [2.869780207429188]
  Large language models (LLMs) have shown remarkable progress in reasoning abilities.<n>Yet their performance on Arabic data, characterized by rich morphology, diverse dialects, and complex script, remains underexplored.<n>This paper presents a comprehensive benchmarking study of multiple reasoning-focused LLMs.
  arXiv  Detail & Related papers  (2025-06-10T13:10:31Z)
• FlowReasoner: Reinforcing Query-Level Meta-Agents [63.602173107171076]
  This paper proposes a query-level meta-agent named FlowReasoner to automate the design of query-level multi-agent systems.<n>Our core idea is to incentivize a reasoning-based meta-agent via external execution feedback.
  arXiv  Detail & Related papers  (2025-04-21T17:35:42Z)
• GeoBenchX: Benchmarking LLMs for Multistep Geospatial Tasks [0.0]
  Sonnet 3.5 and GPT-4o achieve the best overall performance, with Claude models excelling on solvable tasks.<n>Common errors include misunderstanding geometrical relationships, relying on outdated knowledge, and inefficient data manipulation.
  arXiv  Detail & Related papers  (2025-03-23T16:20:14Z)
• START: Self-taught Reasoner with Tools [51.38785489790888]
  We introduce START (Self-Taught Reasoner with Tools), a tool-integrated long Chain-of-thought (CoT) reasoning LLM.<n> START is capable of performing complex computations, self-checking, exploring diverse methods, and self-ging.<n>It significantly outperforms the base QwQ-32B and achieves performance comparable to the state-of-the-art open-weight model R1-Distill-Qwen-32B.
  arXiv  Detail & Related papers  (2025-03-06T17:11:51Z)
• LettuceDetect: A Hallucination Detection Framework for RAG Applications [2.4830926411920315]
  Retrieval Augmented Generation (RAG) systems remain vulnerable to hallucinated answers despite incorporating external knowledge sources.<n>We present LettuceDetect, a framework that addresses two critical limitations in existing hallucination detection methods.<n>We build on ModernBERT's extended context capabilities (up to 8k tokens) and trained on the RAGTruth benchmark dataset.<n>Our approach outperforms all previous encoder-based models and most prompt-based models, while being approximately 30 times smaller than the best models.
  arXiv  Detail & Related papers  (2025-02-24T13:11:47Z)
• Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation? [90.30635552818875]
  We present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs.<n>This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals.<n>We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets.
  arXiv  Detail & Related papers  (2024-11-06T05:09:34Z)
• Toward General Instruction-Following Alignment for Retrieval-Augmented Generation [63.611024451010316]
  Following natural instructions is crucial for the effective application of Retrieval-Augmented Generation (RAG) systems. We propose VIF-RAG, the first automated, scalable, and verifiable synthetic pipeline for instruction-following alignment in RAG systems.
  arXiv  Detail & Related papers  (2024-10-12T16:30:51Z)
• DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation [48.11754113512047]
  This study includes a code generation benchmark dataset DOMAINEVAL, encompassing six popular domains. Our pipeline works in a fully automated manner, enabling a push-bottom construction from code repositories into formatted subjects under study. The contributions of this study include a code generation benchmark dataset DOMAINEVAL, encompassing six popular domains, a fully automated pipeline for constructing code benchmarks, and an identification of the limitations of LLMs in code generation tasks based on their performance on DOMAINEVAL.
  arXiv  Detail & Related papers  (2024-08-23T16:33:58Z)
• Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design [63.24275274981911]
  Compound AI Systems consisting of many language model inference calls are increasingly employed. In this work, we construct systems, which we call Networks of Networks (NoNs) organized around the distinction between generating a proposed answer and verifying its correctness. We introduce a verifier-based judge NoN with K generators, an instantiation of "best-of-K" or "judge-based" compound AI systems.
  arXiv  Detail & Related papers  (2024-07-23T20:40:37Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data [85.50740598523818]
  MUSTARD is a framework that masters uniform synthesis of theorem and proof data of high quality and diversity. We present a theorem-and-proof benchmark MUSTARDSAUCE with 5,866 valid data points. We perform extensive analysis and demonstrate that MUSTARD generates validated high-quality step-by-step data.
  arXiv  Detail & Related papers  (2024-02-14T05:57:58Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers [60.009969929857704]
  Logical reasoning is an important task for artificial intelligence with potential impacts on science, mathematics, and society. In this work, we reformulating such tasks as modular neurosymbolic programming, which we call LINC. We observe significant performance gains on FOLIO and a balanced subset of ProofWriter for three different models in nearly all experimental conditions we evaluate.
  arXiv  Detail & Related papers  (2023-10-23T17:58:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.