GraphRareBench: An Auditable Graph-Evidence Benchmark for Phenotype-Driven Rare-Disease Diagnosis

• URL: http://arxiv.org/abs/2607.24878v1
• Date: Mon, 27 Jul 2026 07:58:29 GMT
• Title: GraphRareBench: An Auditable Graph-Evidence Benchmark for Phenotype-Driven Rare-Disease Diagnosis
• Abstract summary: GraphRareBench is a provenance-preserving benchmark containing 2,365-derived cases and 18,093 target-confounder pairs.<n>Agents instantiated with Agents-A1 and DeepSeek-V4-Flash achieved MRRs of 0.746 and 0.718, respectively.<n>Results indicate that full-pool retrieval, hard-confounder discrimination, and observable evidence access capture complementary aspects of model behavior.
• Score: 4.608113014528089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Phenotype-driven diagnostic benchmarks usually report the rank of the reference disease, but they rarely reveal which plausible alternatives are ranked above it or what evidence a tool-using model examines before making its decision. We introduce GraphRareBench, a provenance-preserving benchmark containing 2,365 ontology-derived cases and 18,093 target-confounder pairs. Each case includes a coarsened HPO query, a fixed candidate pool, graph-defined hard confounders, and source-linked evidence records. On the 237-case gene-component-disjoint test split, supervised rankers using a shared 21-feature interface achieved MRRs ranging from 0.640 to 0.740 and case-averaged target-over-confounder accuracies ranging from 0.898 to 0.916. Agents instantiated with Agents-A1 and DeepSeek-V4-Flash achieved MRRs of 0.746 and 0.718, respectively. Their paired MRR difference was not statistically significant, whereas their target-evidence coverage differed by 0.561. Together with the observation that 22.1% to 43.7% of selected Hit@10 successes still ranked at least one graph-defined hard confounder above the target, these results indicate that full-pool retrieval, hard-confounder discrimination, and observable evidence access capture complementary aspects of model behavior. GraphRareBench therefore provides a foundation for more transparent and evidence-aware evaluation of phenotype-driven diagnostic systems. Code and data are available at https://github.com/GUI0609/GraphRareBench.

Related papers

• Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection [66.44527094471619]
  We present RECAL, an unsupervised framework using relation-balanced masked graph learning to better capture rare interaction patterns.<n>On three DARPA E3 datasets, RECAL achieves F1 scores of 99.99%, 99.93%, and 99.99%, outperforming the best baseline on each dataset by 0.88, 0.82, and 0.42 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-09-15T00:31:01Z)
• CareGraph: An Auditable Hybrid AI Framework for Evidence-Grounded Personalized Longitudinal Health Intelligence [0.0]
  CareGraph organizes evidence without diagnosing, predicting outcomes, selecting treatment, or making autonomous clinical decisions.<n>Tests used synthetic cohorts of 400 patients each for development, validation, and holdout.<n>CareGraph offers a safety bounded foundation for intelligent personalized health systems.
  arXiv  Detail & Related papers  (2026-08-25T20:03:52Z)
• TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis? [1.7077233340294056]
  We introduce TelemetrySuffBench, a benchmark that separates failure detection, fault-origin localization, and safe abstention under insufficient evidence.<n>We evaluate five frontier language models using unified protocols, explicit candidate sets, invalid-output accounting, subgroup analyses, and a frozen blind holdout.
  arXiv  Detail & Related papers  (2026-08-08T03:54:22Z)
• FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses [0.0]
  FailureScope is a behavioral-diagnosis method that clusters evaluation probes by their cross-model pass/fail patterns.<n>We show it yields stable, interpretable failure across three regimes usually studied separately: single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks.
  arXiv  Detail & Related papers  (2026-06-03T01:28:00Z)
• When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors [66.18091962164219]
  Existing metrics fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences.<n>We propose two complementary metrics to isolate non-mandatory behavioral patterns: textbfResponse Pattern Similarity (RPS) for verbal alignment and textbfAction Graph Similarity (AGS) for tool-use habits modeled as directed graphs.
  arXiv  Detail & Related papers  (2026-04-23T03:48:56Z)
• Medical thinking with multiple images [44.04557445622649]
  We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images.<n>The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case.<n>On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy.
  arXiv  Detail & Related papers  (2026-04-14T18:51:07Z)
• ORACAL: A Robust and Explainable Multimodal Framework for Smart Contract Vulnerability Detection with Causal Graph Enrichment [0.754973258028456]
  We propose ORACAL, a graph learning framework that integrates Control Flow Graph, Data Flow Graph, and Call Graph.<n>ORACAL selectively enriches critical subgraphs with expert-level security context from Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs)<n>For transparency, the framework adopts PGExplainer to generate subgraph-level explanations identifying vulnerability triggering paths.
  arXiv  Detail & Related papers  (2026-03-30T07:46:59Z)
• TRAJEVAL: Decomposing Code Agent Trajectories for Fine-Grained Diagnosis [23.834704102474927]
  Code agents can resolve GitHub issues, yet when they fail, current evaluation provides no visibility into where or why.<n>We introduce TRAJEVAL, a diagnostic framework that decomposes agent trajectories into three interpretable stages.<n>We validate that these diagnostics are predictive, achieving model-level Pass@1 prediction within 0.87-2.1% MAE.
  arXiv  Detail & Related papers  (2026-03-25T05:27:03Z)
• How well are open sourced AI-generated image detection models out-of-the-box: A comprehensive benchmark study [5.740397289924559]
  No universal winner exists, with detector rankings exhibiting substantial instability.<n>Our findings challenge the one-size-fits-all'' detector paradigm.
  arXiv  Detail & Related papers  (2026-02-08T04:36:13Z)
• MeCaMIL: Causality-Aware Multiple Instance Learning for Fair and Interpretable Whole Slide Image Diagnosis [40.3028468133626]
  Multiple instance learning (MIL) has emerged as the dominant paradigm for whole slide image (WSI) analysis in computational pathology.<n>textbfMeCaMIL, a causality-aware MIL framework, explicitly models demographic confounders through structured causal graphs.<n>MeCaMIL achieves superior fairness -- demographic disparity variance drops by over 65% relative reduction on average across attributes.
  arXiv  Detail & Related papers  (2025-11-14T06:47:21Z)
• Diagnosing Human-object Interaction Detectors [42.283857276076596]
  We introduce a diagnosis toolbox to provide detailed quantitative break-down analysis of HOI detection models. We analyze eight state-of-the-art HOI detection models and provide valuable diagnosis insights to foster future research.
  arXiv  Detail & Related papers  (2023-08-16T17:39:15Z)
• Homophily Outlier Detection in Non-IID Categorical Data [43.51919113927003]
  This work introduces a novel outlier detection framework and its two instances to identify outliers in categorical data. It first defines and incorporates distribution-sensitive outlier factors and their interdependence into a value-value graph-based representation. The learned value outlierness allows for either direct outlier detection or outlying feature selection.
  arXiv  Detail & Related papers  (2021-03-21T23:29:33Z)
• Chest x-ray automated triage: a semiologic approach designed for clinical implementation, exploiting different types of labels through a combination of four Deep Learning architectures [83.48996461770017]
  This work presents a Deep Learning method based on the late fusion of different convolutional architectures. We built four training datasets combining images from public chest x-ray datasets and our institutional archive. We trained four different Deep Learning architectures and combined their outputs with a late fusion strategy, obtaining a unified tool.
  arXiv  Detail & Related papers  (2020-12-23T14:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.