A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness

• URL: http://arxiv.org/abs/2604.10853v1
• Date: Sun, 12 Apr 2026 23:18:47 GMT
• Title: A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness
• Abstract summary: This paper focuses on gap and overlap analysis for policy-like documents.<n>The resulting gap/overlap determinations are driven by genuine differences in coverage and restrictions rather than missing data.<n>We present an executable and auditable benchmark that aligns natural-language contract text with a formal ontology and evidence-linked ground truth.
• Score: 1.029029220360574
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Task-oriented evaluation of knowledge graph (KG) quality increasingly asks whether an ontology-based representation can answer the competency questions that users actually care about, in a manner that is reproducible, explainable, and traceable to evidence. This paper adopts that perspective and focuses on gap and overlap analysis for policy-like documents (e.g., insurance contracts), where given a scenario, which documents support it (overlap) and which do not (gap), with defensible justifications. The resulting gap/overlap determinations are typically driven by genuine differences in coverage and restrictions rather than missing data, making the task a direct test of KG task readiness rather than a test of missing facts or query expressiveness. We present an executable and auditable benchmark that aligns natural-language contract text with a formal ontology and evidence-linked ground truth, enabling systematic comparison of methods. The benchmark includes: (i) ten simplified yet diverse life-insurance contracts reviewed by a domain expert, (ii) a domain ontology (TBox) with an instantiated knowledge base (ABox) populated from contract facts, and (iii) 58 structured scenarios paired with SPARQL queries with contract-level outcomes and clause-level excerpts that justify each label. Using this resource, we compare a text-only LLM baseline that infers outcomes directly from contract text against an ontology-driven pipeline that answers the same scenarios over the instantiated KG, demonstrating that explicit modeling improves consistency and diagnosis for gap/overlap analyses. Although demonstrated for gap and overlap analysis, the benchmark is intended as a reusable template for evaluating KG quality and supporting downstream work such as ontology learning, KG population, and evidence-grounded question answering.

Related papers

• Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoning [72.86819994769634]
  Whole-slide image (WSI) reasoning requires an agent to sequentially acquire visual evidence before answering a diagnostic question.<n>We propose BEACON, a plug-and-play agentic framework that reformulates WSI reasoning as a Bayesian evidence acquisition problem.
  arXiv  Detail & Related papers  (2026-08-06T08:40:57Z)
• KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text [6.373904244962351]
  We propose a novel, interpretable metric for intrinsic Knowledge Graph quality assessment.<n>It measures how closely an automatically extracted graph approximates an "ideal" graph.<n>We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets.
  arXiv  Detail & Related papers  (2026-07-11T08:45:15Z)
• E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems [4.305347927600694]
  We propose E-MIA, which converts verifiable hard evidence in the target document into an exam with four objectively gradable question types.<n>E-MIA improves member/non-member separability in stringent settings while preserving natural, stealthy queries.
  arXiv  Detail & Related papers  (2026-05-01T12:30:00Z)
• ContextClaim: A Context-Driven Paradigm for Verifiable Claim Detection [17.165016272480262]
  Verifiable claim detection asks whether a claim expresses a factual statement that can, in principle, be assessed against external evidence.<n>We propose Context-Driven Claim Detection (ContextClaim), a paradigm that advances retrieval to the detection stage.<n>We evaluate ContextClaim on two datasets covering different topics and text genres, the CheckThat! 2022 Twitter dataset and the PoliClaim political debate dataset.
  arXiv  Detail & Related papers  (2026-03-31T17:20:46Z)
• KG-CRAFT: Knowledge Graph-based Contrastive Reasoning with LLMs for Enhancing Automated Fact-checking [3.7841869476488044]
  KG-CRAFT is a method that improves automatic claim verification by leveraging large language models (LLMs)<n>It first constructs a knowledge graph from claims and associated reports, then formulates contextually relevant contrastive questions based on the knowledge graph structure.<n>Extensive evaluations on two real-world datasets demonstrate that our method achieves a new state-of-the-art in predictive performance.
  arXiv  Detail & Related papers  (2026-01-27T10:32:42Z)
• SocraticKG: Knowledge Graph Construction via QA-Driven Fact Extraction [4.867319754310031]
  We propose an automated KG construction method that introduces question-answer pairs as a structured intermediate representation.<n>SocraticKG captures contextual dependencies and implicit relational links typically lost in direct KG extraction pipelines.
  arXiv  Detail & Related papers  (2026-01-15T02:26:51Z)
• Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integration [0.0]
  Rule-based reasoning arises in domains where decisions must be auditable and justifiable.<n>Applying rules to such inputs demands both interpretive flexibility and formal guarantees.<n>This paper presents an integration pattern that combines these strengths.
  arXiv  Detail & Related papers  (2026-01-04T17:19:20Z)
• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation [57.97548022208733]
  We show that seemingly superficial choices in key-value extraction can induce shifts in accuracy and stability.<n>We introduce Contextual Normalization, a strategy that adaptively standardizes context representations before generation.
  arXiv  Detail & Related papers  (2025-10-15T06:28:25Z)
• In-Context Learning for Pure Exploration [28.404325855738502]
  We study the problem active sequential hypothesis testing, also known as pure exploration.<n>We introduce In-Context Pure Exploration (ICPE), which meta-trains Transformers to map observation histories to query actions and a predicted hypothesis.<n>ICPE actively gathers evidence on new tasks and infers the true hypothesis without parameter updates.
  arXiv  Detail & Related papers  (2025-06-02T17:04:50Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [74.11403803488643]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation.<n>We show that QASemConsistency yields factual consistency scores that correlate well with human judgments.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition [63.51569687229681]
  We argue for the need to recognize the textual entailment relation of each proposition in a sentence individually. We propose PropSegmEnt, a corpus of over 45K propositions annotated by expert human raters. Our dataset structure resembles the tasks of (1) segmenting sentences within a document to the set of propositions, and (2) classifying the entailment relation of each proposition with respect to a different yet topically-aligned document.
  arXiv  Detail & Related papers  (2022-12-21T04:03:33Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.