Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics

• URL: http://arxiv.org/abs/2502.05239v1
• Date: Fri, 07 Feb 2025 11:19:01 GMT
• Title: Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics
• Authors: Hussam Ghanem, Christophe Cruz,
• Abstract summary: This paper builds upon previous work, which evaluated various models using metrics like precision, recall, F1 score, triple matching, and graph matching.<n>We propose an enhanced evaluation framework incorporating BERTScore for graph similarity, setting a practical threshold of 95% for graph matching.
• Score: 0.9208007322096533
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in large language models have demonstrated significant potential in the automated construction of knowledge graphs from unstructured text. This paper builds upon our previous work [16], which evaluated various models using metrics like precision, recall, F1 score, triple matching, and graph matching, and introduces a refined approach to address the critical issues of hallucination and omission. We propose an enhanced evaluation framework incorporating BERTScore for graph similarity, setting a practical threshold of 95% for graph matching. Our experiments focus on the Mistral model, comparing its original and fine-tuned versions in zero-shot and few-shot settings. We further extend our experiments using examples from the KELM-sub training dataset, illustrating that the fine-tuned model significantly improves knowledge graph construction accuracy while reducing the exact hallucination and omission. However, our findings also reveal that the fine-tuned models perform worse in generalization tasks on the KELM-sub dataset. This study underscores the importance of comprehensive evaluation metrics in advancing the state-of-the-art in knowledge graph construction from textual data.

Related papers

• Combating Spurious Correlations in Graph Interpretability via Self-Reflection [4.81017678027464]
  Interpretable graph learning is a popular research topic in machine learning.<n>One of the most challenging is the Spurious-Motif benchmark, introduced at ICLR 2022.<n>We propose a self-reflection framework that can be integrated with existing interpretable graph learning methods.
  arXiv  Detail & Related papers  (2026-01-16T06:31:16Z)
• TinyGraphEstimator: Adapting Lightweight Language Models for Graph Structure Inference [0.48302896549293584]
  This paper investigates whether compact transformer-based language models can infer graph-theoretic parameters directly from graph representations.<n>We introduce the TinyGraphEstimator dataset - a collection of connected graphs generated from multiple random graph models.<n>We evaluate several small open models on their ability to predict key graph parameters such as density, clustering, and chromatic number.
  arXiv  Detail & Related papers  (2025-10-09T20:47:07Z)
• Revisiting Graph Neural Networks on Graph-level Tasks: Comprehensive Experiments, Analysis, and Improvements [54.006506479865344]
  We propose a unified evaluation framework for graph-level Graph Neural Networks (GNNs) This framework provides a standardized setting to evaluate GNNs across diverse datasets. We also propose a novel GNN model with enhanced expressivity and generalization capabilities.
  arXiv  Detail & Related papers  (2025-01-01T08:48:53Z)
• Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models [19.015202590038996]
  We evaluate the factuality of different models tuned by various preference learning algorithms. We propose textbfAPEFT (textbfAtomic textbfPreference textbfEnhanced textbfFactuality textbfTuning) to enhance model's awareness of factuality.
  arXiv  Detail & Related papers  (2024-06-18T09:07:30Z)
• Does Graph Distillation See Like Vision Dataset Counterpart? [26.530765707382457]
  We propose a novel structure-broadcasting Graph dataset Distillation (SGDD) scheme for broadcasting the original structure information to the generation of the synthetic one. We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-10-13T15:36:48Z)
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
  We present a code that successfully replicates results from six popular and recent graph recommendation models. We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
  arXiv  Detail & Related papers  (2023-08-01T09:31:44Z)
• FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback [69.4639239117551]
  FigCaps-HF is a new framework for figure-caption generation that incorporates domain expert feedback in generating captions optimized for reader preferences.<n>Our framework comprises of 1) an automatic method for evaluating quality of figure-caption pairs, 2) a novel reinforcement learning with human feedback (RLHF) method to optimize a generative figure-to-caption model for reader preferences.
  arXiv  Detail & Related papers  (2023-07-20T13:40:22Z)
• Permutation Equivariant Graph Framelets for Heterophilous Graph Learning [6.679929638714752]
  We develop a new way to implement multi-scale extraction via constructing Haar-type graph framelets. We show that our model can achieve the best performance on certain datasets of heterophilous graphs.
  arXiv  Detail & Related papers  (2023-06-07T09:05:56Z)
• Latent Augmentation For Better Graph Self-Supervised Learning [20.082614919182692]
  We argue that predictive models weaponed with latent augmentations and powerful decoder could achieve comparable or even better representation power than contrastive models. A novel graph decoder named Wiener Graph Deconvolutional Network is correspondingly designed to perform information reconstruction from augmented latent representations.
  arXiv  Detail & Related papers  (2022-06-26T17:41:59Z)
• FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations [114.94628499698096]
  We propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MRs) MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%.
  arXiv  Detail & Related papers  (2022-04-13T16:45:33Z)
• GraphCoCo: Graph Complementary Contrastive Learning [65.89743197355722]
  Graph Contrastive Learning (GCL) has shown promising performance in graph representation learning (GRL) without the supervision of manual annotations. This paper proposes an effective graph complementary contrastive learning approach named GraphCoCo to tackle the above issue.
  arXiv  Detail & Related papers  (2022-03-24T02:58:36Z)
• A Multi-Level Attention Model for Evidence-Based Fact Checking [58.95413968110558]
  We present a simple model that can be trained on sequence structures. Results on a large-scale dataset for Fact Extraction and VERification show that our model outperforms the graph-based approaches.
  arXiv  Detail & Related papers  (2021-06-02T05:40:12Z)
• Model-Agnostic Graph Regularization for Few-Shot Learning [60.64531995451357]
  We present a comprehensive study on graph embedded few-shot learning. We introduce a graph regularization approach that allows a deeper understanding of the impact of incorporating graph information between labels. Our approach improves the performance of strong base learners by up to 2% on Mini-ImageNet and 6.7% on ImageNet-FS.
  arXiv  Detail & Related papers  (2021-02-14T05:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.