Knowledge Graph Quality Evaluation under Incomplete Information
- URL: http://arxiv.org/abs/2212.00994v3
- Date: Wed, 12 Apr 2023 07:53:54 GMT
- Title: Knowledge Graph Quality Evaluation under Incomplete Information
- Authors: Xiaodong Li, Chenxin Zou, Yi Cai, Yuelong Zhu
- Abstract summary: We propose a knowledge graph quality evaluation framework under incomplete information (QEII)
The quality evaluation task is transformed into an adversarial Q&A game between two KGs.
During the evaluation process, no raw data is exposed, which ensures information protection.
- Score: 9.48089663504665
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Knowledge graphs (KGs) have attracted more and more attentions because of
their fundamental roles in many tasks. Quality evaluation for KGs is thus
crucial and indispensable. Existing methods in this field evaluate KGs by
either proposing new quality metrics from different dimensions or measuring
performances at KG construction stages. However, there are two major issues
with those methods. First, they highly rely on raw data in KGs, which makes
KGs' internal information exposed during quality evaluation. Second, they
consider more about the quality at data level instead of ability level, where
the latter one is more important for downstream applications. To address these
issues, we propose a knowledge graph quality evaluation framework under
incomplete information (QEII). The quality evaluation task is transformed into
an adversarial Q&A game between two KGs. Winner of the game is thus considered
to have better qualities. During the evaluation process, no raw data is
exposed, which ensures information protection. Experimental results on four
pairs of KGs demonstrate that, compared with baselines, the QEII implements a
reasonable quality evaluation at ability level under incomplete information.
Related papers
- Q-Ground: Image Quality Grounding with Large Multi-modality Models [61.72022069880346]
We introduce Q-Ground, the first framework aimed at tackling fine-scale visual quality grounding.
Q-Ground combines large multi-modality models with detailed visual quality analysis.
Central to our contribution is the introduction of the QGround-100K dataset.
arXiv Detail & Related papers (2024-07-24T06:42:46Z) - Uncertainty Management in the Construction of Knowledge Graphs: a Survey [3.5639148953570845]
Knowledge Graphs (KGs) are a major asset for companies thanks to their great flexibility in data representation.
To build a KG it is a common practice to rely on automatic methods for extracting knowledge from various heterogeneous sources.
In a noisy and uncertain world, knowledge may not be reliable and conflicts between data sources may occur.
arXiv Detail & Related papers (2024-05-27T08:22:52Z) - Towards Explainable In-the-Wild Video Quality Assessment: A Database and
a Language-Prompted Approach [52.07084862209754]
We collect over two million opinions on 4,543 in-the-wild videos on 13 dimensions of quality-related factors.
Specifically, we ask the subjects to label among a positive, a negative, and a neutral choice for each dimension.
These explanation-level opinions allow us to measure the relationships between specific quality factors and abstract subjective quality ratings.
arXiv Detail & Related papers (2023-05-22T05:20:23Z) - Evaluating the Knowledge Dependency of Questions [12.25396414711877]
We propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA)
We first show how to measure KDA based on student responses from a human survey.
Then, we propose two automatic evaluation metrics, KDA_disc and KDA_cont, that approximate KDA by leveraging pre-trained language models to imitate students' problem-solving behavior.
arXiv Detail & Related papers (2022-11-21T23:08:30Z) - Knowledge Graph Curation: A Practical Framework [0.0]
We propose a practical knowledge graph curation framework for improving the quality of KGs.
First, we define a set of quality metrics for assessing the status of KGs.
Second, we describe the verification and validation of KGs as cleaning tasks.
Third, we present duplicate detection and knowledge fusion strategies for enriching KGs.
arXiv Detail & Related papers (2022-08-17T07:55:28Z) - Knowledge Graph Question Answering Leaderboard: A Community Resource to
Prevent a Replication Crisis [61.740077541531726]
We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community.
Our analysis highlights existing problems during the evaluation of KGQA systems.
arXiv Detail & Related papers (2022-01-20T13:46:01Z) - How Knowledge Graph and Attention Help? A Quantitative Analysis into
Bag-level Relation Extraction [66.09605613944201]
We quantitatively evaluate the effect of attention and Knowledge Graph on bag-level relation extraction (RE)
We find that (1) higher attention accuracy may lead to worse performance as it may harm the model's ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns; and (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior.
arXiv Detail & Related papers (2021-07-26T09:38:28Z) - QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question
Answering [122.84513233992422]
We propose a new model, QA-GNN, which addresses the problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs)
We show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning.
arXiv Detail & Related papers (2021-04-13T17:32:51Z) - Object-QA: Towards High Reliable Object Quality Assessment [71.71188284059203]
In object recognition applications, object images usually appear with different quality levels.
We propose an effective approach named Object-QA to assess high-reliable quality scores for object images.
arXiv Detail & Related papers (2020-05-27T01:46:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.