Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset

• URL: http://arxiv.org/abs/2109.07679v1
• Date: Thu, 16 Sep 2021 02:50:01 GMT
• Title: Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset
• Authors: Tianqing Fang, Weiqi Wang, Sehyun Choi, Shibo Hao, Hongming Zhang, Yangqiu Song, Bin He
• Abstract summary: Reasoning over commonsense knowledge bases (CSKB) whose elements are in the form of free-text is an important yet hard task in NLP. We benchmark the CSKB population task with a new large-scale dataset. We also propose a novel inductive commonsense reasoning model that reasons over graphs.
• Score: 37.02104430195374
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning over commonsense knowledge bases (CSKB) whose elements are in the form of free-text is an important yet hard task in NLP. While CSKB completion only fills the missing links within the domain of the CSKB, CSKB population is alternatively proposed with the goal of reasoning unseen assertions from external resources. In this task, CSKBs are grounded to a large-scale eventuality (activity, state, and event) graph to discriminate whether novel triples from the eventuality graph are plausible or not. However, existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation). In this paper, we benchmark the CSKB population task with a new large-scale dataset by first aligning four popular CSKBs, and then presenting a high-quality human-annotated evaluation set to probe neural models' commonsense reasoning ability. We also propose a novel inductive commonsense reasoning model that reasons over graphs. Experimental results show that generalizing commonsense reasoning on unseen assertions is inherently a hard task. Models achieving high accuracy during training perform poorly on the evaluation set, with a large gap between human performance. We will make the data publicly available for future contributions. Codes and data are available at https://github.com/HKUST-KnowComp/CSKB-Population.

Related papers

• Real World Conversational Entity Linking Requires More Than Zeroshots [50.5691094768954]
  We design targeted evaluation scenarios to measure the efficacy of EL models under resource constraints. We assess EL models' ability to generalize to a new unfamiliar KB using Fandom and a novel zero-shot conversational entity linking dataset. Results indicate that current zero-shot EL models falter when introduced to new, domain-specific KBs without prior training.
  arXiv  Detail & Related papers  (2024-09-02T10:37:53Z)
• BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learning [39.090104460303415]
  offline model-based reinforcement learning (MBRL) enhances data efficiency by utilizing pre-collected datasets to learn models and policies. This paper first identifies the primary source of this mismatch comes from the underlying confounders present in offline data. We introduce textbfBilintextbfEar textbfCAUSal rtextbfEpresentation(BECAUSE), an algorithm to capture causal representation for both states.
  arXiv  Detail & Related papers  (2024-07-15T17:59:23Z)
• Graph Out-of-Distribution Generalization via Causal Intervention [69.70137479660113]
  We introduce a conceptually simple yet principled approach for training robust graph neural networks (GNNs) under node-level distribution shifts. Our method resorts to a new learning objective derived from causal inference that coordinates an environment estimator and a mixture-of-expert GNN predictor. Our model can effectively enhance generalization with various types of distribution shifts and yield up to 27.4% accuracy improvement over state-of-the-arts on graph OOD generalization benchmarks.
  arXiv  Detail & Related papers  (2024-02-18T07:49:22Z)
• CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering [56.592385613002584]
  We propose Conceptualization-Augmented Reasoner (CAR) to tackle the task of zero-shot commonsense question answering. CAR abstracts a commonsense knowledge triple to many higher-level instances, which increases the coverage of CommonSense Knowledge Bases. CAR more robustly generalizes to answering questions about zero-shot commonsense scenarios than existing methods.
  arXiv  Detail & Related papers  (2023-05-24T08:21:31Z)
• PseudoReasoner: Leveraging Pseudo Labels for Commonsense Knowledge Base Population [40.526736652672916]
  We propose PseudoReasoner, a semi-supervised learning framework for CSKB population. It uses a teacher model pre-trained on CSKBs to provide pseudo labels on the unlabeled candidate dataset for a student model to learn from. The framework can improve the backbone model KG-BERT by 3.3 points on the overall performance and especially, 5.3 points on the out-of-domain performance.
  arXiv  Detail & Related papers  (2022-10-14T17:37:30Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• Combining human parsing with analytical feature extraction and ranking schemes for high-generalization person reidentification [0.0]
  Person reidentification (re-ID) has been receiving increasing attention in recent years due to its importance for both science and society. Machine learning and particularly Deep Learning (DL) has become the main re-id tool that allowed researches to achieve unprecedented accuracy levels on benchmark datasets. We present a model without trainable parameters which shows great potential for high generalization.
  arXiv  Detail & Related papers  (2022-07-28T17:22:48Z)
• General Greedy De-bias Learning [163.65789778416172]
  We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space. GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
  arXiv  Detail & Related papers  (2021-12-20T14:47:32Z)
• Zero-shot Node Classification with Decomposed Graph Prototype Network [33.24920910739568]
  We study a zero-shot node classification (ZNC) problem which has a two-stage nature. For the first stage, we give a novel quantitative CSDs evaluation strategy based on estimating the real class relationships. For the second stage, we propose a novel Decomposed Graph Prototype Network (DGPN) method.
  arXiv  Detail & Related papers  (2021-06-15T10:13:20Z)
• Ranking vs. Classifying: Measuring Knowledge Base Completion Quality [10.06803520598035]
  We argue that consideration of binary predictions is essential to reflect the actual KBC quality. We simulate the realistic scenario of real-world entities missing from a KB. We evaluate a number of state-of-the-art KB embeddings models on our new benchmark.
  arXiv  Detail & Related papers  (2021-02-02T17:53:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.