ComFact: A Benchmark for Linking Contextual Commonsense Knowledge
- URL: http://arxiv.org/abs/2210.12678v1
- Date: Sun, 23 Oct 2022 09:30:39 GMT
- Title: ComFact: A Benchmark for Linking Contextual Commonsense Knowledge
- Authors: Silin Gao, Jena D. Hwang, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji,
Antoine Bosselut
- Abstract summary: We propose the new task of commonsense fact linking, where models are given contexts and trained to identify situationally-relevant commonsense knowledge from KGs.
Our novel benchmark, ComFact, contains 293k in-context relevance annotations for commonsense across four stylistically diverse datasets.
- Score: 31.19689856957576
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Understanding rich narratives, such as dialogues and stories, often requires
natural language processing systems to access relevant knowledge from
commonsense knowledge graphs. However, these systems typically retrieve facts
from KGs using simple heuristics that disregard the complex challenges of
identifying situationally-relevant commonsense knowledge (e.g.,
contextualization, implicitness, ambiguity).
In this work, we propose the new task of commonsense fact linking, where
models are given contexts and trained to identify situationally-relevant
commonsense knowledge from KGs. Our novel benchmark, ComFact, contains ~293k
in-context relevance annotations for commonsense triplets across four
stylistically diverse dialogue and storytelling datasets. Experimental results
confirm that heuristic fact linking approaches are imprecise knowledge
extractors. Learned fact linking models demonstrate across-the-board
performance improvements (~34.6% F1) over these heuristics. Furthermore,
improved knowledge retrieval yielded average downstream improvements of 9.8%
for a dialogue response generation task. However, fact linking models still
significantly underperform humans, suggesting our benchmark is a promising
testbed for research in commonsense augmentation of NLP systems.
Related papers
- SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge [60.76719375410635]
We propose a new benchmark (SOK-Bench) consisting of 44K questions and 10K situations with instance-level annotations depicted in the videos.
The reasoning process is required to understand and apply situated knowledge and general knowledge for problem-solving.
We generate associated question-answer pairs and reasoning processes, finally followed by manual reviews for quality assurance.
arXiv Detail & Related papers (2024-05-15T21:55:31Z) - DiffuCOMET: Contextual Commonsense Knowledge Diffusion [29.23102821128395]
In this work, we develop a series of knowledge models, DiffuCOMET, that leverage diffusion to learn to reconstruct the implicit semantic connections between narrative contexts and relevant commonsense knowledge.
To evaluate DiffuCOMET, we introduce new metrics for commonsense inference that more closely measure knowledge diversity and contextual relevance.
Our results on two different benchmarks, ComFact and WebNLG+, show that knowledge generated by DiffuCOMET achieves a better trade-off between commonsense diversity, contextual relevance and alignment to known gold references.
arXiv Detail & Related papers (2024-02-26T20:35:34Z) - Commonsense Knowledge Transfer for Pre-trained Language Models [83.01121484432801]
We introduce commonsense knowledge transfer, a framework to transfer the commonsense knowledge stored in a neural commonsense knowledge model to a general-purpose pre-trained language model.
It first exploits general texts to form queries for extracting commonsense knowledge from the neural commonsense knowledge model.
It then refines the language model with two self-supervised objectives: commonsense mask infilling and commonsense relation prediction.
arXiv Detail & Related papers (2023-06-04T15:44:51Z) - Knowledge Graph-Augmented Language Models for Knowledge-Grounded
Dialogue Generation [58.65698688443091]
We propose SUbgraph Retrieval-augmented GEneration (SURGE), a framework for generating context-relevant and knowledge-grounded dialogues with Knowledge Graphs (KGs)
Our framework first retrieves the relevant subgraph from the KG, and then enforces consistency across facts by perturbing their word embeddings conditioned by the retrieved subgraph.
We validate our SURGE framework on OpendialKG and KOMODIS datasets, showing that it generates high-quality dialogues that faithfully reflect the knowledge from KG.
arXiv Detail & Related papers (2023-05-30T08:36:45Z) - RHO ($\rho$): Reducing Hallucination in Open-domain Dialogues with
Knowledge Grounding [57.46495388734495]
This paper presents RHO ($rho$) utilizing the representations of linked entities and relation predicates from a knowledge graph (KG)
We propose (1) local knowledge grounding to combine textual embeddings with the corresponding KG embeddings; and (2) global knowledge grounding to equip RHO with multi-hop reasoning abilities via the attention mechanism.
arXiv Detail & Related papers (2022-12-03T10:36:34Z) - BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from
Pretrained Language Models [65.51390418485207]
We propose a new approach of harvesting massive KGs of arbitrary relations from pretrained LMs.
With minimal input of a relation definition, the approach efficiently searches in the vast entity pair space to extract diverse accurate knowledge.
We deploy the approach to harvest KGs of over 400 new relations from different LMs.
arXiv Detail & Related papers (2022-06-28T19:46:29Z) - Understanding Few-Shot Commonsense Knowledge Models [39.31365020474205]
We investigate training commonsense knowledge models in a few-shot setting.
We find that human quality ratings for knowledge produced from a few-shot trained system can achieve performance within 6% of knowledge produced from fully supervised systems.
arXiv Detail & Related papers (2021-01-01T19:01:09Z) - Improving Machine Reading Comprehension with Contextualized Commonsense
Knowledge [62.46091695615262]
We aim to extract commonsense knowledge to improve machine reading comprehension.
We propose to represent relations implicitly by situating structured knowledge in a context.
We employ a teacher-student paradigm to inject multiple types of contextualized knowledge into a student machine reader.
arXiv Detail & Related papers (2020-09-12T17:20:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.