A RelEntLess Benchmark for Modelling Graded Relations between Named Entities

• URL: http://arxiv.org/abs/2305.15002v2
• Date: Wed, 31 Jan 2024 03:56:22 GMT
• Title: A RelEntLess Benchmark for Modelling Graded Relations between Named Entities
• Authors: Asahi Ushio and Jose Camacho Collados and Steven Schockaert
• Abstract summary: We introduce a new benchmark, in which entity pairs have to be ranked according to how much they satisfy a given graded relation. We find a strong correlation between model size and performance, with smaller Language Models struggling to outperform a naive baseline. The results of the largest Flan-T5 and OPT models are remarkably strong, although a clear gap with human performance remains.
• Score: 29.528217625083546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Relations such as "is influenced by", "is known for" or "is a competitor of" are inherently graded: we can rank entity pairs based on how well they satisfy these relations, but it is hard to draw a line between those pairs that satisfy them and those that do not. Such graded relations play a central role in many applications, yet they are typically not covered by existing Knowledge Graphs. In this paper, we consider the possibility of using Large Language Models (LLMs) to fill this gap. To this end, we introduce a new benchmark, in which entity pairs have to be ranked according to how much they satisfy a given graded relation. The task is formulated as a few-shot ranking problem, where models only have access to a description of the relation and five prototypical instances. We use the proposed benchmark to evaluate state-of-the-art relation embedding strategies as well as several recent LLMs, covering both publicly available LLMs and closed models such as GPT-4. Overall, we find a strong correlation between model size and performance, with smaller Language Models struggling to outperform a naive baseline. The results of the largest Flan-T5 and OPT models are remarkably strong, although a clear gap with human performance remains.

Related papers

• Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning [5.4141465747474475]
  Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving problems of moderate complexity. We systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph.
  arXiv  Detail & Related papers  (2025-02-19T20:20:24Z)
• CPTuning: Contrastive Prompt Tuning for Generative Relation Extraction [2.883903547507341]
  Generative relation extraction (RE) commonly involves first reformulating RE as a linguistic modeling problem easily tackled with pre-trained language models (PLM) We introduce a novel contrastive prompt tuning method for RE, CPTuning, which learns to associate a candidate relation between two in-context entities. CTPuning also organizes RE as a verbalized relation generation task and uses Trie-constrained decoding to ensure a model generates valid relations.
  arXiv  Detail & Related papers  (2025-01-04T05:17:34Z)
• Exploring Model Kinship for Merging Large Language Models [52.01652098827454]
  We introduce model kinship, the degree of similarity or relatedness between Large Language Models. We find that there is a certain relationship between model kinship and the performance gains after model merging. We propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-16T14:29:29Z)
• Efficient Document Ranking with Learnable Late Interactions [73.41976017860006]
  Cross-Encoder (CE) and Dual-Encoder (DE) models are two fundamental approaches for query-document relevance in information retrieval. To predict relevance, CE models use joint query-document embeddings, while DE models maintain factorized query and document embeddings. Recently, late-interaction models have been proposed to realize more favorable latency-quality tradeoffs, by using a DE structure followed by a lightweight scorer.
  arXiv  Detail & Related papers  (2024-06-25T22:50:48Z)
• RelBERT: Embedding Relations with Language Models [29.528217625083546]
  We propose to extract relation embeddings from relatively small language models. RelBERT captures relational similarity in a surprisingly fine-grained way. It is capable of modelling relations that go well beyond what the model has seen during training.
  arXiv  Detail & Related papers  (2023-09-30T08:15:36Z)
• A Comprehensive Study on Knowledge Graph Embedding over Relational Patterns Based on Rule Learning [49.09125100268454]
  Knowledge Graph Embedding (KGE) has proven to be an effective approach to solving the Knowledge Completion Graph (KGC) task. Relational patterns are an important factor in the performance of KGE models. We introduce a training-free method to enhance KGE models' performance over various relational patterns.
  arXiv  Detail & Related papers  (2023-08-15T17:30:57Z)
• Relational Learning with Gated and Attentive Neighbor Aggregator for Few-Shot Knowledge Graph Completion [33.59045268013895]
  We propose a few-shot relational learning with global-local framework to address the above issues. For the local stage, a meta-learning based TransH method is designed to model complex relations and train our model in a few-shot learning fashion. Our model achieves 5-shot FKGC performance improvements of 8.0% on NELL-One and 2.8% on Wiki-One by the metric Hits@10.
  arXiv  Detail & Related papers  (2021-04-27T10:38:44Z)
• Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction [84.64435075778988]
  We propose a general approach to learn relation prototypes from unlabeled texts. We learn relation prototypes as an implicit factor between entities. We conduct experiments on two publicly available datasets: New York Times and Google Distant Supervision.
  arXiv  Detail & Related papers  (2020-11-27T06:21:12Z)
• Learning to Decouple Relations: Few-Shot Relation Classification with Entity-Guided Attention and Confusion-Aware Training [49.9995628166064]
  We propose CTEG, a model equipped with two mechanisms to learn to decouple easily-confused relations. On the one hand, an EGA mechanism is introduced to guide the attention to filter out information causing confusion. On the other hand, a Confusion-Aware Training (CAT) method is proposed to explicitly learn to distinguish relations.
  arXiv  Detail & Related papers  (2020-10-21T11:07:53Z)
• Type-augmented Relation Prediction in Knowledge Graphs [65.88395564516115]
  We propose a type-augmented relation prediction (TaRP) method, where we apply both the type information and instance-level information for relation prediction. Our proposed TaRP method achieves significantly better performance than state-of-the-art methods on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-16T21:14:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.