A Realistic Study of Auto-regressive Language Models for Named Entity Typing and Recognition

• URL: http://arxiv.org/abs/2108.11857v1
• Date: Thu, 26 Aug 2021 15:29:00 GMT
• Title: A Realistic Study of Auto-regressive Language Models for Named Entity Typing and Recognition
• Authors: Elena V. Epure, Romain Hennequin
• Abstract summary: We study pre-trained language models for named entity recognition in a meta-learning setup. First, we test named entity typing (NET) in a zero-shot transfer scenario. Then, we perform NER by giving few examples at inference. We propose a method to select seen and rare / unseen names when having access only to the pre-trained model and report results on these groups.
• Score: 7.345578385749421
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite impressive results of language models for named entity recognition (NER), their generalization to varied textual genres, a growing entity type set, and new entities remains a challenge. Collecting thousands of annotations in each new case for training or fine-tuning is expensive and time-consuming. In contrast, humans can easily identify named entities given some simple instructions. Inspired by this, we challenge the reliance on large datasets and study pre-trained language models for NER in a meta-learning setup. First, we test named entity typing (NET) in a zero-shot transfer scenario. Then, we perform NER by giving few examples at inference. We propose a method to select seen and rare / unseen names when having access only to the pre-trained model and report results on these groups. The results show: auto-regressive language models as meta-learners can perform NET and NER fairly well especially for regular or seen names; name irregularity when often present for a certain entity type can become an effective exploitable cue; names with words foreign to the model have the most negative impact on results; the model seems to rely more on name than context cues in few-shot NER.

Related papers

• Evaluating Named Entity Recognition Using Few-Shot Prompting with Large Language Models [0.0]
  Few-Shot Prompting or in-context learning enables models to recognize entities with minimal examples. We assess state-of-the-art models like GPT-4 in NER tasks, comparing their few-shot performance to fully supervised benchmarks.
  arXiv  Detail & Related papers  (2024-08-28T13:42:28Z)
• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• Large-Scale Label Interpretation Learning for Few-Shot Named Entity Recognition [5.262708162539423]
  Few-shot named entity recognition (NER) detects named entities within text using only a few examples. One promising line of research is to leverage natural language descriptions of each entity type. In this paper, we explore the impact of a strong semantic prior to interpret verbalizations of new entity types.
  arXiv  Detail & Related papers  (2024-03-21T08:22:44Z)
• Multicultural Name Recognition For Previously Unseen Names [65.268245109828]
  This paper attempts to improve recognition of person names, a diverse category that can grow any time someone is born or changes their name. I look at names from 103 countries to compare how well the model performs on names from different cultures. I find that a model with combined character and word input outperforms word-only models and may improve on accuracy compared to classical NER models.
  arXiv  Detail & Related papers  (2024-01-23T17:58:38Z)
• Less than One-shot: Named Entity Recognition via Extremely Weak Supervision [46.81604901567282]
  We study the named entity recognition problem under the extremely weak supervision setting. We propose a novel method X-NER that can outperform the state-of-the-art one-shot NER methods. X-NER possesses several notable properties, such as inheriting the cross-lingual abilities of the underlying language models.
  arXiv  Detail & Related papers  (2023-11-06T04:20:42Z)
• Learning to Name Classes for Vision and Language Models [57.0059455405424]
  Large scale vision and language models can achieve impressive zero-shot recognition performance by mapping class specific text queries to image content. We propose to leverage available data to learn, for each class, an optimal word embedding as a function of the visual content. By learning new word embeddings on an otherwise frozen model, we are able to retain zero-shot capabilities for new classes, easily adapt models to new datasets, and adjust potentially erroneous, non-descriptive or ambiguous class names.
  arXiv  Detail & Related papers  (2023-04-04T14:34:44Z)
• Simple Questions Generate Named Entity Recognition Datasets [18.743889213075274]
  This work introduces an ask-to-generate approach, which automatically generates NER datasets by asking simple natural language questions. Our models largely outperform previous weakly supervised models on six NER benchmarks across four different domains. Formulating the needs of NER with natural language also allows us to build NER models for fine-grained entity types such as Award.
  arXiv  Detail & Related papers  (2021-12-16T11:44:38Z)
• Interpretable Entity Representations through Large-Scale Typing [61.4277527871572]
  We present an approach to creating entity representations that are human readable and achieve high performance out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types. We show that it is possible to reduce the size of our type set in a learning-based way for particular domains.
  arXiv  Detail & Related papers  (2020-04-30T23:58:03Z)
• Empower Entity Set Expansion via Language Model Probing [58.78909391545238]
  Existing set expansion methods bootstrap the seed entity set by adaptively selecting context features and extracting new entities. A key challenge for entity set expansion is to avoid selecting ambiguous context features which will shift the class semantics and lead to accumulative errors in later iterations. We propose a novel iterative set expansion framework that leverages automatically generated class names to address the semantic drift issue.
  arXiv  Detail & Related papers  (2020-04-29T00:09:43Z)
• Zero-Resource Cross-Domain Named Entity Recognition [68.83177074227598]
  Existing models for cross-domain named entity recognition rely on numerous unlabeled corpus or labeled NER training data in target domains. We propose a cross-domain NER model that does not use any external resources.
  arXiv  Detail & Related papers  (2020-02-14T09:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.