Coreferential Reasoning Learning for Language Representation

• URL: http://arxiv.org/abs/2004.06870v2
• Date: Tue, 6 Oct 2020 12:37:53 GMT
• Title: Coreferential Reasoning Learning for Language Representation
• Authors: Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, Zhiyuan Liu
• Abstract summary: We present CorefBERT, a novel language representation model that can capture the coreferential relations in context. The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consistently on various downstream NLP tasks.
• Score: 88.14248323659267
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language representation models such as BERT could effectively capture contextual semantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropriate fine-tuning. However, most existing language representation models cannot explicitly handle coreference, which is essential to the coherent understanding of the whole discourse. To address this issue, we present CorefBERT, a novel language representation model that can capture the coreferential relations in context. The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consistently on various downstream NLP tasks that require coreferential reasoning, while maintaining comparable performance to previous models on other common NLP tasks. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/CorefBERT.

Related papers

• One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks [26.848664285007022]
  ByT5-Sanskrit is designed for NLP applications involving the morphologically rich language Sanskrit. It is easier to deploy and more robust to data not covered by external linguistic resources. We show that our approach yields new best scores for lemmatization and dependency parsing of other morphologically rich languages.
  arXiv  Detail & Related papers  (2024-09-20T22:02:26Z)
• To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer [23.777874316083984]
  We propose a scoring Language Quotient metric capable of providing a weighted representation of both zero-shot and few-shot evaluation combined. Our analysis reveals that image-based models excel in cross-lingual transfer when languages are closely related and share visually similar scripts. In dependency parsing tasks where word relationships play a crucial role, models with their character-level focus, outperform others.
  arXiv  Detail & Related papers  (2023-10-12T06:59:10Z)
• Pre-Training to Learn in Context [138.0745138788142]
  The ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters.
  arXiv  Detail & Related papers  (2023-05-16T03:38:06Z)
• Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters [35.103851212995046]
  Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. We explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. We develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:56:39Z)
• Visualizing the Relationship Between Encoded Linguistic Information and Task Performance [53.223789395577796]
  We study the dynamic relationship between the encoded linguistic information and task performance from the viewpoint of Pareto Optimality. We conduct experiments on two popular NLP tasks, i.e., machine translation and language modeling, and investigate the relationship between several kinds of linguistic information and task performances. Our empirical findings suggest that some syntactic information is helpful for NLP tasks whereas encoding more syntactic information does not necessarily lead to better performance.
  arXiv  Detail & Related papers  (2022-03-29T19:03:10Z)
• An Application of Pseudo-Log-Likelihoods to Natural Language Scoring [5.382454613390483]
  A language model with relatively few parameters and training steps can outperform it on a recent large data set. We produce some absolute state-of-the-art results for common sense reasoning in binary choice tasks. We argue that robustness of the smaller model ought to be understood in terms of compositionality.
  arXiv  Detail & Related papers  (2022-01-23T22:00:54Z)
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning [97.10875695679499]
  We propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks.
  arXiv  Detail & Related papers  (2020-12-30T03:35:22Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective [84.78604733927887]
  Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
  arXiv  Detail & Related papers  (2020-10-05T20:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.