Explaining and Improving BERT Performance on Lexical Semantic Change Detection

• URL: http://arxiv.org/abs/2103.07259v1
• Date: Fri, 12 Mar 2021 13:29:30 GMT
• Title: Explaining and Improving BERT Performance on Lexical Semantic Change Detection
• Authors: Severin Laicher, Sinan Kurtyigit, Dominik Schlechtweg, Jonas Kuhn, Sabine Schulte im Walde
• Abstract summary: Recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word.
• Score: 22.934650688233734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Type- and token-based embedding architectures are still competing in lexical semantic change detection. The recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models on a variety of other NLP tasks does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word, which is encoded even in the higher layers of BERT representations. By reducing the influence of orthography we considerably improve BERT's performance.

Related papers

• Language Models are Graph Learners [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, including Graph Neural Networks (GNNs) and Graph Transformers (GTs) We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• BERTer: The Efficient One [0.0]
  We explore advanced fine-tuning techniques to boost BERT's performance in sentiment analysis, paraphrase detection, and semantic textual similarity. Our findings reveal substantial improvements in model efficiency and effectiveness when combining multiple fine-tuning architectures.
  arXiv  Detail & Related papers  (2024-07-19T05:33:09Z)
• Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting [0.0]
  This paper proposes a novel approach to improve embedding performance by leveraging large language models (LLMs) to enrich and rewrite input text before the embedding process. The effectiveness of this approach is evaluated on three datasets: Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual Classification.
  arXiv  Detail & Related papers  (2024-04-18T15:58:56Z)
• Make BERT-based Chinese Spelling Check Model Enhanced by Layerwise Attention and Gaussian Mixture Model [33.446533426654995]
  We design a heterogeneous knowledge-infused framework to strengthen BERT-based CSC models. We propose a novel form of n-gram-based layerwise self-attention to generate a multilayer representation. Experimental results show that our proposed framework yields a stable performance boost over four strong baseline models.
  arXiv  Detail & Related papers  (2023-12-27T16:11:07Z)
• Towards preserving word order importance through Forced Invalidation [80.33036864442182]
  We show that pre-trained language models are insensitive to word order. We propose Forced Invalidation to help preserve the importance of word order. Our experiments demonstrate that Forced Invalidation significantly improves the sensitivity of the models to word order.
  arXiv  Detail & Related papers  (2023-04-11T13:42:10Z)
• Learning Context-aware Classifier for Semantic Segmentation [88.88198210948426]
  In this paper, contextual hints are exploited via learning a context-aware classifier. Our method is model-agnostic and can be easily applied to generic segmentation models. With only negligible additional parameters and +2% inference time, decent performance gain has been achieved on both small and large models.
  arXiv  Detail & Related papers  (2023-03-21T07:00:35Z)
• The Topological BERT: Transforming Attention into Topology for Natural Language Processing [0.0]
  This paper introduces a text classifier using topological data analysis. We use BERT's attention maps transformed into attention graphs as the only input to that classifier. The model can solve tasks such as distinguishing spam from ham messages, recognizing whether a sentence is grammatically correct, or evaluating a movie review as negative or positive.
  arXiv  Detail & Related papers  (2022-06-30T11:25:31Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Multitask Learning for Class-Imbalanced Discourse Classification [74.41900374452472]
  We show that a multitask approach can improve 7% Micro F1-score upon current state-of-the-art benchmarks. We also offer a comparative review of additional techniques proposed to address resource-poor problems in NLP.
  arXiv  Detail & Related papers  (2021-01-02T07:13:41Z)
• Augmenting BERT Carefully with Underrepresented Linguistic Features [6.096779295981379]
  Fine-tuned Bidirectional Representations from Transformers (BERT)-based sequence classification models have proven to be effective for detecting Alzheimer's Disease (AD) from transcripts of human speech. Previous research shows it is possible to improve BERT's performance on various tasks by augmenting the model with additional information. We show that jointly fine-tuning BERT in combination with these features improves the performance of AD classification by upto 5% over fine-tuned BERT alone.
  arXiv  Detail & Related papers  (2020-11-12T01:32:41Z)
• Keyphrase Extraction with Dynamic Graph Convolutional Networks and Diversified Inference [50.768682650658384]
  Keyphrase extraction (KE) aims to summarize a set of phrases that accurately express a concept or a topic covered in a given document. Recent Sequence-to-Sequence (Seq2Seq) based generative framework is widely used in KE task, and it has obtained competitive performance on various benchmarks. In this paper, we propose to adopt the Dynamic Graph Convolutional Networks (DGCN) to solve the above two problems simultaneously.
  arXiv  Detail & Related papers  (2020-10-24T08:11:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.