UoB at SemEval-2020 Task 12: Boosting BERT with Corpus Level Information

• URL: http://arxiv.org/abs/2008.08547v1
• Date: Wed, 19 Aug 2020 16:47:15 GMT
• Title: UoB at SemEval-2020 Task 12: Boosting BERT with Corpus Level Information
• Authors: Wah Meng Lim and Harish Tayyar Madabushi
• Abstract summary: We test the effectiveness of integrating Term Frequency-Inverse Document Frequency (TF-IDF) with BERT on the task of identifying abuse on social media. We achieve a score within two points of the top performing team and in Sub-Task B (target detection) wherein we are ranked 4 of the 44 participating teams.
• Score: 0.6980076213134383
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language model word representation, such as BERT, have been extremely successful in several Natural Language Processing tasks significantly improving on the state-of-the-art. This can largely be attributed to their ability to better capture semantic information contained within a sentence. Several tasks, however, can benefit from information available at a corpus level, such as Term Frequency-Inverse Document Frequency (TF-IDF). In this work we test the effectiveness of integrating this information with BERT on the task of identifying abuse on social media and show that integrating this information with BERT does indeed significantly improve performance. We participate in Sub-Task A (abuse detection) wherein we achieve a score within two points of the top performing team and in Sub-Task B (target detection) wherein we are ranked 4 of the 44 participating teams.

Related papers

• Rethinking and Improving Multi-task Learning for End-to-end Speech Translation [51.713683037303035]
  We investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. We propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation.
  arXiv  Detail & Related papers  (2023-11-07T08:48:46Z)
• InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings [61.77760317554826]
  This paper proposes an information-d contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large.
  arXiv  Detail & Related papers  (2022-10-08T15:53:19Z)
• Knowledge-Rich BERT Embeddings for Readability Assessment [0.0]
  We propose an alternative way of utilizing the information-rich embeddings of BERT models through a joint-learning method. Results show that the proposed method outperforms classical approaches in readability assessment using English and Filipino datasets.
  arXiv  Detail & Related papers  (2021-06-15T07:37:48Z)
• Augmenting BERT Carefully with Underrepresented Linguistic Features [6.096779295981379]
  Fine-tuned Bidirectional Representations from Transformers (BERT)-based sequence classification models have proven to be effective for detecting Alzheimer's Disease (AD) from transcripts of human speech. Previous research shows it is possible to improve BERT's performance on various tasks by augmenting the model with additional information. We show that jointly fine-tuning BERT in combination with these features improves the performance of AD classification by upto 5% over fine-tuned BERT alone.
  arXiv  Detail & Related papers  (2020-11-12T01:32:41Z)
• To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging [46.62643525729018]
  Cross-View Training (CVT) and comparing it with task-agnostic BERT in multiple settings that include domain and task relevant English data. We show that it achieves similar performance to BERT on a set of sequence tagging tasks, with lesser financial and environmental impact.
  arXiv  Detail & Related papers  (2020-10-27T04:03:47Z)
• Exploring Cross-sentence Contexts for Named Entity Recognition with BERT [1.4998865865537996]
  We present a study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context in the form of additional sentences to BERT input increases NER performance on all of the tested languages and models. We propose a straightforward method, Contextual Majority Voting (CMV), to combine different predictions for sentences and demonstrate this to further increase NER performance with BERT.
  arXiv  Detail & Related papers  (2020-06-02T12:34:52Z)
• Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-Task Learning for Offensive Language Detection [55.445023584632175]
  We build an offensive language detection system, which combines multi-task learning with BERT-based models. Our model achieves 91.51% F1 score in English Sub-task A, which is comparable to the first place.
  arXiv  Detail & Related papers  (2020-04-28T11:27:24Z)
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT [60.9051207862378]
  multilingual BERT works remarkably well on cross-lingual transfer tasks. Datasize and context window size are crucial factors to the transferability. There is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
  arXiv  Detail & Related papers  (2020-04-20T11:13:16Z)
• Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation [84.64004917951547]
  Fine-tuning pre-trained language models like BERT has become an effective way in NLP. In this paper, we improve the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation.
  arXiv  Detail & Related papers  (2020-02-24T16:17:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.