Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference

• URL: http://arxiv.org/abs/2002.04815v1
• Date: Wed, 12 Feb 2020 06:11:48 GMT
• Title: Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference
• Authors: Youwei Song, Jiahai Wang, Zhiwei Liang, Zhiyue Liu, Tao Jiang
• Abstract summary: This paper explores the potential of utilizing BERT intermediate layers to enhance the performance of fine-tuning of BERT. To show the generality, we also apply this approach to a natural language inference task.
• Score: 19.638239426995973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aspect based sentiment analysis aims to identify the sentimental tendency towards a given aspect in text. Fine-tuning of pretrained BERT performs excellent on this task and achieves state-of-the-art performances. Existing BERT-based works only utilize the last output layer of BERT and ignore the semantic knowledge in the intermediate layers. This paper explores the potential of utilizing BERT intermediate layers to enhance the performance of fine-tuning of BERT. To the best of our knowledge, no existing work has been done on this research. To show the generality, we also apply this approach to a natural language inference task. Experimental results demonstrate the effectiveness and generality of the proposed approach.

Related papers

• Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis [67.41078214475341]
  We propose Dynamic Re-weighting BERT (DR-BERT) to learn dynamic aspect-oriented semantics for ABSA. Specifically, we first take the Stack-BERT layers as a primary encoder to grasp the overall semantic of the sentence. We then fine-tune it by incorporating a lightweight Dynamic Re-weighting Adapter (DRA)
  arXiv  Detail & Related papers  (2022-03-30T14:48:46Z)
• Knowledge-Rich BERT Embeddings for Readability Assessment [0.0]
  We propose an alternative way of utilizing the information-rich embeddings of BERT models through a joint-learning method. Results show that the proposed method outperforms classical approaches in readability assessment using English and Filipino datasets.
  arXiv  Detail & Related papers  (2021-06-15T07:37:48Z)
• An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text [72.62848911347466]
  Unstructured clinical text in EHRs contains crucial information for applications including decision support, trial matching, and retrospective research. Recent work has applied BERT-based models to clinical information extraction and text classification, given these models' state-of-the-art performance in other NLP domains. In this work, we propose a novel fine-tuning approach called SnipBERT. Instead of using entire notes, SnipBERT identifies crucial snippets and feeds them into a truncated BERT-based model in a hierarchical manner.
  arXiv  Detail & Related papers  (2020-11-12T17:14:32Z)
• On the Sentence Embeddings from Pre-trained Language Models [78.45172445684126]
  In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. We propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective.
  arXiv  Detail & Related papers  (2020-11-02T13:14:57Z)
• Understanding Pre-trained BERT for Aspect-based Sentiment Analysis [71.40586258509394]
  This paper analyzes the pre-trained hidden representations learned from reviews on BERT for tasks in aspect-based sentiment analysis (ABSA) It is not clear how the general proxy task of (masked) language model trained on unlabeled corpus without annotations of aspects or opinions can provide important features for downstream tasks in ABSA.
  arXiv  Detail & Related papers  (2020-10-31T02:21:43Z)
• Exploring Cross-sentence Contexts for Named Entity Recognition with BERT [1.4998865865537996]
  We present a study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context in the form of additional sentences to BERT input increases NER performance on all of the tested languages and models. We propose a straightforward method, Contextual Majority Voting (CMV), to combine different predictions for sentences and demonstrate this to further increase NER performance with BERT.
  arXiv  Detail & Related papers  (2020-06-02T12:34:52Z)
• Syntactic Structure Distillation Pretraining For Bidirectional Encoders [49.483357228441434]
  We introduce a knowledge distillation strategy for injecting syntactic biases into BERT pretraining. We distill the approximate marginal distribution over words in context from the syntactic LM. Our findings demonstrate the benefits of syntactic biases, even in representation learners that exploit large amounts of data.
  arXiv  Detail & Related papers  (2020-05-27T16:44:01Z)
• Comparing BERT against traditional machine learning text classification [0.0]
  The BERT model has arisen as a popular state-of-the-art machine learning model in the recent years. Our purpose of this work is to add empirical evidence to support or refuse the use of BERT as a default on NLP tasks.
  arXiv  Detail & Related papers  (2020-05-26T20:14:39Z)
• Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation [84.64004917951547]
  Fine-tuning pre-trained language models like BERT has become an effective way in NLP. In this paper, we improve the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation.
  arXiv  Detail & Related papers  (2020-02-24T16:17:12Z)
• Adversarial Training for Aspect-Based Sentiment Analysis with BERT [3.5493798890908104]
  We propose a novel architecture called BERT Adrial Training (BAT) to utilize adversarial training in sentiment analysis. The proposed model outperforms post-trained BERT in both tasks. To the best of our knowledge, this is the first study on the application of adversarial training in ABSA.
  arXiv  Detail & Related papers  (2020-01-30T13:53:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.