Trading Syntax Trees for Wordpieces: Target-oriented Opinion Words Extraction with Wordpieces and Aspect Enhancement

• URL: http://arxiv.org/abs/2305.11034v1
• Date: Thu, 18 May 2023 15:22:00 GMT
• Title: Trading Syntax Trees for Wordpieces: Target-oriented Opinion Words Extraction with Wordpieces and Aspect Enhancement
• Authors: Samuel Mensah, Kai Sun, Nikolaos Aletras
• Abstract summary: State-of-the-art target-oriented opinion word extraction (TOWE) models typically use BERT-based text encoders that operate on the word level. These methods achieve limited gains with graph convolutional networks (GCNs) and have difficulty using BERT wordpieces. This work trades syntax trees for BERT wordpieces by entirely removing the GCN component from the methods' architectures.
• Score: 33.66973706499751
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art target-oriented opinion word extraction (TOWE) models typically use BERT-based text encoders that operate on the word level, along with graph convolutional networks (GCNs) that incorporate syntactic information extracted from syntax trees. These methods achieve limited gains with GCNs and have difficulty using BERT wordpieces. Meanwhile, BERT wordpieces are known to be effective at representing rare words or words with insufficient context information. To address this issue, this work trades syntax trees for BERT wordpieces by entirely removing the GCN component from the methods' architectures. To enhance TOWE performance, we tackle the issue of aspect representation loss during encoding. Instead of solely utilizing a sentence as the input, we use a sentence-aspect pair. Our relatively simple approach achieves state-of-the-art results on benchmark datasets and should serve as a strong baseline for further research.

Related papers

• MarkBERT: Marking Word Boundaries Improves Chinese BERT [67.53732128091747]
  MarkBERT keeps the vocabulary being Chinese characters and inserts boundary markers between contiguous words. Compared to previous word-based BERT models, MarkBERT achieves better accuracy on text classification, keyword recognition, and semantic similarity tasks.
  arXiv  Detail & Related papers  (2022-03-12T08:43:06Z)
• Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words [50.11559460111882]
  We explore the possibility of developing BERT-style pretrained model over a vocabulary of words instead of wordpieces. Results show that, compared to standard wordpiece-based BERT, WordBERT makes significant improvements on cloze test and machine reading comprehension. Since the pipeline is language-independent, we train WordBERT for Chinese language and obtain significant gains on five natural language understanding datasets.
  arXiv  Detail & Related papers  (2022-02-24T15:15:48Z)
• Incorporating Constituent Syntax for Coreference Resolution [50.71868417008133]
  We propose a graph-based method to incorporate constituent syntactic structures. We also explore to utilise higher-order neighbourhood information to encode rich structures in constituent trees. Experiments on the English and Chinese portions of OntoNotes 5.0 benchmark show that our proposed model either beats a strong baseline or achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-02-22T07:40:42Z)
• BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification [2.982218441172364]
  Graph-based Sentiment Classification (ABSC) approaches have yielded state-of-the-art results, expecially when equipped with contextual word embedding from pre-training language models (PLMs) We propose a novel model, BERT4GCN, which integrates the grammatical sequential features from the PLM of BERT, and the syntactic knowledge from dependency graphs.
  arXiv  Detail & Related papers  (2021-10-01T02:03:43Z)
• An Empirical Study on Leveraging Position Embeddings for Target-oriented Opinion Words Extraction [13.765146062545048]
  Target-oriented opinion words extraction (TOWE) is a new subtask of target-oriented sentiment analysis. We show that BiLSTM-based models can effectively encode position information into word representations. We also adapt a graph convolutional network (GCN) to enhance word representations by incorporating syntactic information.
  arXiv  Detail & Related papers  (2021-09-02T22:49:45Z)
• Dependency Parsing based Semantic Representation Learning with Graph Neural Network for Enhancing Expressiveness of Text-to-Speech [49.05471750563229]
  We propose a semantic representation learning method based on graph neural network, considering dependency relations of a sentence. We show that our proposed method outperforms the baseline using vanilla BERT features both in LJSpeech and Bilzzard Challenge 2013 datasets.
  arXiv  Detail & Related papers  (2021-04-14T13:09:51Z)
• Cross-lingual Word Sense Disambiguation using mBERT Embeddings with Syntactic Dependencies [0.0]
  Cross-lingual word sense disambiguation (WSD) tackles the challenge of disambiguating ambiguous words across languages given context. BERT embedding model has been proven to be effective in contextual information of words. This project investigates how syntactic information can be added into the BERT embeddings to result in both semantics- and syntax-incorporated word embeddings.
  arXiv  Detail & Related papers  (2020-12-09T20:22:11Z)
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters [14.956626084281638]
  We propose a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level and open-vocabulary representations.
  arXiv  Detail & Related papers  (2020-10-20T15:58:53Z)
• Syntactic Structure Distillation Pretraining For Bidirectional Encoders [49.483357228441434]
  We introduce a knowledge distillation strategy for injecting syntactic biases into BERT pretraining. We distill the approximate marginal distribution over words in context from the syntactic LM. Our findings demonstrate the benefits of syntactic biases, even in representation learners that exploit large amounts of data.
  arXiv  Detail & Related papers  (2020-05-27T16:44:01Z)
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models [43.18970770343777]
  A contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models. We propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation.
  arXiv  Detail & Related papers  (2020-02-16T19:02:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.