BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification

• URL: http://arxiv.org/abs/2110.00171v1
• Date: Fri, 1 Oct 2021 02:03:43 GMT
• Title: BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification
• Authors: Zeguan Xiao, Jiarun Wu, Qingliang Chen and Congjian Deng
• Abstract summary: Graph-based Sentiment Classification (ABSC) approaches have yielded state-of-the-art results, expecially when equipped with contextual word embedding from pre-training language models (PLMs) We propose a novel model, BERT4GCN, which integrates the grammatical sequential features from the PLM of BERT, and the syntactic knowledge from dependency graphs.
• Score: 2.982218441172364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph-based Aspect-based Sentiment Classification (ABSC) approaches have yielded state-of-the-art results, expecially when equipped with contextual word embedding from pre-training language models (PLMs). However, they ignore sequential features of the context and have not yet made the best of PLMs. In this paper, we propose a novel model, BERT4GCN, which integrates the grammatical sequential features from the PLM of BERT, and the syntactic knowledge from dependency graphs. BERT4GCN utilizes outputs from intermediate layers of BERT and positional information between words to augment GCN (Graph Convolutional Network) to better encode the dependency graphs for the downstream classification. Experimental results demonstrate that the proposed BERT4GCN outperforms all state-of-the-art baselines, justifying that augmenting GCN with the grammatical features from intermediate layers of BERT can significantly empower ABSC models.

Related papers

• Language Models are Graph Learners [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, including Graph Neural Networks (GNNs) and Graph Transformers (GTs) We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• Enhancing ASL Recognition with GCNs and Successive Residual Connections [0.0]
  This study presents a novel approach for enhancing American Sign Language (ASL) recognition using Graph Convolutional Networks (GCNs) The method leverages the MediaPipe framework to extract key landmarks from each hand gesture, which are then used to construct graph representations. The constructed graphs are fed into a GCN-based neural architecture with residual connections to improve network stability.
  arXiv  Detail & Related papers  (2024-08-18T18:40:30Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Make BERT-based Chinese Spelling Check Model Enhanced by Layerwise Attention and Gaussian Mixture Model [33.446533426654995]
  We design a heterogeneous knowledge-infused framework to strengthen BERT-based CSC models. We propose a novel form of n-gram-based layerwise self-attention to generate a multilayer representation. Experimental results show that our proposed framework yields a stable performance boost over four strong baseline models.
  arXiv  Detail & Related papers  (2023-12-27T16:11:07Z)
• Syntactic Knowledge via Graph Attention with BERT in Machine Translation [0.0]
  We propose Syntactic knowledge via Graph attention with BERT (SGB) in Machine Translation (MT) scenarios. Our experiments use gold syntax-annotation sentences and Quality Estimation (QE) model to obtain interpretability of translation quality improvement. Experiments show that the proposed SGB engines improve translation quality across the three MT tasks without sacrificing BLEU scores.
  arXiv  Detail & Related papers  (2023-05-22T18:56:14Z)
• Graph Contrastive Learning for Skeleton-based Action Recognition [85.86820157810213]
  We propose a graph contrastive learning framework for skeleton-based action recognition. SkeletonGCL associates graph learning across sequences by enforcing graphs to be class-discriminative. SkeletonGCL establishes a new training paradigm, and it can be seamlessly incorporated into current graph convolutional networks.
  arXiv  Detail & Related papers  (2023-01-26T02:09:16Z)
• BertGCN: Transductive Text Classification by Combining GCN and BERT [33.866453485862124]
  BertGCN is a model that combines large scale pretraining and transductive learning for text classification. BertGCN SOTA achieves performances on a wide range of text classification datasets.
  arXiv  Detail & Related papers  (2021-05-12T15:20:01Z)
• Graph Convolutional Network for Swahili News Classification [78.6363825307044]
  This work empirically demonstrates the ability of Text Graph Convolutional Network (Text GCN) to outperform traditional natural language processing benchmarks for the task of semi-supervised Swahili news classification.
  arXiv  Detail & Related papers  (2021-03-16T21:03:47Z)
• An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text [72.62848911347466]
  Unstructured clinical text in EHRs contains crucial information for applications including decision support, trial matching, and retrospective research. Recent work has applied BERT-based models to clinical information extraction and text classification, given these models' state-of-the-art performance in other NLP domains. In this work, we propose a novel fine-tuning approach called SnipBERT. Instead of using entire notes, SnipBERT identifies crucial snippets and feeds them into a truncated BERT-based model in a hierarchical manner.
  arXiv  Detail & Related papers  (2020-11-12T17:14:32Z)
• On the Equivalence of Decoupled Graph Convolution Network and Label Propagation [60.34028546202372]
  Some work shows that coupling is inferior to decoupling, which supports deep graph propagation better. Despite effectiveness, the working mechanisms of the decoupled GCN are not well understood. We propose a new label propagation method named propagation then training Adaptively (PTA), which overcomes the flaws of the decoupled GCN.
  arXiv  Detail & Related papers  (2020-10-23T13:57:39Z)
• VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification [21.96079052962283]
  VGCN-BERT model combines the capability of BERT with a Vocabulary Graph Convolutional Network (VGCN) In our experiments on several text classification datasets, our approach outperforms BERT and GCN alone.
  arXiv  Detail & Related papers  (2020-04-12T22:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.