On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation

• URL: http://arxiv.org/abs/2205.03835v1
• Date: Sun, 8 May 2022 10:36:54 GMT
• Title: On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation
• Authors: Yongjie Wang and Chuan Wang and Ruobing Li and Hui Lin
• Abstract summary: In this paper, we introduce a novel multi-scale essay representation for BERT that can be jointly learned. Experiment results show that our approach derives much benefit from joint learning of multi-scale essay representation. Our multi-scale essay representation also generalizes well to CommonLit Readability Prize data set.
• Score: 12.896747108919968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, pre-trained models have become dominant in most natural language processing (NLP) tasks. However, in the area of Automated Essay Scoring (AES), pre-trained models such as BERT have not been properly used to outperform other deep learning models such as LSTM. In this paper, we introduce a novel multi-scale essay representation for BERT that can be jointly learned. We also employ multiple losses and transfer learning from out-of-domain essays to further improve the performance. Experiment results show that our approach derives much benefit from joint learning of multi-scale essay representation and obtains almost the state-of-the-art result among all deep learning models in the ASAP task. Our multi-scale essay representation also generalizes well to CommonLit Readability Prize data set, which suggests that the novel text representation proposed in this paper may be a new and effective choice for long-text tasks.

Related papers

• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• Improving Text Embeddings with Large Language Models [59.930513259982725]
  We introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data.
  arXiv  Detail & Related papers  (2023-12-31T02:13:18Z)
• UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding [93.92313947913831]
  We introduce UniDoc, a novel multimodal model equipped with text detection and recognition capabilities. To the best of our knowledge, this is the first large multimodal model capable of simultaneous text detection, recognition, spotting, and understanding.
  arXiv  Detail & Related papers  (2023-08-19T17:32:34Z)
• Towards General Text Embeddings with Multi-stage Contrastive Learning [20.803769345818456]
  GTE is a general-purpose text embedding model trained with multi-stage contrastive learning. We train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources.
  arXiv  Detail & Related papers  (2023-08-07T03:52:59Z)
• Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study [68.75670223005716]
  We find that pre-trained language models like BERT have a potential ability to learn sequentially, even without any sparse memory replay. Our experiments reveal that BERT can actually generate high quality representations for previously learned tasks in a long term, under extremely sparse replay or even no replay.
  arXiv  Detail & Related papers  (2023-03-02T09:03:43Z)
• The Next Chapter: A Study of Large Language Models in Storytelling [51.338324023617034]
  The application of prompt-based learning with large language models (LLMs) has exhibited remarkable performance in diverse natural language processing (NLP) tasks. This paper conducts a comprehensive investigation, utilizing both automatic and human evaluation, to compare the story generation capacity of LLMs with recent models. The results demonstrate that LLMs generate stories of significantly higher quality compared to other story generation models.
  arXiv  Detail & Related papers  (2023-01-24T02:44:02Z)
• A Unified Neural Network Model for Readability Assessment with Feature Projection and Length-Balanced Loss [17.213602354715956]
  We propose a BERT-based model with feature projection and length-balanced loss for readability assessment. Our model achieves state-of-the-art performances on two English benchmark datasets and one dataset of Chinese textbooks.
  arXiv  Detail & Related papers  (2022-10-19T05:33:27Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)
• Transferring BERT-like Transformers' Knowledge for Authorship Verification [8.443350618722562]
  We study the effectiveness of several BERT-like transformers for the task of authorship verification. We provide new splits for PAN-2020, where training and test data are sampled from disjoint topics or authors. We show that those splits can enhance the models' capability to transfer knowledge over a new, significantly different dataset.
  arXiv  Detail & Related papers  (2021-12-09T18:57:29Z)
• Comparing BERT against traditional machine learning text classification [0.0]
  The BERT model has arisen as a popular state-of-the-art machine learning model in the recent years. Our purpose of this work is to add empirical evidence to support or refuse the use of BERT as a default on NLP tasks.
  arXiv  Detail & Related papers  (2020-05-26T20:14:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.