RDBE: Reasoning Distillation-Based Evaluation Enhances Automatic Essay Scoring

• URL: http://arxiv.org/abs/2407.13781v1
• Date: Wed, 3 Jul 2024 05:49:01 GMT
• Title: RDBE: Reasoning Distillation-Based Evaluation Enhances Automatic Essay Scoring
• Authors: Ali Ghiasvand Mohammadkhani,
• Abstract summary: Reasoning Distillation-Based Evaluation (RDBE) integrates interpretability to elucidate the rationale behind model scores. Our experimental results demonstrate the efficacy of RDBE across all scoring rubrics considered in the dataset.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, various encoder-only and encoder-decoder pre-trained models like BERT and T5 have been applied to automatic essay scoring (AES) as small language models. However, existing studies have primarily treated this task akin to a classification problem, focusing solely on outputting scores in the target text without offering interpretations for the generated scores. Departing from the approaches, we introduce Reasoning Distillation-Based Evaluation (RDBE), which integrates interpretability to elucidate the rationale behind model scores while enhancing performance through initial reasoning. This interpretive capability is acquired during training by leveraging generated reasoning from a large language model (LLM) to distill a small language model (SLM). Our experimental results demonstrate the efficacy of RDBE across all scoring rubrics considered in the dataset. RDBE outperforms both zero-shot LLM generation and generation from a baseline fine-tuned model, establishing itself as state-of-the-art in the corresponding dataset. This highlights its practical interpretative output and enhanced performance.

Related papers

• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics [49.9329723199239]
  We propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset.
  arXiv  Detail & Related papers  (2024-10-04T13:39:21Z)
• Data Shapley in One Training Run [88.59484417202454]
  Data Shapley provides a principled framework for attributing data's contribution within machine learning contexts. Existing approaches require re-training models on different data subsets, which is computationally intensive. This paper introduces In-Run Data Shapley, which addresses these limitations by offering scalable data attribution for a target model of interest.
  arXiv  Detail & Related papers  (2024-06-16T17:09:24Z)
• Enhancing Retrieval-Augmented LMs with a Two-stage Consistency Learning Compressor [4.35807211471107]
  This work proposes a novel two-stage consistency learning approach for retrieved information compression in retrieval-augmented language models. The proposed method is empirically validated across multiple datasets, demonstrating notable enhancements in precision and efficiency for question-answering tasks.
  arXiv  Detail & Related papers  (2024-06-04T12:43:23Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings [32.03184402316848]
  We analyze attribution scores extracted from prompt-based models w.r.t. plausibility and faithfulness. We find that using the prompting paradigm yields more plausible explanations than fine-tuning the models in low-resource settings.
  arXiv  Detail & Related papers  (2024-03-08T14:14:37Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)
• Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU [70.44344060176952]
  Intent classification is a major task in spoken language understanding (SLU) Recent works have shown that using extra data and labels can improve the OOD detection performance. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection.
  arXiv  Detail & Related papers  (2021-06-28T08:27:38Z)
• Zero-Resource Multi-Dialectal Arabic Natural Language Understanding [0.0]
  We investigate the zero-shot performance on Dialectal Arabic (DA) when fine-tuning a pre-trained language model on modern standard Arabic (MSA) data only. We propose self-training with unlabeled DA data and apply it in the context of named entity recognition (NER), part-of-speech (POS) tagging, and sarcasm detection (SRD) Our results demonstrate the effectiveness of self-training with unlabeled DA data.
  arXiv  Detail & Related papers  (2021-04-14T02:29:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.