Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring

• URL: http://arxiv.org/abs/2008.01441v1
• Date: Tue, 4 Aug 2020 10:17:38 GMT
• Title: Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring
• Authors: Robert Ridley, Liang He, Xinyu Dai, Shujian Huang, Jiajun Chen
• Abstract summary: Cross-prompt automated essay scoring (AES) requires the system to use non target-prompt essays to award scores to a target-prompt essay. This paper introduces Prompt Agnostic Essay Scorer (PAES) for cross-prompt AES. Our method requires no access to labelled or unlabelled target-prompt data during training and is a single-stage approach.
• Score: 61.21967763569547
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-prompt automated essay scoring (AES) requires the system to use non target-prompt essays to award scores to a target-prompt essay. Since obtaining a large quantity of pre-graded essays to a particular prompt is often difficult and unrealistic, the task of cross-prompt AES is vital for the development of real-world AES systems, yet it remains an under-explored area of research. Models designed for prompt-specific AES rely heavily on prompt-specific knowledge and perform poorly in the cross-prompt setting, whereas current approaches to cross-prompt AES either require a certain quantity of labelled target-prompt essays or require a large quantity of unlabelled target-prompt essays to perform transfer learning in a multi-step manner. To address these issues, we introduce Prompt Agnostic Essay Scorer (PAES) for cross-prompt AES. Our method requires no access to labelled or unlabelled target-prompt data during training and is a single-stage approach. PAES is easy to apply in practice and achieves state-of-the-art performance on the Automated Student Assessment Prize (ASAP) dataset.

Related papers

• What You Say = What You Want? Teaching Humans to Articulate Requirements for LLMs [26.398086645901742]
  We introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. In a study with 30 novices, we show that requirement-focused training doubles novices' prompting performance, significantly outperforming conventional prompt engineering training and prompt optimization. Our work paves the way for more effective task delegation in human-LLM collaborative prompting.
  arXiv  Detail & Related papers  (2024-09-13T12:34:14Z)
• LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models [17.15755066370757]
  Label-driven Automated Prompt Tuning (LAPT) is a novel approach to OOD detection that reduces the need for manual prompt engineering. We develop distribution-aware prompts with in-distribution (ID) class names and negative labels mined automatically. LAPT consistently outperforms manually crafted prompts, setting a new standard for OOD detection.
  arXiv  Detail & Related papers  (2024-07-12T03:30:53Z)
• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• TemPrompt: Multi-Task Prompt Learning for Temporal Relation Extraction in RAG-based Crowdsourcing Systems [21.312052922118585]
  Temporal relation extraction (TRE) aims to grasp the evolution of events or actions, and thus shape the workflow of associated tasks. We propose a multi-task prompt learning framework for TRE (TemPrompt), incorporating prompt tuning and contrastive learning to tackle these issues.
  arXiv  Detail & Related papers  (2024-06-21T01:52:37Z)
• Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability [5.426458555881673]
  This study aims to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability. We evaluate nine prominent AES methods and evaluate their performance using seven metrics on an open-sourced dataset.
  arXiv  Detail & Related papers  (2024-01-11T04:28:02Z)
• Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring [3.6825890616838066]
  Automated essay scoring (AES) aims to score essays written for a given prompt, which defines the writing topic. Most existing AES systems assume to grade essays of the same prompt as used in training and assign only a holistic score. We propose a robust model: prompt- and trait relation-aware cross-prompt essay trait scorer.
  arXiv  Detail & Related papers  (2023-05-26T11:11:19Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)
• MetaPrompting: Learning to Learn Better Prompts [52.914694884515534]
  We propose a new soft prompting method called MetaPrompting, which adopts the well-recognized model-agnostic meta-learning algorithm. Extensive experiments show MetaPrompting brings significant improvement on four different datasets.
  arXiv  Detail & Related papers  (2022-09-23T09:01:05Z)
• RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
  This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL) RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs) Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
  arXiv  Detail & Related papers  (2022-05-25T07:50:31Z)
• CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems [56.302581679816775]
  This paper proposes Comprehensive Instruction (CINS) that exploits PLMs with task-specific instructions. We design a schema (definition, constraint, prompt) of instructions and their customized realizations for three important downstream tasks in ToD. Experiments are conducted on these ToD tasks in realistic few-shot learning scenarios with small validation data.
  arXiv  Detail & Related papers  (2021-09-10T03:23:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.