Annotation and Classification of Evidence and Reasoning Revisions in Argumentative Writing

• URL: http://arxiv.org/abs/2107.06990v1
• Date: Wed, 14 Jul 2021 20:58:26 GMT
• Title: Annotation and Classification of Evidence and Reasoning Revisions in Argumentative Writing
• Authors: Tazin Afrin, Elaine Wang, Diane Litman, Lindsay C. Matsumura, Richard Correnti
• Abstract summary: We introduce an annotation scheme to capture the nature of sentence-level revisions of evidence use and reasoning. We show that reliable manual annotation can be achieved and that revision annotations correlate with a holistic assessment of essay improvement.
• Score: 0.9449650062296824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated writing evaluation systems can improve students' writing insofar as students attend to the feedback provided and revise their essay drafts in ways aligned with such feedback. Existing research on revision of argumentative writing in such systems, however, has focused on the types of revisions students make (e.g., surface vs. content) rather than the extent to which revisions actually respond to the feedback provided and improve the essay. We introduce an annotation scheme to capture the nature of sentence-level revisions of evidence use and reasoning (the `RER' scheme) and apply it to 5th- and 6th-grade students' argumentative essays. We show that reliable manual annotation can be achieved and that revision annotations correlate with a holistic assessment of essay improvement in line with the feedback provided. Furthermore, we explore the feasibility of automatically classifying revisions according to our scheme.

Related papers

• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [50.067342343957876]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL) Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions [7.503795054002406]
  We propose an original textual resource on the revision step of the writing process of scientific articles. This new dataset, called CASIMIR, contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews.
  arXiv  Detail & Related papers  (2024-03-01T03:07:32Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
  This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews. The newly emerging AI-generated literature reviews are also appraised. This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Predicting the Quality of Revisions in Argumentative Writing [2.0572032297930503]
  Chain-of-Thought prompts facilitate ChatGPT-generated ACs for AR quality predictions. Experiments on two corpora, our annotated elementary essays and existing college essays benchmark, demonstrate the superiority of the proposed ACs over baselines.
  arXiv  Detail & Related papers  (2023-06-01T13:39:33Z)
• To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support [20.905660642919052]
  We explore the main challenges to identifying argumentative claims in need of specific revisions. We propose a new sampling strategy based on revision distance. We provide evidence that using contextual information and domain knowledge can further improve prediction results.
  arXiv  Detail & Related papers  (2023-05-26T10:19:54Z)
• Predicting Desirable Revisions of Evidence and Reasoning in Argumentative Writing [1.0878040851638]
  We develop models to classify desirable evidence and desirable reasoning revisions in student argumentative writing. We explore two ways to improve performance - using the essay context of the revision, and using the feedback students received before the revision. Our results show that while a model using feedback information improves over a baseline model, models utilizing context - either alone or with feedback - are the most successful in identifying desirable revisions.
  arXiv  Detail & Related papers  (2023-02-10T03:59:59Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• ArgRewrite V.2: an Annotated Argumentative Revisions Corpus [10.65107335326471]
  ArgRewrite V.2 is a corpus of annotated argumentative revisions collected from two cycles of revisions to argumentative essays about self-driving cars. The variety of revision unit scope and purpose granularity levels in ArgRewrite, along with the inclusion of new types of meta-data, can make it a useful resource for research and applications that involve revision analysis.
  arXiv  Detail & Related papers  (2022-06-03T16:40:51Z)
• SNaC: Coherence Error Detection for Narrative Summarization [73.48220043216087]
  We introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators.
  arXiv  Detail & Related papers  (2022-05-19T16:01:47Z)
• A Unified Dual-view Model for Review Summarization and Sentiment Classification with Inconsistency Loss [51.448615489097236]
  Acquiring accurate summarization and sentiment from user reviews is an essential component of modern e-commerce platforms. We propose a novel dual-view model that jointly improves the performance of these two tasks. Experiment results on four real-world datasets from different domains demonstrate the effectiveness of our model.
  arXiv  Detail & Related papers  (2020-06-02T13:34:11Z)
• Code Review in the Classroom [57.300604527924015]
  Young developers in a classroom setting provide a clear picture of the potential favourable and problematic areas of the code review process. Their feedback suggests that the process has been well received with some points to better the process. This paper can be used as guidelines to perform code reviews in the classroom.
  arXiv  Detail & Related papers  (2020-04-19T06:07:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.