Annotation and Classification of Evidence and Reasoning Revisions in
Argumentative Writing
- URL: http://arxiv.org/abs/2107.06990v1
- Date: Wed, 14 Jul 2021 20:58:26 GMT
- Title: Annotation and Classification of Evidence and Reasoning Revisions in
Argumentative Writing
- Authors: Tazin Afrin, Elaine Wang, Diane Litman, Lindsay C. Matsumura, Richard
Correnti
- Abstract summary: We introduce an annotation scheme to capture the nature of sentence-level revisions of evidence use and reasoning.
We show that reliable manual annotation can be achieved and that revision annotations correlate with a holistic assessment of essay improvement.
- Score: 0.9449650062296824
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Automated writing evaluation systems can improve students' writing insofar as
students attend to the feedback provided and revise their essay drafts in ways
aligned with such feedback. Existing research on revision of argumentative
writing in such systems, however, has focused on the types of revisions
students make (e.g., surface vs. content) rather than the extent to which
revisions actually respond to the feedback provided and improve the essay. We
introduce an annotation scheme to capture the nature of sentence-level
revisions of evidence use and reasoning (the `RER' scheme) and apply it to 5th-
and 6th-grade students' argumentative essays. We show that reliable manual
annotation can be achieved and that revision annotations correlate with a
holistic assessment of essay improvement in line with the feedback provided.
Furthermore, we explore the feasibility of automatically classifying revisions
according to our scheme.
Related papers
- ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction [26.64363135181992]
We explore the impact of shifting from sentence-level to paragraph-level scope for the task of scientific text revision.
The paragraph level definition of the task allows for more meaningful changes, and is guided by detailed revision instructions rather than general ones.
Our experiments demonstrate that using detailed instructions significantly improves the quality of automated revisions compared to general approaches.
arXiv Detail & Related papers (2025-01-09T13:19:55Z) - eRevise+RF: A Writing Evaluation System for Assessing Student Essay Revisions and Providing Formative Feedback [1.5367711550341163]
eRevise+RF is an enhanced AWE system for assessing student essay revisions and providing revision feedback.
We deployed the system with 6 teachers and 406 students across 3 schools in Pennsylvania and Louisiana.
The results confirmed its effectiveness in (1) assessing student essays in terms of evidence usage, (2) extracting evidence and reasoning revisions across essays, and (3) determining revision success in responding to feedback.
arXiv Detail & Related papers (2025-01-01T03:49:48Z) - Predicting the Quality of Revisions in Argumentative Writing [2.0572032297930503]
Chain-of-Thought prompts facilitate ChatGPT-generated ACs for AR quality predictions.
Experiments on two corpora, our annotated elementary essays and existing college essays benchmark, demonstrate the superiority of the proposed ACs over baselines.
arXiv Detail & Related papers (2023-06-01T13:39:33Z) - To Revise or Not to Revise: Learning to Detect Improvable Claims for
Argumentative Writing Support [20.905660642919052]
We explore the main challenges to identifying argumentative claims in need of specific revisions.
We propose a new sampling strategy based on revision distance.
We provide evidence that using contextual information and domain knowledge can further improve prediction results.
arXiv Detail & Related papers (2023-05-26T10:19:54Z) - Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation [55.00687185394986]
We propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews.
We introduce the ORSUM dataset covering 15,062 paper meta-reviews and 57,536 paper reviews from 47 conferences.
Our experiments show that (1) human-written summaries do not always satisfy all necessary criteria such as depth of discussion, and identifying consensus and controversy for the specific domain, and (2) the combination of task decomposition and iterative self-refinement shows strong potential for enhancing the opinions.
arXiv Detail & Related papers (2023-05-24T02:33:35Z) - Predicting Desirable Revisions of Evidence and Reasoning in
Argumentative Writing [1.0878040851638]
We develop models to classify desirable evidence and desirable reasoning revisions in student argumentative writing.
We explore two ways to improve performance - using the essay context of the revision, and using the feedback students received before the revision.
Our results show that while a model using feedback information improves over a baseline model, models utilizing context - either alone or with feedback - are the most successful in identifying desirable revisions.
arXiv Detail & Related papers (2023-02-10T03:59:59Z) - Investigating Fairness Disparities in Peer Review: A Language Model
Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs)
We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date.
We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z) - SNaC: Coherence Error Detection for Narrative Summarization [73.48220043216087]
We introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries.
We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries.
Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators.
arXiv Detail & Related papers (2022-05-19T16:01:47Z) - Measuring "Why" in Recommender Systems: a Comprehensive Survey on the
Evaluation of Explainable Recommendation [87.82664566721917]
This survey is based on more than 100 papers from top-tier conferences like IJCAI, AAAI, TheWebConf, Recsys, UMAP, and IUI.
arXiv Detail & Related papers (2022-02-14T02:58:55Z) - A Unified Dual-view Model for Review Summarization and Sentiment
Classification with Inconsistency Loss [51.448615489097236]
Acquiring accurate summarization and sentiment from user reviews is an essential component of modern e-commerce platforms.
We propose a novel dual-view model that jointly improves the performance of these two tasks.
Experiment results on four real-world datasets from different domains demonstrate the effectiveness of our model.
arXiv Detail & Related papers (2020-06-02T13:34:11Z) - Code Review in the Classroom [57.300604527924015]
Young developers in a classroom setting provide a clear picture of the potential favourable and problematic areas of the code review process.
Their feedback suggests that the process has been well received with some points to better the process.
This paper can be used as guidelines to perform code reviews in the classroom.
arXiv Detail & Related papers (2020-04-19T06:07:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.