FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays

• URL: http://arxiv.org/abs/2606.06271v1
• Date: Thu, 04 Jun 2026 15:13:25 GMT
• Title: FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays
• Abstract summary: We introduce FOXGLOVE, a dataset of 696 feedback comments written by trained writing instructors on 69 twelfth-grade argumentative essays.<n>We find that instructors and LLMs distribute feedback similarly across goals and essay positions, yet instructors and models diverge on the specific sentences on which to provide feedback.
• Score: 21.90557945161922
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While large language models (LLMs) are increasingly used to generate writing feedback, there remains no systematic comparison of LLM and expert feedback on the dimensions that writing research identifies as central to revision: goal-orientation, anchoring to specific sentences, and prioritization. We introduce FOXGLOVE, a dataset of 696 feedback comments written by trained writing instructors on 69 twelfth-grade argumentative essays, paired with 1,644 comments generated from four frontier LLMs under a shared protocol, totaling 2,340 comments. We provide expert quality ratings on a subset of both instructor and LLM comments. We find that instructors and LLMs distribute feedback similarly across goals and essay positions, yet instructors and models diverge on the specific sentences on which to provide feedback. Additionally, we find that models tend to write more complex feedback and use fewer questions than instructors. LLM feedback also receives higher ratings on most dimensions of quality, as rated by instructors, but much of this advantage appears to be attributable to lengthier comments. FOXGLOVE enables systematic comparison of where human and LLM feedback align, diverge, and differ.

Related papers

• Consistently Good vs. Occasionally Great: A Rubric for Open-Ended Feedback Quality from Humans and Machines [3.4020399898813145]
  We focus on feedback for open-ended short answer questions in introductory programming.<n>We develop a five-criteria rubric grounded in educational literature for evaluating feedback quality.
  arXiv  Detail & Related papers  (2026-08-22T08:48:53Z)
• SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework [2.7531308685466183]
  SEFORA is a public corpus pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi-draft revisions.<n>UniMatch is a reference-based evaluation framework for open-ended generation.<n>It segments feedback into feedback units, scores semantic correspondence under instructor-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1.
  arXiv  Detail & Related papers  (2026-06-30T23:48:15Z)
• AI Fact-Checking in the Wild: A Field Evaluation of LLM-Written Community Notes on X [1.2423236865734466]
  Large language models show promising capabilities for contextual fact-checking on social media.<n>We present the first field evaluation of LLM-based fact-checking deployed on a live social media platform.<n>Our findings demonstrate that LLMs can contribute high-quality, broadly helpful fact-checking at scale.
  arXiv  Detail & Related papers  (2026-04-03T00:01:29Z)
• Reviewing the Reviewer: Elevating Peer Review Quality through LLM-Guided Feedback [75.31379834079648]
  We introduce an LLM-driven framework that decomposes reviews into argumentative segments.<n>We also release LazyReviewPlus, a dataset of 1,309 sentences labeled for lazy thinking and specificity.
  arXiv  Detail & Related papers  (2026-01-17T20:32:18Z)
• Exposía: Academic Writing Assessment of Exposés and Peer Feedback [56.428320613219306]
  We present Exposa, the first public dataset that connects writing and feedback assessment in higher education.<n>We use Exposa to benchmark state-of-the-art open-source large language models (LLMs) for two tasks: automated scoring of (1) the proposals and (2) the student reviews.
  arXiv  Detail & Related papers  (2026-01-10T11:33:26Z)
• When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review [34.067892820832405]
  This paper presents a systematic evaluation of large language models (LLMs) as academic reviewers.<n>Using a curated dataset of 1,441 papers from ICLR 2023 and NeurIPS 2022, we evaluate GPT-5-mini against human reviewers across ratings, strengths, and weaknesses.<n>Our findings show that LLMs consistently inflate ratings for weaker papers while aligning more closely with human judgments on stronger contributions.
  arXiv  Detail & Related papers  (2025-09-12T00:57:50Z)
• Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedback [57.200668979963694]
  We present a novel test set of 1,300 stories that we corrupted to intentionally introduce writing issues.<n>We study the performance of commonly used LLMs in this task with both automatic and human evaluation metrics.
  arXiv  Detail & Related papers  (2025-07-21T18:56:50Z)
• "I Wrote, I Paused, I Rewrote" Teaching LLMs to Read Between the Lines of Student Writing [0.0]
  Large language models like Gemini are becoming common tools for supporting student writing.<n>Most of their feedback is based only on the final essay missing important context about how that text was written.<n>We built a digital writing tool that captures both what students type and how their essays evolve over time.
  arXiv  Detail & Related papers  (2025-06-09T20:42:02Z)
• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
  Large language models (LLMs) have shown remarkable versatility in various generative tasks. This study focuses on the topic of LLMs assist NLP Researchers. To our knowledge, this is the first work to provide such a comprehensive analysis.
  arXiv  Detail & Related papers  (2024-06-24T01:30:22Z)
• Large Language Models as Evaluators for Recommendation Explanations [23.938202791437337]
  We investigate whether LLMs can serve as evaluators of recommendation explanations. We design and apply a 3-level meta evaluation strategy to measure the correlation between evaluator labels and the ground truth provided by users. Our study verifies that utilizing LLMs as evaluators can be an accurate, reproducible and cost-effective solution for evaluating recommendation explanation texts.
  arXiv  Detail & Related papers  (2024-06-05T13:23:23Z)
• Evaluating Large Language Models at Evaluating Instruction Following [54.49567482594617]
  We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. We discover that different evaluators exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement.
  arXiv  Detail & Related papers  (2023-10-11T16:38:11Z)
• Benchmarking Large Language Models for News Summarization [79.37850439866938]
  Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. We find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability.
  arXiv  Detail & Related papers  (2023-01-31T18:46:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.