Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?

• URL: http://arxiv.org/abs/2605.29857v1
• Date: Thu, 28 May 2026 12:38:39 GMT
• Title: Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
• Abstract summary: We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts.<n>Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments.
• Score: 7.837071197834482
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts. Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments. We evaluate the proposed method in real-world review settings and in controlled settings with reference rubrics. These results show that inline comments can be distilled into reusable rubrics that support comment prediction, rubric understanding, and automatic artifact revision.

Related papers

• ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation [54.82704239643649]
  We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans.<n>Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback.<n>We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.
  arXiv  Detail & Related papers  (2026-09-08T17:30:16Z)
• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation [74.44235943118994]
  GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
  arXiv  Detail & Related papers  (2026-08-30T15:39:39Z)
• Rethinking Training Data for Generating Code Review Comments [1.7212147528714599]
  We argue that problematic training instances are not homogeneous and that some limitations stem from deeper issues in the task formulation itself.<n>We derive a taxonomy of misalignment capturing semantic ambiguity, lack of actionability, and context dependence.
  arXiv  Detail & Related papers  (2026-07-28T15:24:30Z)
• Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments [3.55827089220224]
  Code review is a cornerstone of software development, where reviewers provide feedback through written comments to ensure code quality, maintainability, and correctness.<n>Existing code review datasets are often noisy, inconsistent, or poorly structured.<n>We propose two different curation pipelines designed to improve the quality and the utility of large-scale code review datasets.
  arXiv  Detail & Related papers  (2026-07-10T15:33:47Z)
• ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents [50.27474750319121]
  Review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages.
  arXiv  Detail & Related papers  (2026-04-15T16:33:04Z)
• When the Gold Standard isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content [23.830040496621965]
  We examine the human translation guidelines of four datasets.<n>We derive a taxonomy of twelve non-standard phenomena and five translation actions.<n>We call for clear guidelines during creation and for the development of controllable, guideline-aware evaluation frameworks.
  arXiv  Detail & Related papers  (2025-12-19T16:17:23Z)
• Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems [0.0]
  We integrate two types of feedback-driven annotations: those that identify spelling and grammatical errors, and those that highlight argumentative components.<n>To illustrate how this method could be applied in real-world scenarios, we employ two LLMs to generate annotations -- a generative language model used for spell correction and an encoder-based token-classifier trained to identify and mark argumentative elements.
  arXiv  Detail & Related papers  (2025-05-28T18:39:56Z)
• Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries [85.81295563405433]
  We present a protocol that synthetically constructs context surrounding an under-specified query and provides it during evaluation.<n>We find that the presence of context can 1) alter conclusions drawn from evaluation, even flipping benchmark rankings between model pairs, 2) nudge evaluators to make fewer judgments based on surface-level criteria, like style, and 3) provide new insights about model behavior across diverse contexts.
  arXiv  Detail & Related papers  (2024-11-11T18:58:38Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Annotation and Classification of Evidence and Reasoning Revisions in Argumentative Writing [0.9449650062296824]
  We introduce an annotation scheme to capture the nature of sentence-level revisions of evidence use and reasoning. We show that reliable manual annotation can be achieved and that revision annotations correlate with a holistic assessment of essay improvement.
  arXiv  Detail & Related papers  (2021-07-14T20:58:26Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.