Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction

• URL: http://arxiv.org/abs/2607.12835v1
• Date: Tue, 14 Jul 2026 14:52:42 GMT
• Title: Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
• Abstract summary: We present the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction.<n>We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models.
• Score: 38.170020271877085
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucination. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the affordances and limitations.

Related papers

• Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG [3.597778914286147]
  Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being input to a model's input context.<n>Prior work reports position-based effects such as lost in the middle and related long-context phenomena.<n>We show that topic sampling is a major source of variance: small topic sets can exaggerate ordering effects.<n>We also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents.
  arXiv  Detail & Related papers  (2026-05-26T14:44:19Z)
• Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning [47.963236269307735]
  Rerankers play a pivotal role in refining retrieval results for Retrieval-Augmented Generation.<n>Current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process.<n>We introduce ReRanking Preference Optimization (RRPO), a reinforcement learning framework that directly aligns reranking with the LLM's generation quality.
  arXiv  Detail & Related papers  (2026-04-02T14:19:47Z)
• Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators [2.952559770088323]
  Large language models (LLMs) offer new opportunities for automated research evaluation based on textual content.<n>This study examines whether LLMs can support post-publication peer review tasks by benchmarking their outputs against expert judgments and citation-based indicators.
  arXiv  Detail & Related papers  (2026-03-27T11:32:18Z)
• RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation [11.21565372620296]
  Large language models (LLMs) are increasingly evaluated and sometimes trained using automated graders that output scalar scores or preferences.<n>This lack of interpretability limits their usefulness for model development, dataset curation, and high-stakes deployment.<n>We introduce a simple strategy,RAG, which retrieves domain knowledge via rubrics at inference time from related queries.
  arXiv  Detail & Related papers  (2026-03-21T17:10:14Z)
• Structured Prompting Enables More Robust Evaluation of Language Models [38.53918044830268]
  We present a DSPy+HELM framework that introduces structured prompting methods which elicit reasoning.<n>We find that without structured prompting, HELM underestimates LM performance (by 4% average) and performance estimates vary more across benchmarks.<n>This is the first benchmarking study to systematically integrate structured prompting into an established evaluation framework.
  arXiv  Detail & Related papers  (2025-11-25T20:37:59Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews [18.50142644126276]
  We evaluate the alignment of automatic paper reviews with human reviews using an arena of human preferences by pairwise comparisons. We fine-tune an LLM to predict human preferences, predicting which reviews humans will prefer in a head-to-head battle between LLMs. We make the reviews of publicly available arXiv and open-access Nature journal papers available online, along with a free service which helps authors review and revise their research papers and improve their quality.
  arXiv  Detail & Related papers  (2024-08-19T19:10:38Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References [123.39034752499076]
  Div-Ref is a method to enhance evaluation benchmarks by enriching the number of references. We conduct experiments to empirically demonstrate that diversifying the expression of reference can significantly enhance the correlation between automatic evaluation and human evaluation.
  arXiv  Detail & Related papers  (2023-05-24T11:53:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.