PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

• URL: http://arxiv.org/abs/2603.10477v1
• Date: Wed, 11 Mar 2026 07:00:59 GMT
• Title: PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses
• Abstract summary: We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for evaluation of both prompts and responses.<n>PEEM defines a structured rubric with 9 axes: 3 prompt criteria (clarity/structure, linguistic quality, fairness) and 6 response criteria (accuracy, objectivity, relevance, clarity, conciseness)
• Score: 5.396994583933599
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt design is a primary control interface for large language models (LLMs), yet standard evaluations largely reduce performance to answer correctness, obscuring why a prompt succeeds or fails and providing little actionable guidance. We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for joint and interpretable evaluation of both prompts and responses. PEEM defines a structured rubric with 9 axes: 3 prompt criteria (clarity/structure, linguistic quality, fairness) and 6 response criteria (accuracy, coherence, relevance, objectivity, clarity, conciseness), and uses an LLM-based evaluator to output (i) scalar scores on a 1-5 Likert scale and (ii) criterion-specific natural-language rationales grounded in the rubric. Across 7 benchmarks and 5 task models, PEEM's accuracy axis strongly aligns with conventional accuracy while preserving model rankings (aggregate Spearman rho about 0.97, Pearson r about 0.94, p < 0.001). A multi-evaluator study with four models shows consistent relative judgments (pairwise rho = 0.68-0.85), supporting evaluator-agnostic deployment. Beyond alignment, PEEM captures complementary linguistic failure modes and remains informative under prompt perturbations: prompt-quality trends track downstream accuracy under iterative rewrites, semantic adversarial manipulations induce clear score degradation, and meaning-preserving paraphrases yield high stability (robustness rate about 76.7-80.6%). Finally, using only PEEM scores and rationales as feedback, a zero-shot prompt rewriting loop improves downstream accuracy by up to 11.7 points, outperforming supervised and RL-based prompt-optimization baselines. Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.

Related papers

• Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges [2.8523456804049885]
  Safety judges are increasingly deployed to evaluate model outputs against evolving criteria.<n>Recent meta-evaluation work shows they remain brittle under prompt and rubric variation.<n>We propose a training strategy that combines instance-conditioned dynamic rubrics and a reliable-to-expressive curriculum.
  arXiv  Detail & Related papers  (2026-06-08T08:02:57Z)
• Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment [2.8217668849101405]
  We introduce Generative-Evaluative Agreement (GEA), a validity criterion measuring whether an LLM's scoring function recovers the skill levels its generative function was instructed to produce.<n>We argue that granular, skill-decomposed rubrics are the principal proposed mechanism for strengthening GEA and outline complementary mitigations.
  arXiv  Detail & Related papers  (2026-05-19T08:30:39Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Structured Prompting Enables More Robust Evaluation of Language Models [38.53918044830268]
  We present a DSPy+HELM framework that introduces structured prompting methods which elicit reasoning.<n>We find that without structured prompting, HELM underestimates LM performance (by 4% average) and performance estimates vary more across benchmarks.<n>This is the first benchmarking study to systematically integrate structured prompting into an established evaluation framework.
  arXiv  Detail & Related papers  (2025-11-25T20:37:59Z)
• Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise [41.16092952642748]
  This study investigates the potential for enhancing Automated Scoring (AES) by refining rubrics used by Large Language Models (LLMs)<n> Experiments on datasets using GPT-4.1, Gemini-2.5-Pro, and Qwen-3-Next-80B-A3B-Instruct show Quadratic Weighted Kappa (QWK) improvements of up to 0.19 and 0.47, respectively.
  arXiv  Detail & Related papers  (2025-10-10T06:05:38Z)
• Re-FRAME the Meeting Summarization SCOPE: Fact-Based Summarization and Personalization via Questions [9.381476258394175]
  We present FRAME, a modular pipeline that reframes summarization as a semantic enrichment task.<n> FRAME extracts and scores salient facts, organizes them thematically, and uses these to enrich an outline into an abstractive summary.<n>To personalize summaries, we introduce SCOPE, a reason-out-loud protocol that has the model build a reasoning trace by answering nine questions before content selection.
  arXiv  Detail & Related papers  (2025-09-19T11:58:17Z)
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning.<n>Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate.<n>We propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision.
  arXiv  Detail & Related papers  (2025-06-19T08:49:13Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)
• ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments [23.514446188485838]
  We argue for a method of moments evaluation over the space of meaning-preserving prompt perturbations.<n>We show that even top-performing models like GPT-4o and Claude-3.7-Sonnet exhibit substantial prompt sensitivity.
  arXiv  Detail & Related papers  (2025-05-28T09:40:48Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step.<n>Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy.<n>We propose Reasoning-Driven Process Reward Modeling (R-PRM)<n>R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.