GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs

• URL: http://arxiv.org/abs/2503.06139v1
• Date: Sat, 08 Mar 2025 09:44:24 GMT
• Title: GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs
• Authors: Mingyang Song, Mao Zheng, Xuan Luo,
• Abstract summary: Using Large Language Models (LLMs) to evaluate and compare two answers typically involves having LLM-based judges select the better answer.<n>We propose a Goal-Reversed Prompting (GRP) approach for pairwise evaluation that shifts the original task from selecting the better answer to choosing the worse one.
• Score: 14.906150451947443
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Using Large Language Models (LLMs) to evaluate and compare two answers from different models typically involves having LLM-based judges select the better answer. However, humans often approach problem-solving from a reverse perspective, for instance, by choosing the worse option instead of the better one in a pairwise comparison. Generally, this kind of reverse thinking plays a crucial role in human reasoning and decision-making and can further test the difference between original and reverse thought processes simultaneously. To address the above issue, in this paper, we propose a Goal-Reversed Prompting (GRP) approach for pairwise evaluation that shifts the original task from selecting the better answer to choosing the worse one. We encourage LLMs to think in reverse by prompting LLMs to identify the worse response. Experiments on closed-source models demonstrate that GRP significantly enhances evaluation capabilities, outperforming the prompt template with the original goal.

Related papers

• Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering [78.89231943329885]
  One of the most widely used tasks to evaluate Large Language Models (LLMs) is Multiple-Choice Question Answering (MCQA) In this work, we shed light on the inconsistencies of MCQA evaluation strategies, which can lead to inaccurate and misleading model comparisons.
  arXiv  Detail & Related papers  (2025-03-19T08:45:03Z)
• LLMs Can Generate a Better Answer by Aggregating Their Own Responses [83.69632759174405]
  Large Language Models (LLMs) have shown remarkable capabilities across tasks, yet they often require additional prompting techniques when facing complex problems.<n>We argue this limitation stems from the fact that common LLM post-training procedures lack explicit supervision for discriminative judgment tasks.<n>We propose Generative Self-Aggregation (GSA), a novel prompting method that improves answer quality without requiring the model's discriminative capabilities.
  arXiv  Detail & Related papers  (2025-03-06T05:25:43Z)
• Time-Reversal Provides Unsupervised Feedback to LLMs [31.575024356581846]
  Time Reversed Language Models (TRLMs) can score and generate queries when conditioned on responses. We show that TRLM scoring outperforms conventional forward scoring of response given query.
  arXiv  Detail & Related papers  (2024-12-03T17:54:12Z)
• Prompt Optimization with Human Feedback [69.95991134172282]
  We study the problem of prompt optimization with human feedback (POHF) We introduce our algorithm named automated POHF (APOHF) The results demonstrate that our APOHF can efficiently find a good prompt using a small number of preference feedback instances.
  arXiv  Detail & Related papers  (2024-05-27T16:49:29Z)
• Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts [95.09994361995389]
  Relative Preference Optimization (RPO) is designed to discern between more and less preferred responses derived from both identical and related prompts. RPO has demonstrated a superior ability to align large language models with user preferences and to improve their adaptability during the training process.
  arXiv  Detail & Related papers  (2024-02-12T22:47:57Z)
• LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback [65.84061725174269]
  Recent large language models (LLM) are leveraging human feedback to improve their generation quality. We propose LLMRefine, an inference time optimization method to refine LLM's output. We conduct experiments on three text generation tasks, including machine translation, long-form question answering (QA), and topical summarization. LLMRefine consistently outperforms all baseline approaches, achieving improvements up to 1.7 MetricX points on translation tasks, 8.1 ROUGE-L on ASQA, 2.2 ROUGE-L on topical summarization.
  arXiv  Detail & Related papers  (2023-11-15T19:52:11Z)
• The ART of LLM Refinement: Ask, Refine, and Trust [85.75059530612882]
  We propose a reasoning with refinement objective called ART: Ask, Refine, and Trust. It asks necessary questions to decide when an LLM should refine its output. It achieves a performance gain of +5 points over self-refinement baselines.
  arXiv  Detail & Related papers  (2023-11-14T07:26:32Z)
• PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations [10.709365940160685]
  Modern large language models (LLMs) are hard to evaluate and compare automatically.<n>We propose a peer rank (PR) algorithm that takes into account each peer LLM's pairwise preferences of all answer pairs.<n>We find that our approaches achieve higher accuracy and align better with human judgments.
  arXiv  Detail & Related papers  (2023-07-06T04:05:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.