Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation

• URL: http://arxiv.org/abs/2311.09136v3
• Date: Tue, 23 Jul 2024 04:35:45 GMT
• Title: Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation
• Authors: Yikun Wang, Rui Zheng, Haoming Li, Qi Zhang, Tao Gui, Fei Liu,
• Abstract summary: We develop a novel method to optimize LLMs using ranking metrics. Rather than a traditional full ordering, we advocate for a partial ordering. We test our system's improved response generation ability using benchmark datasets.
• Score: 28.89786334298637
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Customizing LLMs for a specific task involves separating high-quality responses from lower-quality ones. This skill can be developed using supervised fine-tuning with extensive human preference data. However, obtaining a large volume of expert-annotated data is costly for most tasks. In this paper, we explore a novel method to optimize LLMs using ranking metrics. This method trains the model to prioritize the best responses from a pool of candidates created for a particular task. Rather than a traditional full ordering, we advocate for a partial ordering, as achieving consensus on the perfect order of candidate responses can be challenging. Our partial ordering is more robust, less sensitive to noise, and can be achieved with limited human annotations or through heuristic methods. We test our system's improved response generation ability using benchmark datasets, including textual entailment and multi-document question answering. We conduct ablation studies to understand crucial factors, such as how to gather candidate responses for a specific task, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named Rescue, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.

Related papers

• REAL: Response Embedding-based Alignment for LLMs [1.9513983244114355]
  We propose a strategy for sampling a high-quality training dataset that focuses on acquiring the most informative response pairs. Experimental results indicate that choosing dissimilar response pairs enhances the direct alignment of LLMs. Our findings suggest that focusing on less similar pairs can improve the efficiency of LLM alignment, saving up to 65% of annotators' work.
  arXiv  Detail & Related papers  (2024-09-17T22:40:54Z)
• Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter [17.736962215696366]
  We introduce single-round instance-level prompt optimization, referred to as question rewriter. By enhancing the intelligibility of human questions for black-box LLMs, our question rewriter improves the quality of generated answers.
  arXiv  Detail & Related papers  (2024-08-20T06:24:47Z)
• QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
  We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries. We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
  arXiv  Detail & Related papers  (2024-08-20T03:06:48Z)
• Few-shot Personalization of LLMs with Mis-aligned Responses [40.0349773257245]
  This paper proposes a new approach for a few-shot personalization of large language models (LLMs) Our key idea is to learn a set of personalized prompts for each user by progressively improving the prompts using LLMs. During an iterative process of prompt improvement, we incorporate the contexts of mis-aligned responses by LLMs.
  arXiv  Detail & Related papers  (2024-06-26T18:29:12Z)
• Towards Hierarchical Multi-Agent Workflows for Zero-Shot Prompt Optimization [19.200989737492595]
  Large language models (LLMs) have shown great progress in responding to user questions. The quality of LLM outputs heavily depends on the prompt design, where a good prompt might enable the LLM to answer a very challenging question correctly. We propose a hierarchy of LLMs, first constructing a prompt with precise instructions and accurate wording in a hierarchical manner, and then using this prompt to generate the final answer to the user query.
  arXiv  Detail & Related papers  (2024-05-30T17:05:45Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs [85.54906813106683]
  We propose a simple yet effective framework to enhance open-domain question answering (ODQA) with large language models (LLMs) SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval (SuRe) Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches.
  arXiv  Detail & Related papers  (2024-04-17T01:15:54Z)
• LiPO: Listwise Preference Optimization through Learning-to-Rank [62.02782819559389]
  Policy can learn more effectively from a ranked list of plausible responses given the prompt. We show that LiPO-$lambda$ can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks.
  arXiv  Detail & Related papers  (2024-02-02T20:08:10Z)
• Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves [57.974103113675795]
  We present a method named Rephrase and Respond' (RaR) which allows Large Language Models to rephrase and expand questions posed by humans. RaR serves as a simple yet effective prompting method for improving performance. We show that RaR is complementary to the popular Chain-of-Thought (CoT) methods, both theoretically and empirically.
  arXiv  Detail & Related papers  (2023-11-07T18:43:34Z)
• Large Language Models are Zero-Shot Rankers for Recommender Systems [76.02500186203929]
  This work aims to investigate the capacity of large language models (LLMs) to act as the ranking model for recommender systems. We show that LLMs have promising zero-shot ranking abilities but struggle to perceive the order of historical interactions. We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies.
  arXiv  Detail & Related papers  (2023-05-15T17:57:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.