Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together

• URL: http://arxiv.org/abs/2407.10930v2
• Date: Mon, 7 Oct 2024 15:52:48 GMT
• Title: Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together
• Authors: Dilara Soylu, Christopher Potts, Omar Khattab,
• Abstract summary: We seek strategies to optimize both the module-level LM weights and the associated prompt templates of such systems to maximize a downstream task metric. We propose for the first time combining the weight and prompt optimization strategies to optimize a modular LM pipeline by alternating between the two to get the same LM to teach itself.
• Score: 21.797319884895025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural Language Processing (NLP) systems are increasingly taking the form of sophisticated modular pipelines, e.g., Retrieval Augmented Generation (RAG), where each module may involve a distinct Language Model (LM) and an associated prompt template. These compound systems often lack intermediate labels or gradient flow to optimize each module, making their end-to-end optimization challenging. Here we seek strategies to optimize both the module-level LM weights and the associated prompt templates of such systems to maximize a downstream task metric. We propose for the first time combining the weight and prompt optimization strategies to optimize a modular LM pipeline by alternating between the two to get the same LM to teach itself. In experiments with multi-hop QA, mathematical reasoning, and feature-based classification using mistral-7b, llama-2-7b, and llama-3-8b, these BetterTogether strategies optimizing the weights and prompts of a pipeline together outperform directly optimizing weights alone and prompts alone by up to 60% and 6%, respectively, on average across LMs and tasks. BetterTogether optimizer is released in DSPy at http://dspy.ai

Related papers

• Optimizing Model Selection for Compound AI Systems [76.69936664916061]
  We propose an efficient framework for model selection in compound systems. It iteratively selects one module and allocates to it the model with the highest module-wise performance. It confers 5%-70% accuracy gains compared to using the same LLM for all modules.
  arXiv  Detail & Related papers  (2025-02-20T18:36:25Z)
• Using Large Language Models for Parametric Shape Optimization [2.464331481632096]
  We develop an optimization framework, LLM-PSO, to determine the optimal shape of parameterized engineering designs. Our preliminary exploration may inspire further investigations into harnessing LLMs for shape optimization and engineering design more broadly.
  arXiv  Detail & Related papers  (2024-12-11T03:35:38Z)
• Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization [65.64108848398696]
  We introduce a preference optimization process to enhance the multimodal reasoning capabilities of MLLMs. We develop a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10x larger InternVL2-76B.
  arXiv  Detail & Related papers  (2024-11-15T18:59:27Z)
• LLM-based Optimization of Compound AI Systems: A Survey [64.39860384538338]
  In a compound AI system, components such as an LLM call, a retriever, a code interpreter, or tools are interconnected. Recent advancements enable end-to-end optimization of these parameters using an LLM. This paper presents a survey of the principles and emerging trends in LLM-based optimization of compound AI systems.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs [40.159064885288245]
  We study prompt optimization for Language Model Programs. We factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module. We develop MIPRO, a novel algorithm for optimizing LM programs.
  arXiv  Detail & Related papers  (2024-06-17T16:12:03Z)
• LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning [69.95292905263393]
  We show that gradient-based and high-level LLMs can effectively collaborate a combined optimization framework. In this paper, we show that these complementary to each other and can effectively collaborate a combined optimization framework.
  arXiv  Detail & Related papers  (2024-05-30T06:24:14Z)
• Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers [108.72225067368592]
  We propose a novel perspective to investigate the design of large language models (LLMs)-based prompts. We identify two pivotal factors in model parameter learning: update direction and update method. We develop a capable Gradient-inspired Prompt-based GPO.
  arXiv  Detail & Related papers  (2024-02-27T15:05:32Z)
• Large Language Models as Optimizers [106.52386531624532]
  We propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as prompts. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values. We demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
  arXiv  Detail & Related papers  (2023-09-07T00:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.