Towards Optimizing with Large Language Models

• URL: http://arxiv.org/abs/2310.05204v3
• Date: Mon, 27 May 2024 09:13:26 GMT
• Title: Towards Optimizing with Large Language Models
• Authors: Pei-Fu Guo, Ying-Hsuan Chen, Yun-Da Tsai, Shou-De Lin,
• Abstract summary: We conduct an assessment of the optimization capabilities of LLMs across various tasks and data sizes. We introduce three distinct metrics for a comprehensive assessment of task performance from various perspectives.
• Score: 3.80039497875781
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we conduct an assessment of the optimization capabilities of LLMs across various tasks and data sizes. Each of these tasks corresponds to unique optimization domains, and LLMs are required to execute these tasks with interactive prompting. That is, in each optimization step, the LLM generates new solutions from the past generated solutions with their values, and then the new solutions are evaluated and considered in the next optimization step. Additionally, we introduce three distinct metrics for a comprehensive assessment of task performance from various perspectives. These metrics offer the advantage of being applicable for evaluating LLM performance across a broad spectrum of optimization tasks and are less sensitive to variations in test samples. By applying these metrics, we observe that LLMs exhibit strong optimization capabilities when dealing with small-sized samples. However, their performance is significantly influenced by factors like data size and values, underscoring the importance of further research in the domain of optimization tasks for LLMs.

Related papers

• Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System [75.25394449773052]
  Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving. Yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness.
  arXiv  Detail & Related papers  (2024-10-10T17:00:06Z)
• EVOLvE: Evaluating and Optimizing LLMs For Exploration [76.66831821738927]
  Large language models (LLMs) remain under-studied in scenarios requiring optimal decision-making under uncertainty. We measure LLMs' (in)ability to make optimal decisions in bandits, a state-less reinforcement learning setting relevant to many applications. Motivated by the existence of optimal exploration algorithms, we propose efficient ways to integrate this algorithmic knowledge into LLMs.
  arXiv  Detail & Related papers  (2024-10-08T17:54:03Z)
• QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
  We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries. We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks. Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
  arXiv  Detail & Related papers  (2024-08-20T03:06:48Z)
• MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization [73.7779735046424]
  We show that different prompts should be adapted to different Large Language Models (LLM) to enhance their capabilities across various downstream tasks in NLP. We then propose a model-adaptive prompt (MAPO) method that optimize the original prompts for each specific LLM in downstream tasks.
  arXiv  Detail & Related papers  (2024-07-04T18:39:59Z)
• Iterative or Innovative? A Problem-Oriented Perspective for Code Optimization [81.88668100203913]
  Large language models (LLMs) have demonstrated strong capabilities in solving a wide range of programming tasks. In this paper, we explore code optimization with a focus on performance enhancement, specifically aiming to optimize code for minimal execution time.
  arXiv  Detail & Related papers  (2024-06-17T16:10:10Z)
• OptLLM: Optimal Assignment of Queries to Large Language Models [12.07164196530872]
  We propose a framework for addressing the cost-effective query allocation problem for large language models (LLMs) Our framework, named OptLLM, provides users with a range of optimal solutions to choose from, aligning with their budget constraints and performance preferences. To evaluate the effectiveness of OptLLM, we conduct extensive experiments on various types of tasks, including text classification, question answering, sentiment analysis, reasoning, and log parsing.
  arXiv  Detail & Related papers  (2024-05-24T01:05:37Z)
• Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers [15.809293135844756]
  We revisit OPRO for automated prompting with relatively small-scale Language Models (LLMs) OPRO shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. We suggest future automatic prompting engineering to consider both model capabilities and computational costs.
  arXiv  Detail & Related papers  (2024-05-16T17:33:50Z)
• Exploring the True Potential: Evaluating the Black-box Optimization Capability of Large Language Models [32.859634302766146]
  Large language models (LLMs) have demonstrated exceptional performance in natural language processing tasks. This paper endeavors to offer deep insights into the potential of LLMs in optimization. Our findings reveal both the limitations and advantages of LLMs in optimization.
  arXiv  Detail & Related papers  (2024-04-09T13:17:28Z)
• Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation [33.41556606816004]
  Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to- task. There is still no consensus on the optimal prompt templates and design frameworks. Existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to- process.
  arXiv  Detail & Related papers  (2024-03-05T13:23:48Z)
• How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problems [33.33996058215666]
  Large language models (LLMs) have positioned themselves as capable tools for addressing complex optimization challenges. We propose to enhance the optimization performance using multimodal LLM capable of processing both textual and visual prompts.
  arXiv  Detail & Related papers  (2024-03-04T06:24:21Z)
• Large Language Models as Optimizers [106.52386531624532]
  We propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as prompts. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values. We demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
  arXiv  Detail & Related papers  (2023-09-07T00:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.