Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization

• URL: http://arxiv.org/abs/2403.00067v3
• Date: Mon, 22 Jul 2024 03:53:32 GMT
• Title: Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization
• Authors: Md Tahmid Rahman Laskar, Elena Khasanova, Xue-Yong Fu, Cheng Chen, Shashi Bhushan TN,
• Abstract summary: We investigate whether combining the queries for the same input context in a single prompt to minimize repeated calls can be successfully used in meeting summarization. We observe that 100% reliability in generating the response in the expected format is usually limited to certain closed-source LLMs.
• Score: 7.674972936853123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work focuses on the task of query-based meeting summarization in which the summary of a context (meeting transcript) is generated in response to a specific query. When using Large Language Models (LLMs) for this task, usually a new call to the LLM inference endpoint/API is triggered for each new query, even if the context stays the same. However, repeated calls to the LLM inference endpoints would significantly increase the costs of using them in production, making LLMs impractical for many real-world use cases. To address this problem, in this paper, we investigate whether combining the queries for the same input context in a single prompt to minimize repeated calls can be successfully used in meeting summarization. In this regard, we conduct extensive experiments by comparing the performance of various popular LLMs: GPT-4, Gemini, Claude-3, LLaMA-2, Mistral, Phi-3, and Qwen-2 in single-query and multi-query settings. We observe that 100% reliability in generating the response in the expected format is usually limited to certain closed-source LLMs, with most open-source LLMs lagging behind (except a few 7B parameters LLMs like Mistral and Phi-3). We conclude that multi-query prompting could be useful to significantly optimize the inference costs in meeting summarization.

Related papers

• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs [78.5013630951288]
  This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs) We first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. We propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers.
  arXiv  Detail & Related papers  (2024-11-04T20:06:34Z)
• LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification [34.9210323553677]
  We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications.
  arXiv  Detail & Related papers  (2024-08-06T15:55:05Z)
• Optimizing LLM Queries in Relational Workloads [58.254894049950366]
  We show how to optimize Large Language Models (LLMs) inference for analytical workloads that invoke LLMs within relational queries. We implement these optimizations in Apache Spark, with vLLM as the model serving backend. We achieve up to 4.4x improvement in end-to-end latency on a benchmark of diverse LLM-based queries on real datasets.
  arXiv  Detail & Related papers  (2024-03-09T07:01:44Z)
• Benchmarking LLMs on the Semantic Overlap Summarization Task [9.656095701778975]
  This paper comprehensively evaluates Large Language Models (LLMs) on the Semantic Overlap Summarization (SOS) task. We report well-established metrics like ROUGE, BERTscore, and SEM-F1$ on two different datasets of alternative narratives.
  arXiv  Detail & Related papers  (2024-02-26T20:33:50Z)
• PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We introduce the PowerPoint Task Completion benchmark to assess the ability of Large Language Models to finish multi-turn, multi-modal instructions. We also propose the PPTX-Match Evaluation System that evaluates if LLMs finish the instruction based on the prediction file rather than the label API sequence. The results show that GPT-4 outperforms other LLMs with 75.1% accuracy in single-turn dialogue testing but faces challenges in completing entire sessions, achieving just 6% session accuracy.
  arXiv  Detail & Related papers  (2023-11-03T08:06:35Z)
• Enhancing In-Context Learning with Answer Feedback for Multi-Span Question Answering [9.158919909909146]
  In this paper, we propose a novel way of employing labeled data such as it informs LLM of some undesired output. Experiments on three multi-span question answering datasets and a keyphrase extraction dataset show that our new prompting strategy consistently improves LLM's in-context learning performance.
  arXiv  Detail & Related papers  (2023-06-07T15:20:24Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.