QuickLLaMA: Query-aware Inference Acceleration for Large Language Models

• URL: http://arxiv.org/abs/2406.07528v2
• Date: Thu, 22 Aug 2024 06:09:53 GMT
• Title: QuickLLaMA: Query-aware Inference Acceleration for Large Language Models
• Authors: Jingyao Li, Han Shi, Xin Jiang, Zhenguo Li, Hong Xu, Jiaya Jia,
• Abstract summary: We introduce Query-aware Inference for Large Language Models (Q-LLM) Q-LLM is designed to process extensive sequences akin to human cognition. It can accurately capture pertinent information within a fixed window size and provide precise answers to queries.
• Score: 94.82978039567236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The capacity of Large Language Models (LLMs) to comprehend and reason over long contexts is pivotal for advancements in diverse fields. Yet, they still stuggle with capturing long-distance dependencies within sequences to deeply understand semantics. To address this issue, we introduce Query-aware Inference for LLMs (Q-LLM), a system designed to process extensive sequences akin to human cognition. By focusing on memory data relevant to a given query, Q-LLM can accurately capture pertinent information within a fixed window size and provide precise answers to queries. It doesn't require extra training and can be seamlessly integrated with any LLMs. Q-LLM using LLaMA3 (QuickLLaMA) can read Harry Potter within 30s and accurately answer the questions. On widely recognized benchmarks, Q-LLM improved by 7.17% compared to the current state-of-the-art on LLaMA3, and by 3.26% on Mistral on the $\infty$-bench. In the Needle-in-a-Haystack and BABILong task, Q-LLM improved upon the current SOTA by 7.0% and 6.1%. Our code can be found in https://github.com/dvlab-research/Q-LLM.

Related papers

• MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language [7.488965571323756]
  We propose Multi-hop Questioning Answering under Knowledge Editing for Arabic Language (MQA-KEAL) MQA-KEAL stores knowledge edits as structured knowledge units in the external memory. We also contribute MQUAKE-AR (Arabic translation of English benchmark MQUAKE) as well as a new benchmark MQA-AEVAL for rigorous performance evaluation of MQA under KE for Arabic language.
  arXiv  Detail & Related papers  (2024-09-18T18:40:02Z)
• Revisiting Multi-Modal LLM Evaluation [29.094387692681337]
  We pioneer evaluating recent MLLMs (LLaVA 1.5, LLaVA-NeXT, BLIP2, InstructBLIP, GPT-4V, and GPT-4o) on datasets designed to address weaknesses in earlier ones. Our code is integrated into the widely used LAVIS framework for MLLM evaluation, enabling the rapid assessment of future MLLMs.
  arXiv  Detail & Related papers  (2024-08-09T20:55:46Z)
• Make Your LLM Fully Utilize the Context [70.89099306100155]
  We show that FILM-7B can robustly retrieve information from different positions in its 32K context window. FILM-7B significantly improves the performance on real-world long-context tasks.
  arXiv  Detail & Related papers  (2024-04-25T17:55:14Z)
• Porting Large Language Models to Mobile Devices for Question Answering [1.0878040851637998]
  We describe how we managed to port state of the art Large Language Models to mobile devices. We employ the llama framework, a flexible and self-contained C++ framework for LLM inference. Experimental results show that LLM inference runs in interactive speed on a Galaxy S21 smartphone.
  arXiv  Detail & Related papers  (2024-04-24T12:59:54Z)
• Can multiple-choice questions really be useful in detecting the abilities of LLMs? [15.756543037102256]
  Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) The misalignment between the task and the evaluation method demands a thoughtful analysis of MCQ's efficacy. We evaluate nine LLMs on four question-answering (QA) datasets in two languages: Chinese and English.
  arXiv  Detail & Related papers  (2024-03-26T14:43:48Z)
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [58.20031627237889]
  LongBench is the first bilingual, multi-task benchmark for long context understanding. It comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese)
  arXiv  Detail & Related papers  (2023-08-28T11:53:40Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)
• Self-Prompting Large Language Models for Zero-Shot Open-Domain QA [67.08732962244301]
  Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing background documents. This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models. We propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of Large Language Models.
  arXiv  Detail & Related papers  (2022-12-16T18:23:43Z)
• Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models [68.37431984231338]
  Large language models (LLMs) have shown impressive results across a variety of tasks while requiring little or no direct supervision. We believe the ability of an LLM to an attribute to the text that it generates is likely to be crucial for both system developers and users in this setting.
  arXiv  Detail & Related papers  (2022-12-15T18:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.