Performance Law of Large Language Models

• URL: http://arxiv.org/abs/2408.09895v4
• Date: Fri, 13 Sep 2024 12:28:45 GMT
• Title: Performance Law of Large Language Models
• Authors: Chuhan Wu, Ruiming Tang,
• Abstract summary: Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources. Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources without extensive experiments.
• Score: 58.32539851241063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Guided by the belief of the scaling law, large language models (LLMs) have achieved impressive performance in recent years. However, scaling law only gives a qualitative estimation of loss, which is influenced by various factors such as model architectures, data distributions, tokenizers, and computation precision. Thus, estimating the real performance of LLMs with different training settings rather than loss may be quite useful in practical development. In this article, we present an empirical equation named "Performance Law" to directly predict the MMLU score of an LLM, which is a widely used metric to indicate the general capability of LLMs in real-world conversations and applications. Based on only a few key hyperparameters of the LLM architecture and the size of training data, we obtain a quite accurate MMLU prediction of various LLMs with diverse sizes and architectures developed by different organizations in different years. Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources without extensive experiments.

Related papers

• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families [43.36524246307057]
  Scaling laws for large language models (LLMs) predict performance based on parameters like size and training data. We propose Skills Scaling Laws (SSLaws), a novel scaling law that leverages publicly available benchmark data. We present both theoretical results on parameter identification and empirical evaluations on 12 prominent benchmarks.
  arXiv  Detail & Related papers  (2024-12-09T14:51:26Z)
• Densing Law of LLMs [81.06644243978101]
  Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases. This paper introduces the concept of textitcapacity density'' as a new metric to evaluate the quality of the LLMs across different scales.
  arXiv  Detail & Related papers  (2024-12-05T16:31:13Z)
• LLaVA-KD: A Framework of Distilling Multimodal Large Language Models [70.19607283302712]
  We propose a novel framework to transfer knowledge from l-MLLM to s-MLLM. Specifically, we introduce Multimodal Distillation (MDist) to minimize the divergence between the visual-textual output distributions of l-MLLM and s-MLLM. We also propose a three-stage training scheme to fully exploit the potential of s-MLLM.
  arXiv  Detail & Related papers  (2024-10-21T17:41:28Z)
• Achieving Peak Performance for Large Language Models: A Systematic Review [0.0]
  Large language models (LLMs) have achieved remarkable success in natural language processing (NLP) As models grow into the trillion- parameter range, computational and memory costs increase significantly. This makes it difficult for many researchers to access the resources needed to train or apply these models.
  arXiv  Detail & Related papers  (2024-09-07T13:57:41Z)
• Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices [32.61693246340064]
  We study how a resource-constrained computing environment would affect the design choices for a personalized LLM. We consider the tradeoffs among a number of key design factors and their intertwined impacts on learning efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-06-06T06:41:53Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• A Comprehensive Evaluation of Quantization Strategies for Large Language Models [42.03804933928227]
  Increasing the number of parameters in large language models (LLMs) usually improves performance in downstream tasks but raises compute and memory costs. Quantization techniques, which reduce the bits needed for model weights or activations with minimal performance loss, have become popular. We propose a structured evaluation framework consisting of three critical dimensions: knowledge & capacity, (2) alignment, and (3) efficiency.
  arXiv  Detail & Related papers  (2024-02-26T17:45:36Z)
• Knowledge Fusion of Large Language Models [73.28202188100646]
  This paper introduces the notion of knowledge fusion for large language models (LLMs) We externalize their collective knowledge and unique strengths, thereby elevating the capabilities of the target model beyond those of any individual source LLM. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation.
  arXiv  Detail & Related papers  (2024-01-19T05:02:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.