Position: Enough of Scaling LLMs! Lets Focus on Downscaling

• URL: http://arxiv.org/abs/2505.00985v3
• Date: Sun, 25 May 2025 14:53:34 GMT
• Title: Position: Enough of Scaling LLMs! Lets Focus on Downscaling
• Authors: Yash Goel, Ayan Sengupta, Tanmoy Chakraborty,
• Abstract summary: We advocate for a paradigm shift toward downscaling in the development of large language models (LLMs)<n>We propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands.<n>This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.
• Score: 20.62274005080048
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.

Related papers

• Revisiting LLM Reasoning via Information Bottleneck [57.519119962528166]
  Large language models (LLMs) have recently demonstrated remarkable progress in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR)<n>We present a theoretical characterization of LLM reasoning grounded in information bottleneck (IB) principle.<n>We propose IB-aware reasoning optimization (IBRO), a framework that encourages reasoning trajectories to be both informative about the final correct answer and generalizable.
  arXiv  Detail & Related papers  (2025-07-24T13:14:25Z)
• A Call for New Recipes to Enhance Spatial Reasoning in MLLMs [85.67171333213301]
  Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in general vision-language tasks.<n>Recent studies have exposed critical limitations in their spatial reasoning capabilities.<n>This deficiency in spatial reasoning significantly constrains MLLMs' ability to interact effectively with the physical world.
  arXiv  Detail & Related papers  (2025-04-21T11:48:39Z)
• A Survey of Scaling in Large Language Model Reasoning [62.92861523305361]
  We provide a comprehensive examination of scaling in large Language models (LLMs) reasoning.<n>We analyze scaling in reasoning steps that improves multi-step inference and logical consistency.<n>We discuss scaling in training-enabled reasoning, focusing on optimization through iterative model improvement.
  arXiv  Detail & Related papers  (2025-04-02T23:51:27Z)
• A Survey on Post-training of Large Language Models [185.51013463503946]
  Large Language Models (LLMs) have fundamentally transformed natural language processing, making them indispensable across domains ranging from conversational systems to scientific exploration.<n>These challenges necessitate advanced post-training language models (PoLMs) to address shortcomings, such as restricted reasoning capacities, ethical uncertainties, and suboptimal domain-specific performance.<n>This paper presents the first comprehensive survey of PoLMs, systematically tracing their evolution across five core paradigms: Fine-tuning, which enhances task-specific accuracy; Alignment, which ensures ethical coherence and alignment with human preferences; Reasoning, which advances multi-step inference despite challenges in reward design; Integration and Adaptation, which
  arXiv  Detail & Related papers  (2025-03-08T05:41:42Z)
• LLM Post-Training: A Deep Dive into Reasoning Large Language Models [131.10969986056]
  Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications.<n>Post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations.
  arXiv  Detail & Related papers  (2025-02-28T18:59:54Z)
• Has LLM Reached the Scaling Ceiling Yet? Unified Insights into LLM Regularities and Constraints [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their scalability raises a critical question: Have we reached the scaling ceiling?<n>This paper develops a unified theoretical framework that integrates mathematical and statistical insights to explain the scaling dynamics of LLMs.<n>Future progress will require a shift from brute-force scaling to innovations in architecture, data quality, and training paradigms.
  arXiv  Detail & Related papers  (2024-12-21T02:19:07Z)
• LLMs are Also Effective Embedding Models: An In-depth Overview [40.53941563464671]
  Large language models (LLMs) have revolutionized natural language processing by achieving state-of-the-art performance across various tasks.<n>Recently, their effectiveness as embedding models has gained attention, marking a paradigm shift from traditional encoder-only models like ELMo and BERT to decoder-only, large-scale LLMs like GPT, LLaMA, and Mistral.
  arXiv  Detail & Related papers  (2024-12-17T06:48:24Z)
• Densing Law of LLMs [81.06644243978101]
  Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases.<n>This paper introduces the concept of textitcapacity density'' as a new metric to evaluate the quality of the LLMs across different scales.
  arXiv  Detail & Related papers  (2024-12-05T16:31:13Z)
• Designing Large Foundation Models for Efficient Training and Inference: A Survey [35.40505841618305]
  This paper focuses on modern efficient training and inference technologies on foundation models.<n>Model and System Design optimize LLM training and inference from different aspects to save computational resources.
  arXiv  Detail & Related papers  (2024-09-03T15:35:01Z)
• A Survey on Efficient Inference for Large Language Models [25.572035747669275]
  Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. The substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. This paper presents a comprehensive survey of the existing literature on efficient LLM inference.
  arXiv  Detail & Related papers  (2024-04-22T15:53:08Z)
• Extending Token Computation for LLM Reasoning [5.801044612920816]
  Large Language Models (LLMs) are pivotal in advancing natural language processing. LLMs often struggle with complex reasoning tasks due to inefficient attention distributions. We introduce a novel method for extending computed tokens in the Chain-of-Thought process, utilizing attention mechanism optimization.
  arXiv  Detail & Related papers  (2024-03-22T03:23:58Z)
• SparseLLM: Towards Global Pruning for Pre-trained Language Models [12.057369029549534]
  We propose SparseLLM, a novel framework that redefines the global pruning process into manageable, coordinated subproblems. SparseLLM's approach conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition. It demonstrates significant performance improvements, particularly in high-sparsity regimes.
  arXiv  Detail & Related papers  (2024-02-28T00:09:07Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning [76.3114831562989]
  It requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent environments. We propose a novel framework: "K-Level Reasoning with Large Language Models (K-R)"
  arXiv  Detail & Related papers  (2024-02-02T16:07:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.