Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

• URL: http://arxiv.org/abs/2603.21389v1
• Date: Sun, 22 Mar 2026 20:19:45 GMT
• Title: Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
• Abstract summary: Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments.<n>This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks.
• Score: 2.067969102829454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks. We introduce the Performance-Efficiency Ratio (PER), a novel metric integrating accuracy, throughput, memory, and latency through geometric mean normalization. Our systematic evaluation reveals that small models (0.5--3B parameters) achieve superior PER scores across all given tasks. These findings establish quantitative foundations for deploying small models in production environments prioritizing inference efficiency over marginal accuracy gains.

Related papers

• Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression [53.39128997308138]
  We introduce information capacity, a measure of model efficiency based on text compression performance.<n> Empirical evaluations on mainstream open-source models show that models of varying sizes within a series exhibit consistent information capacity.<n>A distinctive feature of information capacity is that it incorporates tokenizer efficiency, which affects both input and output token counts.
  arXiv  Detail & Related papers  (2025-11-11T10:07:32Z)
• MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP [0.0]
  We evaluate the performance of 55 publicly available Large Language Models (LLMs) on Maltese, a low-resource language.<n>Our experiments highlight that many models perform poorly, particularly on generative tasks.<n>We conclude that prior exposure to Maltese during pre-training and instruction-tuning emerges as the most important factor.
  arXiv  Detail & Related papers  (2025-06-04T18:59:52Z)
• Ensemble Bayesian Inference: Leveraging Small Language Models to Achieve LLM-level Accuracy in Profile Matching Tasks [0.0]
  This study explores the potential of small language model(SLM) ensembles to achieve accuracy comparable to proprietary large language models (LLMs)<n>We propose Ensemble Bayesian Inference (EBI), a novel approach that applies Bayesian estimation to combine judgments from multiple SLMs.
  arXiv  Detail & Related papers  (2025-04-24T15:55:10Z)
• EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models [64.18350535770357]
  We propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning.<n>Our approach only leverages a small number of samples to search for the desired pruning policy.<n>We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering.
  arXiv  Detail & Related papers  (2025-03-19T16:07:04Z)
• Scaling BERT Models for Turkish Automatic Punctuation and Capitalization Correction [0.0]
  The study presents a systematic comparison of the performance metrics precision, recall, and F1 score of each model.<n>The results demonstrate a significant improvement in text readability and accuracy as model size increases.
  arXiv  Detail & Related papers  (2024-12-03T18:59:51Z)
• InkubaLM: A small language model for low-resource African languages [9.426968756845389]
  InkubaLM is a small language model with 0.4 billion parameters. It achieves performance comparable to models with significantly larger parameter counts. It demonstrates remarkable consistency across multiple languages.
  arXiv  Detail & Related papers  (2024-08-30T05:42:31Z)
• TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale [66.01943465390548]
  We introduce TriSum, a framework for distilling large language models' text summarization abilities into a compact, local model. Our method enhances local model performance on various benchmarks. It also improves interpretability by providing insights into the summarization rationale.
  arXiv  Detail & Related papers  (2024-03-15T14:36:38Z)
• L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models [15.726224465017596]
  We propose an approach that focuses on extracting meaningful representations from unseen data and constructing a structured knowledge base. We conducted experiments on various NLP tasks to validate its effectiveness, including benchmarks like GLUE and SuperGLUE. The proposed L3 ensemble method increases the model accuracy by 4% 36% compared to the fine-tuned FLM.
  arXiv  Detail & Related papers  (2023-11-11T06:59:50Z)
• GPT-Neo for commonsense reasoning -- a theoretical and practical lens [0.46040036610482665]
  We evaluate the performance of the GPT-neo model using $6$ commonsense reasoning benchmark tasks. We aim to examine the performance of smaller models using the GPT-neo models against several larger model baselines.
  arXiv  Detail & Related papers  (2022-11-28T17:49:38Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)
• Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data [66.11139091362078]
  We provide the first model selection results on large pretrained Transformers from Huggingface using generalization metrics. Despite their niche status, we find that metrics derived from the heavy-tail (HT) perspective are particularly useful in NLP tasks.
  arXiv  Detail & Related papers  (2022-02-06T20:07:35Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.