SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version

• URL: http://arxiv.org/abs/2508.15478v2
• Date: Thu, 04 Sep 2025 11:23:22 GMT
• Title: SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
• Abstract summary: Small Language Models (SLMs) offer computational efficiency and accessibility.<n>We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions.<n>SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs.
• Score: 3.4737901412533634
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.

Related papers

• Latent Performance Profiling of Large Language Models [47.009623327601226]
  We introduce Latent Performance Profiling (LPP) -- a framework that derives task-agnostic diagnostics from hidden activations and output distributions.<n>Unlike static accuracy scores, LPP provides stable, architecture-sensitive signatures across models of similar size.<n>We show that models with similar benchmark scores can exhibit contrasting latent profiles, such as differences in entropy or adaptability.
  arXiv  Detail & Related papers  (2026-05-28T14:41:26Z)
• Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark [38.42021928363628]
  Existing benchmarks face two major challenges: (1) the escalating cost of constructing high-quality test suites and reference solutions, and (2) the increasing risk of data contamination.<n>We propose BIS, a prompt-centric evaluation framework that enables ground-truth-free prediction of LLM performance on code generation tasks.<n>Our framework achieves an average absolute prediction error of 1.1% for code correctness scores, with best- and worst-case errors of 0.3% and 1.9%, respectively.
  arXiv  Detail & Related papers  (2025-08-02T05:34:05Z)
• TMIQ: Quantifying Test and Measurement Domain Intelligence in Large Language Models [0.0]
  We introduce the Test and Measurement Intelligence Quotient (TMIQ), a benchmark designed to quantitatively assess Large Language Models (LLMs)<n>TMIQ offers a comprehensive set of scenarios and metrics for detailed evaluation, including SCPI command matching accuracy, ranked response evaluation, Chain-of-Thought Reasoning (CoT)<n>In testing various LLMs, our findings indicate varying levels of proficiency, with exact SCPI command match accuracy ranging from around 56% to 73%, and ranked matching first-position scores achieving around 33%.
  arXiv  Detail & Related papers  (2025-03-03T23:12:49Z)
• ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning [61.750373974799366]
  ThinkBench is an evaluation framework for large language models (LLMs)<n>It unifies the evaluation of reasoning models and non-reasoning models.<n>ThinkBench effectively provides a reliable evaluation of LLMs and reduces the impact of data contamination.
  arXiv  Detail & Related papers  (2025-02-22T15:41:51Z)
• LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient [19.673388630963807]
  We propose an automated and unbiased evaluation framework, structured around four dimensions and ten criteria.<n>Under this framework, we analyze the advantages and weaknesses of directly prompting large language models (LLMs) as generic benchmark generators.<n>We then introduce a series of methods to address the identified weaknesses and integrate them as BenchMaker.<n>Experiments confirm that BenchMaker achieves superior or comparable performance to human-annotated benchmarks on all metrics.
  arXiv  Detail & Related papers  (2025-02-02T06:36:01Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities.<n>LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands.<n>We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models [71.8065384742686]
  LMMS-EVAL is a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models.<n>LMMS-EVAL LITE is a pruned evaluation toolkit that emphasizes both coverage and efficiency.<n>Multimodal LIVEBENCH utilizes continuously updating news and online forums to assess models' generalization abilities in the wild.
  arXiv  Detail & Related papers  (2024-07-17T17:51:53Z)
• Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production [3.41402911469979]
  Many companies use large language models (LLMs) offered as a service, like OpenAI's GPT-4, to create AI-enabled product experiences. At the same time, a flurry of open-source small language models (SLMs) has been made available for commercial use. This paper presents a systematic evaluation methodology and a characterization of modern open-source SLMs.
  arXiv  Detail & Related papers  (2023-12-20T19:27:59Z)
• Assessing the Reliability of Large Language Model Knowledge [78.38870272050106]
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? We propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs' factual reliability.
  arXiv  Detail & Related papers  (2023-10-15T12:40:30Z)
• Efficient Benchmarking of Language Models [22.696230279151166]
  We present the problem of Efficient Benchmarking, namely, intelligently reducing the costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability.
  arXiv  Detail & Related papers  (2023-08-22T17:59:30Z)
• Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations [111.88727295707454]
  This paper reexamines the research on out-of-distribution (OOD) robustness in the field of NLP. We propose a benchmark construction protocol that ensures clear differentiation and challenging distribution shifts. We conduct experiments on pre-trained language models for analysis and evaluation of OOD robustness.
  arXiv  Detail & Related papers  (2023-06-07T17:47:03Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.