An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models

• URL: http://arxiv.org/abs/2604.02754v1
• Date: Fri, 03 Apr 2026 06:00:15 GMT
• Title: An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models
• Abstract summary: This study empirically examines the environmental and performance tradeoffs of small language models (SLMs) during prompt-driven unit-test script generation.<n>Our results show that different SLMs exhibit distinct sustainability profiles - some favor lower energy use and faster execution, while others maintain higher stability or coverage under comparable conditions.
• Score: 0.42970700836450487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increasing use of language models in automated test script generation raises concerns about their environmental impact, yet existing sustainability analyses focus predominantly on large language models. As a result, the energy and carbon characteristics of small language models (SLMs) during prompt-driven unit-test script generation remain largely unexplored. To address this gap, this study empirically examines the environmental and performance tradeoffs of SLMs (in the 2B-8B parameter range) using the HumanEval benchmark and adaptive prompt variants (based on the Anthropic template). The analysis uses CodeCarbon to characterize energy consumption carbon emissions and duration under controlled conditions, with unit-test script coverage serving as an initial proxy for generated test quality. Our results show that different SLMs exhibit distinct sustainability profiles - some favor lower energy use and faster execution, while others maintain higher stability or coverage under comparable conditions. Overall, this work provides focused empirical evidence on sustainable SLM-based test script generation, clarifying how prompt structure and model selection jointly shape environmental and performance outcomes.

Related papers

• Sparsity as a Key: Unlocking New Insights from Latent Structures for Out-of-Distribution Detection [4.496267803613718]
  We present the first application of SAEs to the ViT [] token for out-of-distribution (OOD) detection.<n>Our findings demonstrate that the sparse, disentangled features revealed by SAEs can serve as a powerful, interpretable tool for robust OOD detection in vision models.
  arXiv  Detail & Related papers  (2026-04-29T08:23:38Z)
• Sustainability Analysis of Prompt Strategies for SLM-based Automated Test Generation [0.42970700836450487]
  This paper presents the first systematic sustainability evaluation of prompt engineering strategies for automated test generation using SLMs.<n>We analyze seven prompt strategies across three open-source SLMs under a controlled experimental setup.<n>The results show that prompt strategies have a substantial and independent impact on sustainability outcomes, often outweighing the effect of model choice.
  arXiv  Detail & Related papers  (2026-04-03T06:10:07Z)
• DeCEAT: Decoding Carbon Emissions for AI-driven Software Testing [0.42970700836450487]
  This work introduces the DeCEAT framework, which systematically evaluates the environmental and performance trade-offs of small language models (SLMs)<n>Our results show that different SLMs exhibit distinct sustainability strengths, while others maintain higher stability or accuracy under carbon constraints.<n>This work provides a focused sustainability evaluation framework specifically tailored to automated SLM-based test generation.
  arXiv  Detail & Related papers  (2026-02-20T05:54:58Z)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition [72.24180896265192]
  We introduce AgentNoiseBench, a framework for evaluating robustness of agentic models under noisy environments.<n>We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios.<n>We then categorize environmental noise into two primary types: user-noise and tool-noise.<n>Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T20:33:10Z)
• Emissions and Performance Trade-off Between Small and Large Language Models [1.0863226323853896]
  This study investigates the potential of using fine-tuned Small Language Models (SLMs) as a sustainable alternative for predefined tasks.<n>Our results show that in four out of the six selected tasks, SLMs maintained comparable performances for a significant reduction in carbon emissions during inference.
  arXiv  Detail & Related papers  (2025-12-21T07:00:22Z)
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs [49.66344956133349]
  Reasoning capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models.<n>This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a latent variable for strategic contextualization.
  arXiv  Detail & Related papers  (2025-12-19T03:32:53Z)
• BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models [1.5637023740732419]
  Sentiment Analysis (SA) models harbor inherent social biases that can be harmful in real-world applications.<n>We present a novel bias testing framework, BTC-SAM, which generates high-quality test cases for bias testing in SA models with minimal specification.
  arXiv  Detail & Related papers  (2025-09-28T22:39:40Z)
• A Controllable Examination for Long-Context Language Models [62.845852724511964]
  This study introduces $textbfLongBioBench, a benchmark for evaluating long-context language models.<n>We show that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results.<n>Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence.
  arXiv  Detail & Related papers  (2025-06-03T14:23:06Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain [0.0]
  This paper examines the performance of two Large Language Models (LLMs), GPT3.5 and Llama2 and one Small Language Model (SLM) Gemma, across three different classification tasks within the climate change (CC) and environmental domain.
  arXiv  Detail & Related papers  (2024-08-30T15:52:41Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.