Sustainability Analysis of Prompt Strategies for SLM-based Automated Test Generation

• URL: http://arxiv.org/abs/2604.02761v1
• Date: Fri, 03 Apr 2026 06:10:07 GMT
• Title: Sustainability Analysis of Prompt Strategies for SLM-based Automated Test Generation
• Abstract summary: This paper presents the first systematic sustainability evaluation of prompt engineering strategies for automated test generation using SLMs.<n>We analyze seven prompt strategies across three open-source SLMs under a controlled experimental setup.<n>The results show that prompt strategies have a substantial and independent impact on sustainability outcomes, often outweighing the effect of model choice.
• Score: 0.42970700836450487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The growing adoption of prompt-based automation in software testing raises important issues regarding its computational and environmental sustainability. Existing sustainability studies in AI-driven testing primarily focus on large language models, leaving the impact of prompt engineering strategies largely unexplored - particularly in the context of Small Language Models (SLMs). This gap is critical, as prompt design directly influences inference behavior, execution cost, and resource utilization, even when model size is fixed. To the best of our knowledge, this paper presents the first systematic sustainability evaluation of prompt engineering strategies for automated test generation using SLMs. We analyze seven prompt strategies across three open-source SLMs under a controlled experimental setup. Our evaluation jointly considers execution time, token usage, energy consumption, carbon emissions, and coverage test quality, the latter assessed through coverage analysis of the generated test scripts. The results show that prompt strategies have a substantial and independent impact on sustainability outcomes, often outweighing the effect of model choice. Reasoning intensive strategies such as Chain of Thought and Self-Consistency achieve higher coverage but incur significantly higher execution time, energy consumption, and carbon emissions. In contrast, simpler strategies such as Zero-Shot and ReAct deliver competitive coverage test quality with markedly lower environmental cost, while Least-to-Most and Program of Thought offer balanced trade-offs.

Related papers

• Pen-Strategist: A Reasoning Framework for Penetration Testing Strategy Formation and Analysis [1.0312951426012829]
  Pen-Strategist framework consists of a novel domain-specific reasoning model that derives pentesting strategies via logical reasoning.<n>We fine-tune a Qwen-3-14B model for strategy generation using reinforcement learning.<n>Experiments on the CTFKnow benchmark show an 18% performance gain over the base model.
  arXiv  Detail & Related papers  (2026-05-06T05:02:40Z)
• Evaluating the Environmental Impact of using SLMs and Prompt Engineering for Code Generation [2.2165845825140695]
  This paper presents the first systematic empirical study investigating how different prompt engineering strategies impact code generation accuracy alongside sustainability factors.<n>We evaluate six prominent prompting strategies across 11 open-source models (ranging from 1B to 34B parameters) using the HumanEval+ and MBPP+ benchmarks.<n>Our findings indicate that Chain-of-Thought, being a simpler prompting technique, can provide a near-optimal balance between reasoning capability and energy efficiency.
  arXiv  Detail & Related papers  (2026-04-03T06:37:51Z)
• An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models [0.42970700836450487]
  This study empirically examines the environmental and performance tradeoffs of small language models (SLMs) during prompt-driven unit-test script generation.<n>Our results show that different SLMs exhibit distinct sustainability profiles - some favor lower energy use and faster execution, while others maintain higher stability or coverage under comparable conditions.
  arXiv  Detail & Related papers  (2026-04-03T06:00:15Z)
• Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments [15.538910160052964]
  This paper introduces Strategic Tactical Agent Reasoning (STAR) Benchmark, a multi-agent evaluation framework.<n>STAR supports both turn-based and real-time settings, enabling controlled analysis of long-horizon strategic planning.<n>We show that strategic intelligence in interactive environments depends not only on reasoning depth, but also on the ability to translate plans into timely actions.
  arXiv  Detail & Related papers  (2026-03-10T08:14:13Z)
• Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance [86.46794021499511]
  We show a previously underexplored gap between strategy usage and strategy executability.<n>We propose Selective Strategy Retrieval (SSR), a test-time framework that explicitly models executability.<n> SSR yields reliable and consistent improvements over direct solving, in-context learning, and single-source guidance.
  arXiv  Detail & Related papers  (2026-02-26T03:34:23Z)
• DeCEAT: Decoding Carbon Emissions for AI-driven Software Testing [0.42970700836450487]
  This work introduces the DeCEAT framework, which systematically evaluates the environmental and performance trade-offs of small language models (SLMs)<n>Our results show that different SLMs exhibit distinct sustainability strengths, while others maintain higher stability or accuracy under carbon constraints.<n>This work provides a focused sustainability evaluation framework specifically tailored to automated SLM-based test generation.
  arXiv  Detail & Related papers  (2026-02-20T05:54:58Z)
• EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies [61.267836443398124]
  We introduce EcoGym, a benchmark for continuous plan-and-execute decision making in interactive economies.<n>EcoGym is released as an open, testbed for transparent long-horizonity agent evaluation and for studying controllability-utility trade-offs in realistic economic settings.
  arXiv  Detail & Related papers  (2026-02-10T08:12:23Z)
• Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory [79.63672515243765]
  In this paper, we focus on a standard and realistic scaling setting: majority voting.<n>We show that as the sampling time and computational overhead increase, complicated prompting strategies with superior initial performance gradually fall behind simple Chain-of-Thought.<n>We propose a probabilistic method to efficiently predict scaling performance and identify the best prompting strategy under large sampling times.
  arXiv  Detail & Related papers  (2025-05-16T08:28:57Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation [52.76508734756661]
  Auto-PRE is an automatic evaluation framework inspired by the peer review process.<n>Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluators based on three core traits.<n> Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-10-16T06:06:06Z)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models [81.27391252152199]
  Large language models (LLMs) have achieved impressive performance across various natural language benchmarks. We propose to automate dataset updating and provide systematic analysis regarding its effectiveness. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, and 2) extending strategy that further expands existing samples.
  arXiv  Detail & Related papers  (2024-02-19T07:15:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.