Explanation-Guided Metamorphic Testing of Specialized Language Models: An Empirical Study

• URL: http://arxiv.org/abs/2608.07076v1
• Date: Fri, 07 Aug 2026 10:28:50 GMT
• Title: Explanation-Guided Metamorphic Testing of Specialized Language Models: An Empirical Study
• Abstract summary: Task-specialized language models are increasingly integrated into software engineering to support vertical-domain activities such as issue triaging, document classification, and automated analysis.<n>This paper investigates whether explainability-guided metamorphic testing can improve the effectiveness and validity of robustness testing for specialized language models.
• Score: 0.5368630420272898
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: \head{Background} Task-specialized language models are increasingly integrated into software engineering workflows to support vertical-domain activities such as issue triaging, document classification, and automated analysis. Despite their adoption, there is limited empirical evidence on how to test their robustness and detect brittle behaviors under semantics-preserving input transformations. \head{Aims} This paper investigates whether explainability-guided metamorphic testing can improve the effectiveness and validity of robustness testing for specialized language models compared to heuristic mutation strategies. \head{Method} We conduct a large-scale empirical study of explanation-guided metamorphic testing across three datasets, four model architectures, and 20 testing configurations derived from combinations of attribution methods and mutation strategies. The evaluated configurations combine attribution-based token prioritization, LLM-driven mutation, and automated semantic verification to generate linguistically valid test variants. We assess failure discovery capability, semantic validity, and testing efficiency against heuristic baselines. \head{Results} Explanation-guided metamorphic testing generates 2.30$\times$ more verified failure-inducing test cases than heuristic mutation strategies. Semantic verification substantially improves mutation validity and achieves high label-preservation precision among gate-accepted variants according to human annotation. The study further reveals systematic shortcut behaviors across models, including over-reliance on named entities and formatting cues. \head{Conclusions} The results provide evidence that explanation-guided metamorphic testing is an effective and practical approach for empirically evaluating the robustness of task-specialized language models used in vertical AI applications.

Related papers

• Rethinking Entropy Minimization in Test-Time Adaptation for Autoregressive Models [52.18157855348584]
  Test-Time Adaptation (TTA) via entropy minimization (EM) has proven effective for classification tasks, yet its application remains theoretically fragmented.<n>In this work, we deriving a rigorous formulation of EM tailored to autoregressive models.<n>We show that the exact objective naturally decomposes into a token-level policy gradient loss and a token-level entropy loss, and we reinterpret prior methods as partial realizations of this unified formulation.<n>Using Whisper ASR as a testbed, we demonstrate that our approach consistently improves performance across more than 20 diverse domains, including acoustic noise, accents, and multilingual settings.
  arXiv  Detail & Related papers  (2026-05-05T12:00:06Z)
• FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detection [63.91351553178842]
  FACTOR is a training-free Test-time adaptation for Open-vocabulary object detection.<n>It quantifies attribute sensitivity, semantic relevance, and prediction variation to selectively suppress attribute-dependent predictions.<n>Experiments on PASCAL-C, COCO-C, and FoggyCityscapes show that FACTOR consistently outperforms prior TTA methods.
  arXiv  Detail & Related papers  (2026-05-05T02:31:18Z)
• Feature-Aware Test Generation for Deep Learning Models [0.5368630420272898]
  We introduce Detect, a feature-aware test generation framework for vision-based deep learning (DL) models.<n>It generates inputs by perturbing disentangled semantic attributes within the latent space.<n>It identifies which features lead to behavior shifts and uses a vision-language model for semantic attribution.
  arXiv  Detail & Related papers  (2026-01-20T15:41:06Z)
• Autoformalizer with Tool Feedback [52.334957386319864]
  Autoformalization addresses the scarcity of data for Automated Theorem Proving (ATP) by translating mathematical problems from natural language into formal statements.<n>Existing formalizer still struggles to consistently generate valid statements that meet syntactic validity and semantic consistency.<n>We propose the Autoformalizer with Tool Feedback (ATF), a novel approach that incorporates syntactic and consistency information as tools into the formalization process.
  arXiv  Detail & Related papers  (2025-10-08T10:25:12Z)
• Test Case Generation from Bug Reports via Large Language Models: A Cognitive Layered Evaluation Framework [10.919459368597295]
  We present a systematic evaluation of Large Language Models (LLMs) reasoning in test case generation.<n>We evaluate StarCoder and GPT-4o on Defects4J, GHRB, and mutated variants that introduce linguistic and semantic challenges.
  arXiv  Detail & Related papers  (2025-10-06T20:47:12Z)
• Generating Highly Structured Test Inputs Leveraging Constraint-Guided Graph Refinement [4.121384394709256]
  This study investigates whether test inputs for structured domains can be unified through a graph-based representation.<n>We will evaluate the effectiveness of this approach in enhancing input validity and semantic preservation across eight AI systems.
  arXiv  Detail & Related papers  (2025-07-28T18:54:04Z)
• Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing [19.880543046739252]
  Deep learning (DL) frameworks are essential to DL-based software systems, and framework bugs may lead to substantial disasters.<n>Researchers adopt DL models or single interfaces as test inputs and analyze their execution results to detect bugs.<n> floating-point errors, inherent randomness, and the complexity of test inputs make it challenging to analyze execution results effectively.
  arXiv  Detail & Related papers  (2025-07-06T11:38:14Z)
• Automatically Generating Single-Responsibility Unit Tests [0.0]
  I aim to investigate the effects of providing a pre-process structure to the generated tests.<n>We propose to implement different test representations for evolution and evaluate their impact on coverage, fault detection, and understandability.
  arXiv  Detail & Related papers  (2025-04-08T21:05:04Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMs [54.309127753635366]
  We present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic actions.<n>Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.
  arXiv  Detail & Related papers  (2025-01-28T15:41:54Z)
• Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models [49.06068319380296]
  We introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures.
  arXiv  Detail & Related papers  (2024-10-31T15:06:16Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.