Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study

• URL: http://arxiv.org/abs/2607.00511v1
• Date: Wed, 01 Jul 2026 06:46:13 GMT
• Title: Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study
• Abstract summary: Large language models (LLMs) have recently demonstrated remarkable performance across diverse code-related tasks by better capturing program semantics.<n>This paper presents the first comprehensive empirical study on LLMs for, using 3, Java and 1,088 C mutant pairs to benchmark against state-of-the-art methods.<n>LLMs achieve higher F1-scores than the evaluated traditional methods, with fine-tuned code embedding yielding the highest detection accuracy.
• Score: 8.372242332246508
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mutation testing is a powerful technique for ensuring software quality. However, the presence of equivalent mutants introduces unnecessary costs and biases, limiting its practical effectiveness. Although numerous equivalent mutant detection (EMD) methods have been proposed, they often face distinct challenges: pure-code analysis methods can be limited by their reliance on specific compiler infrastructures, while existing machine-learning approaches remain constrained by scarce training data and limited generalization to unseen mutants. Large language models (LLMs) have recently demonstrated remarkable performance across diverse code-related tasks by better capturing program semantics. Yet their potential for EMD remains largely unexplored, particularly in the multi-lingual context. This paper presents the first comprehensive empirical study on LLMs for EMD, using 3,302 Java and 1,088 C mutant pairs to benchmark against state-of-the-art methods, explore strategy variations, assess efficiency, and evaluate cross-lingual generalization. Experimental results show that LLM-based approaches achieve higher F1-scores than the evaluated traditional methods, with fine-tuned code embedding yielding the highest detection accuracy among the tested strategies. Moreover, LLM-based approaches strike a practical balance between effectiveness and efficiency with inference times comparable to existing machine-learning models. Importantly, fine-tuned LLMs demonstrate measurable generalization across programming languages. These findings establish LLMs as a viable and efficient approach for tackling the longstanding challenge of equivalent mutant detection, offering new directions for advancing mutation testing in practice.

Related papers

• An Extensive Empirical Study on Code Translation Technique [3.398993044674732]
  We compare representative code translation techniques across methodological paradigms and translation granularities.<n>Our analysis considers executable correctness, code similarity, translation direction, translation, and failure patterns.<n>These findings provide empirical evidence and practical guidance for developing more robust, type-aware, structure-aware, and context-aware code translation techniques.
  arXiv  Detail & Related papers  (2026-08-21T06:41:04Z)
• Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters [56.051481747681315]
  Cross-modal skill injection aims to induce emergent cross-modal capabilities by integrating a domain-expert LLM into a Vision-Language Models (VLMs)<n>For scenarios, we find that cross-modal skill injection generally performs well in instruction-following and cross-lingual settings, yet struggles with mathematical reasoning.<n>For methods, we find that classic approaches such as TA and DARE consistently achieve superior performance over alternative merging methods.
  arXiv  Detail & Related papers  (2026-05-19T08:24:19Z)
• Robust Uncertainty Quantification for Factual Generation of Large Language Models [22.060021788289202]
  Large language model(LLM) technology has facilitated its integration into various domains of professional and daily life.<n>The persistent challenge of LLM hallucination has emerged as a critical limitation, significantly compromising the reliability and trustworthiness of AI-generated content.<n>This study proposes an uncertainty quantification scenario in the task of generating with multiple facts.
  arXiv  Detail & Related papers  (2026-01-01T14:06:58Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks [0.0]
  We evaluate the performance of Poly-Coder, a pioneering open-source, multilingual CLM built for code generation. Our results suggest that the outcomes observed in these translated benchmarks align well with evaluation metrics used during the training phase. These initial insights highlight the need for more comprehensive empirical studies.
  arXiv  Detail & Related papers  (2024-11-23T06:40:47Z)
• Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models [104.55763564037831]
  We train a regression model that leverages attention maps, probabilities on the current generation step, and recurrently computed uncertainty scores from previously generated tokens.<n>Our evaluation shows that the proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.
  arXiv  Detail & Related papers  (2024-08-20T09:42:26Z)
• FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models [50.331708897857574]
  We introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications. FactorLLM achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed.
  arXiv  Detail & Related papers  (2024-08-15T16:45:16Z)
• Large Language Models for Equivalent Mutant Detection: How Far Are We? [9.126998558502914]
  We conduct an empirical study on 3,302 method-level Java mutant pairs to investigate the effectiveness and efficiency of large language models (LLMs) for equivalent mutant detection. Our findings demonstrate that LLM-based techniques significantly outperform existing techniques, with the fine-tuned code embedding strategy being the most effective.
  arXiv  Detail & Related papers  (2024-08-03T11:58:16Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• A Comprehensive Study on Large Language Models for Mutation Testing [36.00296047226433]
  Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice.<n>We evaluate BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches) on 851 real bugs from two Java real-world bug benchmarks.<n>Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection.
  arXiv  Detail & Related papers  (2024-06-14T08:49:41Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement [69.51496713076253]
  In this paper, we focus on the aforementioned efficiency aspects of existing MTL methods. We first carry out large-scale experiments of the methods with smaller backbones and on a the MetaGraspNet dataset as a new test ground. We also propose Feature Disentanglement measure as a novel and efficient identifier of the challenges in MTL.
  arXiv  Detail & Related papers  (2024-02-05T22:15:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.