Examining LLMs Ability to Summarize Code Through Mutation-Analysis

• URL: http://arxiv.org/abs/2602.17838v1
• Date: Thu, 19 Feb 2026 21:00:49 GMT
• Title: Examining LLMs Ability to Summarize Code Through Mutation-Analysis
• Abstract summary: We present a mutation-based evaluation methodology that directly tests whether a summary truly matches the code's logic.<n>We validate it through three experiments totalling 624 mutation-summary evaluations across 62 programs.<n>We find that summary accuracy decreases sharply with complexity, while mutation type and location exhibit weaker effects.
• Score: 9.187740569361804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As developers increasingly rely on LLM-generated code summaries for documentation, testing, and review, it is important to study whether these summaries accurately reflect what the program actually does. LLMs often produce confident descriptions of what the code looks like it should do (intent), while missing subtle edge cases or logic changes that define what it actually does (behavior). We present a mutation-based evaluation methodology that directly tests whether a summary truly matches the code's logic. Our approach generates a summary, injects a targeted mutation into the code, and checks if the LLM updates its summary to reflect the new behavior. We validate it through three experiments totalling 624 mutation-summary evaluations across 62 programs. First, on 12 controlled synthetic programs with 324 mutations varying in type (statement, value, decision) and location (beginning, middle, end). We find that summary accuracy decreases sharply with complexity from 76.5% for single functions to 17.3% for multi-threaded systems, while mutation type and location exhibit weaker effects. Second, testing 150 mutated samples on 50 human-written programs from the Less Basic Python Problems (LBPP) dataset confirms the same failure patterns persist as models often describe algorithmic intent rather than actual mutated behavior with a summary accuracy rate of 49.3%. Furthermore, while a comparison between GPT-4 and GPT-5.2 shows a substantial performance leap (from 49.3% to 85.3%) and an improved ability to identify mutations as "bugs", both models continue to struggle with distinguishing implementation details from standard algorithmic patterns. This work establishes mutation analysis as a systematic approach for assessing whether LLM-generated summaries reflect program behavior rather than superficial textual patterns.

Related papers

• Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations [0.0]
  Large language models (LLMs) are increasingly being explored for automated SystemVerilog Assertion (SVA) generation.<n>Most evaluations report correctness on a single syntactic representation of an input.<n>Such point accuracy does not reveal whether a model's correct output is stable when the same RTL behavior is written differently.
  arXiv  Detail & Related papers  (2026-09-04T18:41:58Z)
• Regression Accumulation in Multi-Turn LLM Programming Conversations [25.94870617800846]
  regression accumulation appears across all six models.<n>Cross-Turn Conflict, where later code conflicts with earlier requirements, is the main failure class.<n> Verification Gate is the only strategy that consistently improves all models.
  arXiv  Detail & Related papers  (2026-07-02T08:15:40Z)
• Evaluating the Effectiveness of LLMs in Aiding Compliance Testing of PKCS#1-v1.5 [0.0]
  Testing implementations of binary protocols for specification compliance requires inputs that satisfy both structural and semantic constraints.<n>This work investigates whether grammar-level mutation combined with LLM-based code synthesis can serve as a viable, more generalizable approach to specification compliance testing.
  arXiv  Detail & Related papers  (2026-06-16T18:54:57Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• Boosting LLMs for Mutation Generation [35.905252475438466]
  We introduce SMART (Semantic Mutation with Adaptive Retrieval and Tuning)<n>We conduct an empirical study of SMART using 1,991 real-world Java bugs from Defects4J and ConDefects datasets.<n>The results reveal that SMART substantially improves mutation validity, effectiveness, and efficiency.
  arXiv  Detail & Related papers  (2026-03-25T17:42:17Z)
• From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents [51.12380889298618]
  POISE is a framework for automated discovery of policy optimization algorithms for language models.<n>PoISE maintains a structured, genealogically linked archive linking proposals, executable implementations, standardized evaluations, and natural-language reflections.
  arXiv  Detail & Related papers  (2026-03-25T05:19:23Z)
• ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization [6.572539312871392]
  Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk.<n>We introduce ReLoop, addressing silent failures from two complementary directions.
  arXiv  Detail & Related papers  (2026-02-17T20:20:33Z)
• Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique [1.5749416770494706]
  Large language models (LLMs) allow for sophisticated qualitative coding of large datasets.<n>We present a simple, generalizable two-stage workflow: an LLM applies a human-designed codebook; a secondary LLM critic performs self-reflection on each positive label.<n>We evaluate this approach on six qualitative codes over 3,000 high-content emails from Apache Software Foundation project evaluation discussions.
  arXiv  Detail & Related papers  (2026-01-14T22:27:13Z)
• Assertion-Aware Test Code Summarization with Large Language Models [0.0]
  Unit tests often lack concise summaries that convey test intent.<n>This paper presents a new benchmark of 91 real-world Java test cases paired with developer-written summaries.
  arXiv  Detail & Related papers  (2025-11-09T04:58:32Z)
• Understanding Robustness of Model Editing in Code LLMs: An Empirical Study [1.5624785508022727]
  We present a systematic study of five state-of-the-art model editing methods.<n>We apply these methods to three leading open-source code LLMs, CodeLlama, CodeQwen1.5, and DeepSeek-Coder.<n>Instant edits consistently degrade model performance, with syntactic validity dropping by up to 86 percentage points and functional correctness declining by 45 points even in the best-performing setting.
  arXiv  Detail & Related papers  (2025-11-05T04:58:13Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations? [18.52519530244078]
  We evaluate whether state-of-the-art Large Language Models with up to 8B parameters can reason about Python programs or are simply guessing.<n>We apply five semantics-preserving code mutations: renaming variables, mirroring comparison expressions, swapping if-else branches, converting for loops to while, and loop unrolling.<n>Our findings show that LLMs trained for code produce correct predictions based on flawed reasoning between 10% and 50% of cases.
  arXiv  Detail & Related papers  (2025-05-15T16:04:25Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• A Comprehensive Study on Large Language Models for Mutation Testing [36.00296047226433]
  Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice.<n>We evaluate BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches) on 851 real bugs from two Java real-world bug benchmarks.<n>Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection.
  arXiv  Detail & Related papers  (2024-06-14T08:49:41Z)
• Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs [5.549095839198671]
  Large Language Models (LLMs) have shown remarkable capabilities in processing both natural and programming languages. We propose a novel method to assess the code understanding performance of LLMs, particularly focusing on subtle differences between code and its descriptions. We apply different types of code mutations, such as operator replacement and statement deletion, to generate inconsistent code-description pairs. We conduct a case study on the two popular LLMs, GPT-3.5 and GPT-4, using the state-of-the-art code generation benchmark, HumanEval-X.
  arXiv  Detail & Related papers  (2024-01-11T14:27:43Z)
• See, Say, and Segment: Teaching LMMs to Overcome False Premises [67.36381001664635]
  We propose a cascading and joint training approach for LMMs to solve this task. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, and finally "segment" by outputting the mask of the desired objects if they exist.
  arXiv  Detail & Related papers  (2023-12-13T18:58:04Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Evaluating the Factual Consistency of Large Language Models Through News Summarization [97.04685401448499]
  We propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. For factually inconsistent summaries, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.
  arXiv  Detail & Related papers  (2022-11-15T18:50:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.