Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework

• URL: http://arxiv.org/abs/2606.28618v1
• Date: Fri, 26 Jun 2026 21:34:33 GMT
• Title: Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework
• Abstract summary: Developers frequently reuse code by copying fragments and adapting them to fit new contexts.<n>Existing benchmarks for evaluating large language models (LLMs) on code adaptation either rely on explicit step-by-step instructions, cover only narrow change types such as variable wiring, or operate exclusively at function-level granularity.<n>We investigate instruction-free code snippet adaptation in which an LLM must adapt a code fragment to fit its target context without any explicit edit guidance.
• Score: 9.514883757685725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Developers frequently reuse code by copying fragments and adapting them to fit new contexts. Existing benchmarks for evaluating large language models (LLMs) on code adaptation either rely on explicit step-by-step instructions, cover only narrow change types such as variable wiring, or operate exclusively at function-level granularity. It remains unknown how well LLMs can adapt code fragments without explicit edit guidance when the required changes are varied and controlled. Objective: We investigate instruction-free code snippet adaptation in which an LLM must adapt a code fragment to fit its target context without any explicit edit guidance. We study three dimensions: which adaptation types are hardest (RQ1), how performance scales with adaptation complexity (RQ2), and how much surrounding context the model needs (RQ3). Method: We will construct a dataset of Java code fragments from open-source repositories with strong test coverage and apply a taxonomy of adaptation operators, derived from empirical findings on how developers adapt copied code, using a mutation-injection framework. Working at the code fragment level and controlling the injected changes lets us know exactly what adaptations the model must perform. The unmutated fragment serves as a plausible reference for the changes the model needs to make. LLMs will be evaluated on instruction-free adaptation tasks across three context granularity levels. Correctness will be measured primarily via test-suite re-insertion, complemented by mutation-level inspection.

Related papers

• AdaptAgent: A Multi-agent, Domain-Guided Reasoning Framework for Code Adaptation [14.734454356396157]
  We present AdaptAgent, a multi-agent, domain-guided reasoning framework for code adaptation.<n>Rather than relying on single-shot prompting, AdaptAgent decomposes adaptation into specialized agents that communicate via typed artifacts.<n>On a real-world dataset, AdaptAgent outperforms strong baselines in semantic correctness and produces patches that mirror developers' actual adaptation patterns.
  arXiv  Detail & Related papers  (2026-08-05T05:34:40Z)
• Feedback-based Automated Verification in Vibe Coding of CAS Adaptation Built on Constraint Logic [0.0]
  In CAS adaptation, a challenge is to define the dynamic architecture of the system and changes in its behavior.<n>With the advances of generative LLMs, generating AM code based on system specification and desired AM behavior is a tempting opportunity.<n>We show that generating an AM via vibe coding feedback loops is a viable option when the verification of the generated AM is based on a very precise formulation of the functional requirements.
  arXiv  Detail & Related papers  (2026-02-20T20:49:12Z)
• Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptation [16.969255848886693]
  A key challenge is to resolve Context Adaptation Bugs (CtxBugs)<n>Unlike isolated bugs, CtxBugs cannot be resolved through local fixes and require cross-context reasoning.<n>Although Large Language Models (LLMs) show great potential in automating code-related tasks, their ability to resolve CtxBugs remains a significant and unexplored obstacle to their practical use in code adaptation.
  arXiv  Detail & Related papers  (2026-01-10T09:14:00Z)
• AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation [22.341339973062702]
  AdaptEval is a benchmark designed to evaluate large language models (LLMs) on code snippet adaptation.<n>Each task is annotated with requirements at both task and adaptation levels, supporting the evaluation of LLMs across diverse adaptation scenarios.<n>We conduct the first empirical study to evaluate six instruction-tuned LLMs and especially three reasoning LLMs on code snippet adaptation.
  arXiv  Detail & Related papers  (2026-01-08T03:13:20Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification [10.964060011243234]
  This paper presents a study of using large language models (LLMs) in modifying existing code.<n>"prompting" serves as the primary interface for developers to communicate intents to LLMs.<n>This study investigates two prompting strategies for LLM-assisted code modification.
  arXiv  Detail & Related papers  (2025-08-02T23:52:49Z)
• Joint Localization and Activation Editing for Low-Resource Fine-Tuning [73.64004083269424]
  We propose a joint localization and activation editing (JoLA) method.<n>JoLA learns (1) which heads in the Transformer to edit (2) whether the intervention should be additive, multiplicative, or both and (3) the intervention parameters themselves.<n>We demonstrate that JoLA consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2025-02-03T09:13:09Z)
• Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering [19.019004855931676]
  Large language models (LLMs) have confirmed their effectiveness in the code generation task with promising results. Their performance on adaptation, a reuse-oriented and context-dependent code change prediction task, is still unclear. We propose an interactive prompting approach to elicit LLMs' adaptation ability.
  arXiv  Detail & Related papers  (2024-11-23T09:40:36Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning [101.81127587760831]
  Current fine-tuning methods build adapters widely of the context of downstream task to learn, or the context of important knowledge to maintain.<n>We propose CorDA, a Context-oriented Decomposition Adaptation method that builds learnable task-aware adapters.<n>Our method enables two options, the knowledge-preserved adaptation and the instruction-previewed adaptation.
  arXiv  Detail & Related papers  (2024-06-07T19:10:35Z)
• Towards Modular LLMs by Building and Reusing a Library of LoRAs [64.43376695346538]
  We study how to best build a library of adapters given multi-task data. We introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters.
  arXiv  Detail & Related papers  (2024-05-18T03:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.