Automatically Testing Functional Properties of Code Translation Models

• URL: http://arxiv.org/abs/2309.12813v2
• Date: Tue, 30 Jan 2024 04:20:27 GMT
• Title: Automatically Testing Functional Properties of Code Translation Models
• Authors: Hasan Ferit Eniser, Valentin W\"ustholz, Maria Christakis
• Abstract summary: We devise the first approach for automated, functional, property-based testing of code translation models. Our general, user-provided specifications about the transpiled code capture a range of properties. We also develop the first property-guided search procedure for code translation models.
• Score: 3.5757761767474867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are becoming increasingly practical for translating code across programming languages, a process known as $transpiling$. Even though automated transpilation significantly boosts developer productivity, a key concern is whether the generated code is correct. Existing work initially used manually crafted test suites to test the translations of a small corpus of programs; these test suites were later automated. In contrast, we devise the first approach for automated, functional, property-based testing of code translation models. Our general, user-provided specifications about the transpiled code capture a range of properties, from purely syntactic to purely semantic ones. As shown by our experiments, this approach is very effective in detecting property violations in popular code translation models, and therefore, in evaluating model quality with respect to given properties. We also go a step further and explore the usage scenario where a user simply aims to obtain a correct translation of some code with respect to certain properties without necessarily being concerned about the overall quality of the model. To this purpose, we develop the first property-guided search procedure for code translation models, where a model is repeatedly queried with slightly different parameters to produce alternative and potentially more correct translations. Our results show that this search procedure helps to obtain significantly better code translations.

Related papers

• Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs [69.28193153685893]
  Large Language Models (LLMs) frequently exhibit strong translation abilities, even without task-specific fine-tuning.<n>To demystify this process, we leverage Sparse Autoencoders (SAEs) and introduce a novel framework for identifying task-specific features.<n>Our work not only decodes a core component of the translation mechanism in LLMs but also provides a blueprint for using internal model mechanism to create more robust and efficient models.
  arXiv  Detail & Related papers  (2026-01-16T06:29:07Z)
• Advancing Automated In-Isolation Validation in Repository-Level Code Translation [9.753507630426832]
  Repository-level code translation aims to migrate entire repositories across programming languages while preserving functionality automatically.<n>This paper proposes TRAM, which combines context-aware type resolution with mock-based in-isolation validation.<n>TRAM demonstrates state-of-the-art performance in Java-to-Python translation, underscoring the effectiveness of its integration of RAG-based type resolution with reliable in-isolation validation.
  arXiv  Detail & Related papers  (2025-11-26T19:53:46Z)
• Function-to-Style Guidance of LLMs for Code Translation [59.487054943812836]
  We propose F2STrans, a function-to-style guiding paradigm designed to improve the performance of large language models in code translation.<n>Our approach comprises two key stages: (1) Functional learning, which optimize translation correctness using high-quality source-target code pairs.<n>We introduce a novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth translations.
  arXiv  Detail & Related papers  (2025-07-15T08:25:02Z)
• Use Property-Based Testing to Bridge LLM Code Generation and Validation [38.25155484701058]
  Large Language Models (LLMs) excel at code generation, but ensuring their outputs to be functionally correct is a persistent challenge.<n>This paper introduces Property-Generated Solver, a novel framework that leverages Property-Based Testing (PBT) to validate high-level program properties.<n>Property-Generated Solver employs two collaborative LLM-based agents: a Generator dedicated to code generation and iterative refinement, and a Tester that manages the PBT life-cycle.
  arXiv  Detail & Related papers  (2025-06-23T06:01:12Z)
• GenX: Mastering Code and Test Generation with Execution Feedback [7.225594526057816]
  We propose a novel approach that concurrently trains a code generation model and a test generation model. We introduce two strategies for test and code data augmentation and a new scoring function for code and test ranking. The results demonstrate that our models, when iteratively trained with an increasing number of test cases and code solutions, outperform those trained on the original dataset.
  arXiv  Detail & Related papers  (2024-12-18T03:18:21Z)
• A test-free semantic mistakes localization framework in Neural Code Translation [32.5036379897325]
  We present EISP, a static analysis framework based on the Large Language Model (LLM) The framework generates a semantic mapping between source code and translated code. EISP connects each pair of sub-code fragments with fine-grained knowledge hints through an AI chain.
  arXiv  Detail & Related papers  (2024-10-30T08:53:33Z)
• CAT-LM: Training Language Models on Aligned Code And Tests [19.526181671936243]
  Testing is an integral part of the software development process. Yet, writing tests is time-consuming and therefore often neglected. We propose the Aligned Code And Tests Language Model (CAT-LM), a GPT-style language model with 2.7 Billion parameters, trained on a corpus of Python and Java projects.
  arXiv  Detail & Related papers  (2023-10-02T19:52:22Z)
• A Static Evaluation of Code Completion by Large Language Models [65.18008807383816]
  Execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. We propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees.
  arXiv  Detail & Related papers  (2023-06-05T19:23:34Z)
• Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study [4.438873396405334]
  We aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. For comments, we find that the models perform better in the presence of multi-line comments.
  arXiv  Detail & Related papers  (2023-04-24T17:09:14Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing [55.058258437125524]
  We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing. We benchmark eight language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models can achieve similar performance or surpass state-of-the-art methods for syntactic and semantic parsing when the model output is constrained to be valid.
  arXiv  Detail & Related papers  (2022-06-21T18:34:11Z)
• Using Document Similarity Methods to create Parallel Datasets for Code Translation [60.36392618065203]
  Translating source code from one programming language to another is a critical, time-consuming task. We propose to use document similarity methods to create noisy parallel datasets of code. We show that these models perform comparably to models trained on ground truth for reasonable levels of noise.
  arXiv  Detail & Related papers  (2021-10-11T17:07:58Z)
• BERT2Code: Can Pretrained Language Models be Leveraged for Code Search? [0.7953229555481884]
  We show that our model learns the inherent relationship between the embedding spaces and further probes into the scope of improvement. In this analysis, we show that the quality of the code embedding model is the bottleneck for our model's performance.
  arXiv  Detail & Related papers  (2021-04-16T10:28:27Z)
• Zero-Shot Cross-lingual Semantic Parsing [56.95036511882921]
  We study cross-lingual semantic parsing as a zero-shot problem without parallel data for 7 test languages. We propose a multi-task encoder-decoder model to transfer parsing knowledge to additional languages using only English-Logical form paired data. Our system frames zero-shot parsing as a latent-space alignment problem and finds that pre-trained models can be improved to generate logical forms with minimal cross-lingual transfer penalty.
  arXiv  Detail & Related papers  (2021-04-15T16:08:43Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.