Understanding Programs by Exploiting (Fuzzing) Test Cases
- URL: http://arxiv.org/abs/2305.13592v2
- Date: Mon, 12 Jun 2023 00:56:21 GMT
- Title: Understanding Programs by Exploiting (Fuzzing) Test Cases
- Authors: Jianyu Zhao and Yuyang Rong and Yiwen Guo and Yifeng He and Hao Chen
- Abstract summary: We propose to incorporate the relationship between inputs and possible outputs/behaviors into learning, for achieving a deeper semantic understanding of programs.
To obtain inputs that are representative enough to trigger the execution of most part of the code, we resort to fuzz testing and propose fuzz tuning.
The effectiveness of the proposed method is verified on two program understanding tasks including code clone detection and code classification, and it outperforms current state-of-the-arts by large margins.
- Score: 26.8259045248779
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Semantic understanding of programs has attracted great attention in the
community. Inspired by recent successes of large language models (LLMs) in
natural language understanding, tremendous progress has been made by treating
programming language as another sort of natural language and training LLMs on
corpora of program code. However, programs are essentially different from texts
after all, in a sense that they are normally heavily structured and
syntax-strict. In particular, programs and their basic units (i.e., functions
and subroutines) are designed to demonstrate a variety of behaviors and/or
provide possible outputs, given different inputs. The relationship between
inputs and possible outputs/behaviors represents the functions/subroutines and
profiles the program as a whole. Therefore, we propose to incorporate such a
relationship into learning, for achieving a deeper semantic understanding of
programs. To obtain inputs that are representative enough to trigger the
execution of most part of the code, we resort to fuzz testing and propose fuzz
tuning to boost the performance of program understanding and code
representation learning, given a pre-trained LLM. The effectiveness of the
proposed method is verified on two program understanding tasks including code
clone detection and code classification, and it outperforms current
state-of-the-arts by large margins. Code is available at
https://github.com/rabbitjy/FuzzTuning.
Related papers
- Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance? [26.91104188917787]
Large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and logical reasoning tasks.
Our research aims to verify which programming languages and features during pre-training affect logical inference performance.
arXiv Detail & Related papers (2024-10-09T10:13:13Z) - Learning to Reason via Program Generation, Emulation, and Search [33.11955431589091]
Program synthesis with language models (LMs) has unlocked a large set of reasoning abilities.
Not all reasoning tasks are easily expressible as code, e.g. tasks involving commonsense reasoning, moral decision-making, and sarcasm understanding.
We propose Code Generation and Emulated EXecution (CoGEX) to extend an LM's program synthesis skills to such tasks.
arXiv Detail & Related papers (2024-05-25T19:40:50Z) - AIOS Compiler: LLM as Interpreter for Natural Language Programming and Flow Programming of AI Agents [38.580779075892636]
We develop a novel system for Code Representation and Execution (CoRE)
The proposed system unifies natural language programming, pseudo-code programming, and flow programming under the same representation for constructing language agents.
During the execution, we incorporate external memory to minimize redundancy.
arXiv Detail & Related papers (2024-05-11T04:29:03Z) - CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
We propose CodeGRAG, a Graphical Retrieval Augmented Code Generation framework to enhance the performance of LLMs.
CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to fill the gap between programming languages and natural language.
Various experiments and ablations are done on four datasets including both the C++ and python languages to validate the hard meta-graph prompt, the soft prompting technique, and the effectiveness of the objectives for pretrained GNN expert.
arXiv Detail & Related papers (2024-05-03T02:48:55Z) - Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning [84.12154024070024]
We propose natural language embedded programs (NLEP) as a unifying framework for addressing math/symbolic reasoning, natural language understanding, and instruction following tasks.
Our approach prompts a language model to generate full Python programs that define functions over data structures which contain natural language representations of structured knowledge.
A Python interpreter then executes the generated code and prints the output.
arXiv Detail & Related papers (2023-09-19T17:54:21Z) - Code Representation Pre-training with Complements from Program
Executions [29.148208436656216]
We propose FuzzPretrain to explore the dynamic information of programs revealed by their test cases and embed it into the feature representations of code as complements.
FuzzyPretrain yielded more than 6%/9% mAP improvements on code search over its counterparts trained with only source code or AST.
arXiv Detail & Related papers (2023-09-04T01:57:22Z) - LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results.
Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results.
LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
arXiv Detail & Related papers (2023-02-16T18:23:22Z) - Natural Language to Code Translation with Execution [82.52142893010563]
Execution result--minimum Bayes risk decoding for program selection.
We show that it improves the few-shot performance of pretrained code models on natural-language-to-code tasks.
arXiv Detail & Related papers (2022-04-25T06:06:08Z) - Unified Pre-training for Program Understanding and Generation [46.89905110678675]
PLBART is a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks.
PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding.
arXiv Detail & Related papers (2021-03-10T20:32:59Z) - Latent Programmer: Discrete Latent Codes for Program Synthesis [56.37993487589351]
In many sequence learning tasks, such as program synthesis and document summarization, a key problem is searching over a large space of possible output sequences.
We propose to learn representations of the outputs that are specifically meant for search: rich enough to specify the desired output but compact enough to make search more efficient.
We introduce the emphLatent Programmer, a program synthesis method that first predicts a discrete latent code from input/output examples, and then generates the program in the target language.
arXiv Detail & Related papers (2020-12-01T10:11:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.