Toward Trustworthy Neural Program Synthesis

• URL: http://arxiv.org/abs/2210.00848v2
• Date: Mon, 9 Oct 2023 21:21:31 GMT
• Title: Toward Trustworthy Neural Program Synthesis
• Authors: Darren Key, Wen-Ding Li, Kevin Ellis
• Abstract summary: We develop an approach to estimate the probability that a program sampled from a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave.
• Score: 6.3557174349423455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop an approach to estimate the probability that a program sampled from a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave. This allows learning a model that forms a well-calibrated probabilistic prediction of program correctness. Our system also infers which predicates are useful to explain the behavior of the generated code, and humans preferred these in a human study over raw language model outputs. Our method is simple, easy to implement, and maintains state of the art generation accuracy results.

Related papers

• Improving Open-Ended Text Generation via Adaptive Decoding [6.746656673563492]
  This study introduces adaptive decoding, a mechanism that dynamically empowers language models to ascertain a sensible candidate set during generation. Experimental results reveal that our method balances diversity and coherence well. Our method can potentially improve the reasoning ability of language models.
  arXiv  Detail & Related papers  (2024-02-28T10:38:21Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)
• Learning from Self-Sampled Correct and Partially-Correct Programs [96.66452896657991]
  We propose to let the model perform sampling during training and learn from both self-sampled fully-correct programs and partially-correct programs. We show that our use of self-sampled correct and partially-correct programs can benefit learning and help guide the sampling process. Our proposed method improves the pass@k performance by 3.1% to 12.3% compared to learning from a single reference program with MLE.
  arXiv  Detail & Related papers  (2022-05-28T03:31:07Z)
• Foundation Posteriors for Approximate Probabilistic Inference [11.64841553345271]
  We formulate inference as masked language modeling in a probabilistic program. We train a neural network to unmask the random values, defining an approximate posterior distribution. We show the efficacy of the approach, zero-shot and fine-tuned, on a benchmark of STAN programs.
  arXiv  Detail & Related papers  (2022-05-19T17:42:37Z)
• Program Analysis of Probabilistic Programs [3.299672391663527]
  dissertation presents three novel techniques to improve probabilistic programming using program analysis. The techniques analyse a probabilistic program and adapt it to make inference more efficient, sometimes in a way that would have been tedious or impossible to do by hand.
  arXiv  Detail & Related papers  (2022-04-14T10:40:54Z)
• A Conversational Paradigm for Program Synthesis [110.94409515865867]
  We propose a conversational program synthesis approach via large language models. We train a family of large language models, called CodeGen, on natural language and programming language data. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm.
  arXiv  Detail & Related papers  (2022-03-25T06:55:15Z)
• Typical Decoding for Natural Language Generation [76.69397802617064]
  We study why high-probability texts can be dull or repetitive. We show that typical sampling offers competitive performance in terms of quality.
  arXiv  Detail & Related papers  (2022-02-01T18:58:45Z)
• Latent Programmer: Discrete Latent Codes for Program Synthesis [56.37993487589351]
  In many sequence learning tasks, such as program synthesis and document summarization, a key problem is searching over a large space of possible output sequences. We propose to learn representations of the outputs that are specifically meant for search: rich enough to specify the desired output but compact enough to make search more efficient. We introduce the emphLatent Programmer, a program synthesis method that first predicts a discrete latent code from input/output examples, and then generates the program in the target language.
  arXiv  Detail & Related papers  (2020-12-01T10:11:35Z)
• Transforming Probabilistic Programs for Model Checking [0.0]
  We apply static analysis to probabilistic programs to automate large parts of two crucial model checking methods. Our method transforms a probabilistic program specifying a density function into an efficient forward-sampling form. We present an implementation targeting the popular Stan probabilistic programming language.
  arXiv  Detail & Related papers  (2020-08-21T21:06:34Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.