Natural Language to Code Generation in Interactive Data Science
Notebooks
- URL: http://arxiv.org/abs/2212.09248v1
- Date: Mon, 19 Dec 2022 05:06:00 GMT
- Title: Natural Language to Code Generation in Interactive Data Science
Notebooks
- Authors: Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen,
Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk
Michalewski, Alex Polozov, Charles Sutton
- Abstract summary: We build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks.
We develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs.
- Score: 35.621936471322385
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Computational notebooks, such as Jupyter notebooks, are interactive computing
environments that are ubiquitous among data scientists to perform data
wrangling and analytic tasks. To measure the performance of AI pair programmers
that automatically synthesize programs for those tasks given natural language
(NL) intents from users, we build ARCADE, a benchmark of 1082 code generation
problems using the pandas data analysis framework in data science notebooks.
ARCADE features multiple rounds of NL-to-code problems from the same notebook.
It requires a model to understand rich multi-modal contexts, such as existing
notebook cells and their execution states as well as previous turns of
interaction. To establish a strong baseline on this challenging task, we
develop PaChiNCo, a 62B code language model (LM) for Python computational
notebooks, which significantly outperforms public code LMs. Finally, we explore
few-shot prompting strategies to elicit better code with step-by-step
decomposition and NL explanation, showing the potential to improve the
diversity and explainability of model predictions.
Related papers
- Code Needs Comments: Enhancing Code LLMs with Comment Augmentation [91.52444946362547]
We introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language.
We conducted experiments on three code-focused Large Language Models and observed consistent improvements in performance on two widely-used programming skill benchmarks.
arXiv Detail & Related papers (2024-02-20T13:56:38Z) - MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible
Pipeline [12.186691561822256]
We postulate that the inherent nature of large language models (LLMs) presents challenges in modeling mathematical reasoning.
This paper introduces a novel math dataset, enhanced with a capability to utilize a Python code interpreter.
We propose a tentative, easily replicable protocol for the fine-tuning of math-specific LLMs.
arXiv Detail & Related papers (2024-01-16T08:08:01Z) - Leveraging Generative AI: Improving Software Metadata Classification
with Generated Code-Comment Pairs [0.0]
In software development, code comments play a crucial role in enhancing code comprehension and collaboration.
This research paper addresses the challenge of objectively classifying code comments as "Useful" or "Not Useful"
We propose a novel solution that harnesses contextualized embeddings, particularly BERT, to automate this classification process.
arXiv Detail & Related papers (2023-10-14T12:09:43Z) - The Consensus Game: Language Model Generation via Equilibrium Search [73.51411916625032]
We introduce a new, a training-free, game-theoretic procedure for language model decoding.
Our approach casts language model decoding as a regularized imperfect-information sequential signaling game.
Applying EQUILIBRIUM-RANKING to LLaMA-7B outperforms the much larger LLaMA-65B and PaLM-540B models.
arXiv Detail & Related papers (2023-10-13T14:27:21Z) - Pair Programming with Large Language Models for Sampling and Estimation
of Copulas [0.0]
An example Monte Carlo simulation based application for dependence modeling with copulas is developed using a state-of-the-art large language model (LLM)
This includes interaction with ChatGPT in natural language and using mathematical formalism, which led to producing a working code in Python and R.
Through careful prompt engineering, we separate successful solutions generated by ChatGPT from unsuccessful ones, resulting in a comprehensive list of related pros and cons.
arXiv Detail & Related papers (2023-03-31T15:02:48Z) - A Conversational Paradigm for Program Synthesis [110.94409515865867]
We propose a conversational program synthesis approach via large language models.
We train a family of large language models, called CodeGen, on natural language and programming language data.
Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm.
arXiv Detail & Related papers (2022-03-25T06:55:15Z) - Efficient Nearest Neighbor Language Models [114.40866461741795]
Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore.
We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
arXiv Detail & Related papers (2021-09-09T12:32:28Z) - Data-to-text Generation with Macro Planning [61.265321323312286]
We propose a neural model with a macro planning stage followed by a generation stage reminiscent of traditional methods.
Our approach outperforms competitive baselines in terms of automatic and human evaluation.
arXiv Detail & Related papers (2021-02-04T16:32:57Z) - Automated Source Code Generation and Auto-completion Using Deep
Learning: Comparing and Discussing Current Language-Model-Related Approaches [0.0]
This paper compares different deep learning architectures to create and use language models based on programming code.
We discuss each approach's different strengths and weaknesses and what gaps we find to evaluate the language models or apply them in a real programming context.
arXiv Detail & Related papers (2020-09-16T15:17:04Z) - Leveraging Code Generation to Improve Code Retrieval and Summarization
via Dual Learning [18.354352985591305]
Code summarization generates brief natural language description given a source code snippet, while code retrieval fetches relevant source code given a natural language query.
Recent studies have combined these two tasks to improve their performance.
We propose a novel end-to-end model for the two tasks by introducing an additional code generation task.
arXiv Detail & Related papers (2020-02-24T12:26:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.