Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges

• URL: http://arxiv.org/abs/2002.05442v1
• Date: Thu, 13 Feb 2020 11:02:51 GMT
• Title: Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges
• Authors: Triet H. M. Le, Hao Chen, M. Ali Babar
• Abstract summary: Deep Learning (DL) techniques for Natural Language Processing have been evolving remarkably fast. Recently, the DL advances in language modeling, machine translation and paragraph understanding are so prominent that the potential of DL in Software Engineering cannot be overlooked.
• Score: 5.4052819252055055
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Learning (DL) techniques for Natural Language Processing have been evolving remarkably fast. Recently, the DL advances in language modeling, machine translation and paragraph understanding are so prominent that the potential of DL in Software Engineering cannot be overlooked, especially in the field of program learning. To facilitate further research and applications of DL in this field, we provide a comprehensive review to categorize and investigate existing DL methods for source code modeling and generation. To address the limitations of the traditional source code models, we formulate common program learning tasks under an encoder-decoder framework. After that, we introduce recent DL mechanisms suitable to solve such problems. Then, we present the state-of-the-art practices and discuss their challenges with some recommendations for practitioners and researchers as well.

Related papers

• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities [89.40778301238642]
  Model merging is an efficient empowerment technique in the machine learning community. There is a significant gap in the literature regarding a systematic and thorough review of these techniques.
  arXiv  Detail & Related papers  (2024-08-14T16:58:48Z)
• DeepCodeProbe: Towards Understanding What Models Trained on Code Learn [13.135962181354465]
  We introduce DeepCodeProbe, a probing approach that examines the syntax and representation learning abilities of ML models. Our study applies DeepCodeProbe to state-of-the-art models for code clone detection, code summarization, and comment generation. Findings reveal that while small models capture abstract syntactic representations, their ability to fully grasp programming language syntax is limited.
  arXiv  Detail & Related papers  (2024-07-11T23:16:44Z)
• Science-Informed Deep Learning (ScIDL) With Applications to Wireless Communications [11.472232944923558]
  This article provides a tutorial on science-informed deep learning (ScIDL) ScIDL aims to integrate existing scientific knowledge with DL techniques to develop more powerful algorithms. We discuss both recent applications of ScIDL and potential future research directions in the field of wireless communications.
  arXiv  Detail & Related papers  (2024-06-29T02:35:39Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  We propose CodeGRAG, a Graphical Retrieval Augmented Code Generation framework to enhance the performance of LLMs. CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to fill the gap between programming languages and natural language. Various experiments and ablations are done on four datasets including both the C++ and python languages to validate the hard meta-graph prompt, the soft prompting technique, and the effectiveness of the objectives for pretrained GNN expert.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• CodeTF: One-stop Transformer Library for State-of-the-art Code LLM [72.1638273937025]
  We present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Our library supports a collection of pretrained Code LLM models and popular code benchmarks. We hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering.
  arXiv  Detail & Related papers  (2023-05-31T05:24:48Z)
• Deep Learning in Healthcare: An In-Depth Analysis [1.892561703051693]
  We provide a review of Deep Learning models and their broad application in bioinformatics and healthcare. We also go over some of the key challenges that still exist and can show up while conducting DL research.
  arXiv  Detail & Related papers  (2023-02-12T20:55:34Z)
• A Survey on Pretrained Language Models for Neural Code Intelligence [4.020523898765404]
  The field of Neural Code Intelligence (NCI) has emerged as a promising solution to tackle analytical tasks on source code. NCI aims to improve programming efficiency and minimize human errors within the software industry. Pretrained language models have become a dominant force in NCI research, consistently delivering state-of-the-art results.
  arXiv  Detail & Related papers  (2022-12-20T08:34:56Z)
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning [92.36705236706678]
  "CodeRL" is a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning. During inference, we introduce a new generation procedure with a critical sampling strategy. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives.
  arXiv  Detail & Related papers  (2022-07-05T02:42:15Z)
• Towards Training Reproducible Deep Learning Models [26.547756923322126]
  Deep Learning (DL) models are challenging to be reproduced due to issues like randomness in the software and non-determinism in the hardware. This paper proposes a systematic approach to training reproducible DL models. Case study results show our approach can successfully reproduce six open source and one commercial DL models.
  arXiv  Detail & Related papers  (2022-02-04T18:14:39Z)
• Reprogramming Language Models for Molecular Representation Learning [65.00999660425731]
  We propose Representation Reprogramming via Dictionary Learning (R2DL) for adversarially reprogramming pretrained language models for molecular learning tasks. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver. R2DL achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data and outperforms the baseline in a limited training-data setting.
  arXiv  Detail & Related papers  (2020-12-07T05:50:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.