DemoCraft: Using In-Context Learning to Improve Code Generation in Large Language Models

• URL: http://arxiv.org/abs/2411.00865v1
• Date: Wed, 30 Oct 2024 19:45:50 GMT
• Title: DemoCraft: Using In-Context Learning to Improve Code Generation in Large Language Models
• Authors: Nirmal Joshua Kapu, Mihit Sreejith,
• Abstract summary: We propose DemoCraft, which enhances code generation by leveraging in-context learning and demonstration selection. Latent concept learning introduces additional concept tokens, which are trainable embeddings that capture task-specific knowledge. Our experimental results demonstrate that the proposed system achieves an approximate 2x increase in the pass@k metric. Our empirical studies indicate that our system attains nearly a 3x improvement in these metrics as well.
• Score: 0.0
• License:
• Abstract: Generating executable code from natural language instructions using Large Language Models (LLMs) poses challenges such as semantic ambiguity and understanding taskspecific contexts. To address these issues, we propose a system called DemoCraft, which enhances code generation by leveraging in-context learning and demonstration selection, combined with latent concept learning. Latent concept learning introduces additional concept tokens, which are trainable embeddings that capture task-specific knowledge. We then test our system on two major datasets: MBPP and Humaneval. Our experimental results demonstrate that the proposed system achieves an approximate 2x increase in the pass@k metric compared to baseline models. Furthermore, we introduce two novel evaluation metrics: correctness@k and similarity@k. Our empirical studies indicate that our system attains nearly a 3x improvement in these metrics as well.

Related papers

• Code Representation Learning At Scale [75.04686476303436]
  We fuel code representation learning with a vast amount of code data via a two-stage pretraining scheme. We first train the encoders via a mix that leverages both randomness in masking language modeling and the structure aspect of programming language. We then enhance the representations via contrastive learning with hard negative and hard positive constructed in an unsupervised manner.
  arXiv  Detail & Related papers  (2024-02-02T22:19:15Z)
• Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
  We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases. We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
  arXiv  Detail & Related papers  (2024-01-16T08:44:29Z)
• Social Learning: Towards Collaborative Learning with Large Language Models [10.24107243529341]
  We introduce the framework of "social learning" in the context of large language models (LLMs) We present and evaluate two approaches for knowledge transfer between LLMs. We show that performance using these methods is comparable to results with the use of original labels and prompts.
  arXiv  Detail & Related papers  (2023-12-18T18:44:10Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• Learning to Retrieve In-Context Examples for Large Language Models [69.9707552694766]
  Large language models (LLMs) have demonstrated their ability to learn in-context. The effectiveness of in-context learning is heavily reliant on the quality of the selected examples. We propose a novel framework to iteratively train dense retrievers that can identify high-quality in-context examples.
  arXiv  Detail & Related papers  (2023-07-14T05:23:08Z)
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs? [60.668318972782295]
  Large language models have shown the ability of in-context learning (ICL) ICL employs task instructions and a few examples as demonstrations, and then inputs the demonstrations to the language models for making predictions. It is important to systematically investigate how to construct a good demonstration for code-related tasks.
  arXiv  Detail & Related papers  (2023-04-15T15:13:58Z)
• A Cohesive Distillation Architecture for Neural Language Models [0.0]
  A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size. This study investigates methods for Knowledge Distillation (KD) to provide efficient alternatives to large-scale models.
  arXiv  Detail & Related papers  (2023-01-12T08:01:53Z)
• Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models [42.38563175680914]
  This paper studies how to automatically generate a natural language text that describes the facts in knowledge graph (KG) Considering the few-shot setting, we leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation.
  arXiv  Detail & Related papers  (2021-06-03T06:48:00Z)
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training [135.12061144759517]
  We present an information-theoretic framework that formulates cross-lingual language model pre-training. We propose a new pre-training task based on contrastive learning. By leveraging both monolingual and parallel corpora, we jointly train the pretext to improve the cross-lingual transferability of pre-trained models.
  arXiv  Detail & Related papers  (2020-07-15T16:58:01Z)
• Learning Spoken Language Representations with Neural Lattice Language Modeling [39.50831917042577]
  We propose a framework that trains neural lattice language models to provide contextualized representations for spoken language understanding tasks. The proposed two-stage pre-training approach reduces the demands of speech data and has better efficiency.
  arXiv  Detail & Related papers  (2020-07-06T10:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.