Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models

• URL: http://arxiv.org/abs/2106.13928v1
• Date: Sat, 26 Jun 2021 03:02:49 GMT
• Title: Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models
• Authors: Jingxuan Li, Rui Huang, Wei Li, Kai Yao, Weiguo Tan
• Abstract summary: We develop an ensemble framework that can combine results from multiple models to draw merits and offset defects of each model. This paper conducts a coding simulation to collect data from code context and different code completion models. We propose a new code completion evaluation metric, Benefit-Cost Ratio(BCR), taking into account the benefit of keystrokes saving and hidden cost of completion list browsing.
• Score: 12.736207952790618
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Code completion is widely used by software developers to provide coding suggestions given a partially written code snippet. Apart from the traditional code completion methods, which only support single token completion at minimal positions, recent studies show the ability to provide longer code completion at more flexible positions. However, such frequently triggered and longer completion results reduce the overall precision as they generate more invalid results. Moreover, different studies are mostly incompatible with each other. Thus, it is vital to develop an ensemble framework that can combine results from multiple models to draw merits and offset defects of each model. This paper conducts a coding simulation to collect data from code context and different code completion models and then apply the data in two tasks. First, we introduce an acceptance model which can dynamically control whether to display completion results to the developer. It uses simulation features to predict whether correct results exist in the output of these models. Our best model reduces the percentage of false-positive completion from 55.09% to 17.44%. Second, we design a fusion ranking scheme that can automatically identify the priority of the completion results and reorder the candidates from multiple code completion models. This scheme is flexible in dealing with various models, regardless of the type or the length of their completion results. We integrate this ranking scheme with two frequency models and a GPT-2 styled language model, along with the acceptance model to yield 27.80% and 37.64% increase in TOP1 and TOP5 accuracy, respectively. In addition, we propose a new code completion evaluation metric, Benefit-Cost Ratio(BCR), taking into account the benefit of keystrokes saving and hidden cost of completion list browsing, which is closer to real coder experience scenario.

Related papers

• Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach [66.51005288743153]
  We investigate the legal and ethical issues of current neural code completion models. We tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks. We evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models.
  arXiv  Detail & Related papers  (2024-04-22T15:54:53Z)
• When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference [11.704110756342212]
  We propose a novel dynamic inference method specifically tailored for code completion models. It can averagely skip 1.7 layers out of 16 layers in the models, leading to an 11.2% speedup with only a marginal 1.1% reduction in ROUGE-L.
  arXiv  Detail & Related papers  (2024-01-18T13:26:53Z)
• Multi-Candidate Speculative Decoding [82.05519287513444]
  Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments from a fast draft model that is then verified in parallel by the target model. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model.
  arXiv  Detail & Related papers  (2024-01-12T17:15:23Z)
• Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
  TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution. We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
  arXiv  Detail & Related papers  (2023-06-09T06:02:01Z)
• Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study [4.438873396405334]
  We aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. For comments, we find that the models perform better in the presence of multi-line comments.
  arXiv  Detail & Related papers  (2023-04-24T17:09:14Z)
• CodeExp: Explanatory Code Document Generation [94.43677536210465]
  Existing code-to-text generation models produce only high-level summaries of code. We conduct a human study to identify the criteria for high-quality explanatory docstring for code. We present a multi-stage fine-tuning strategy and baseline models for the task.
  arXiv  Detail & Related papers  (2022-11-25T18:05:44Z)
• DORE: Document Ordered Relation Extraction based on Generative Framework [56.537386636819626]
  This paper investigates the root cause of the underwhelming performance of the existing generative DocRE models. We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn. Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.
  arXiv  Detail & Related papers  (2022-10-28T11:18:10Z)
• Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems [16.03416381009787]
  Currently, large pre-trained language models are widely applied in neural code completion systems. Around 70% of displayed code completions from Github Copilot are not accepted by developers. We propose an early-rejection mechanism to turn down low-return prompts by foretelling the code completion qualities.
  arXiv  Detail & Related papers  (2022-09-13T12:43:41Z)
• When Liebig's Barrel Meets Facial Landmark Detection: A Practical Model [87.25037167380522]
  We propose a model that is accurate, robust, efficient, generalizable, and end-to-end trainable. In order to achieve a better accuracy, we propose two lightweight modules. DQInit dynamically initializes the queries of decoder from the inputs, enabling the model to achieve as good accuracy as the ones with multiple decoder layers. QAMem is designed to enhance the discriminative ability of queries on low-resolution feature maps by assigning separate memory values to each query rather than a shared one.
  arXiv  Detail & Related papers  (2021-05-27T13:51:42Z)
• Towards Full-line Code Completion with Neural Language Models [25.458883198815393]
  We discuss the probability of directly completing a whole line of code instead of a single token. Recent neural language models have been adopted as a preferred approach for code completion.
  arXiv  Detail & Related papers  (2020-09-18T03:12:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.