BERT2Code: Can Pretrained Language Models be Leveraged for Code Search?

• URL: http://arxiv.org/abs/2104.08017v1
• Date: Fri, 16 Apr 2021 10:28:27 GMT
• Title: BERT2Code: Can Pretrained Language Models be Leveraged for Code Search?
• Authors: Abdullah Al Ishtiaq, Masum Hasan, Md. Mahim Anjum Haque, Kazi Sajeed Mehrab, Tanveer Muttaqueen, Tahmid Hasan, Anindya Iqbal, Rifat Shahriyar
• Abstract summary: We show that our model learns the inherent relationship between the embedding spaces and further probes into the scope of improvement. In this analysis, we show that the quality of the code embedding model is the bottleneck for our model's performance.
• Score: 0.7953229555481884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Millions of repetitive code snippets are submitted to code repositories every day. To search from these large codebases using simple natural language queries would allow programmers to ideate, prototype, and develop easier and faster. Although the existing methods have shown good performance in searching codes when the natural language description contains keywords from the code, they are still far behind in searching codes based on the semantic meaning of the natural language query and semantic structure of the code. In recent years, both natural language and programming language research communities have created techniques to embed them in vector spaces. In this work, we leverage the efficacy of these embedding models using a simple, lightweight 2-layer neural network in the task of semantic code search. We show that our model learns the inherent relationship between the embedding spaces and further probes into the scope of improvement by empirically analyzing the embedding methods. In this analysis, we show that the quality of the code embedding model is the bottleneck for our model's performance, and discuss future directions of study in this area.

Related papers

• Survey of Code Search Based on Deep Learning [11.94599964179766]
  This survey focuses on code search, that is, to retrieve code that matches a given query. Deep learning, being able to extract complex semantics information, has achieved great success in this field. We propose a new taxonomy to illustrate the state-of-the-art deep learning-based code search.
  arXiv  Detail & Related papers  (2023-05-10T08:07:04Z)
• Generation-Augmented Query Expansion For Code Retrieval [51.20943646688115]
  We propose a generation-augmented query expansion framework. Inspired by the human retrieval process - sketching an answer before searching. We achieve new state-of-the-art results on the CodeSearchNet benchmark.
  arXiv  Detail & Related papers  (2022-12-20T23:49:37Z)
• Python Code Generation by Asking Clarification Questions [57.63906360576212]
  In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions. We collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers.
  arXiv  Detail & Related papers  (2022-12-19T22:08:36Z)
• Enhancing Semantic Code Search with Multimodal Contrastive Learning and Soft Data Augmentation [50.14232079160476]
  We propose a new approach with multimodal contrastive learning and soft data augmentation for code search. We conduct extensive experiments to evaluate the effectiveness of our approach on a large-scale dataset with six programming languages.
  arXiv  Detail & Related papers  (2022-04-07T08:49:27Z)
• Learning Deep Semantic Model for Code Search using CodeSearchNet Corpus [17.6095840480926]
  We propose a novel deep semantic model which makes use of the utilities of multi-modal sources. We apply the proposed model to tackle the CodeSearchNet challenge about semantic code search. Our model is trained on CodeSearchNet corpus and evaluated on the held-out data, the final model achieves 0.384 NDCG and won the first place in this benchmark.
  arXiv  Detail & Related papers  (2022-01-27T04:15:59Z)
• CodeRetriever: Unimodal and Bimodal Contrastive Learning [128.06072658302165]
  We propose the CodeRetriever model, which combines the unimodal and bimodal contrastive learning to train function-level code semantic representations. For unimodal contrastive learning, we design a semantic-guided method to build positive code pairs based on the documentation and function name. For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build text-code pairs.
  arXiv  Detail & Related papers  (2022-01-26T10:54:30Z)
• CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model [23.947178895479464]
  We propose CLSEBERT, a Constrastive Learning Framework for Syntax Enhanced Code Pre-Trained Model. In the pre-training stage, we consider the code syntax and hierarchy contained in the Abstract Syntax Tree (AST) We also introduce two novel pre-training objectives. One is to predict the edges between nodes in the abstract syntax tree, and the other is to predict the types of code tokens.
  arXiv  Detail & Related papers  (2021-08-10T10:08:21Z)
• COSEA: Convolutional Code Search with Layer-wise Attention [90.35777733464354]
  We propose a new deep learning architecture, COSEA, which leverages convolutional neural networks with layer-wise attention to capture the code's intrinsic structural logic. COSEA can achieve significant improvements over state-of-the-art methods on code search tasks.
  arXiv  Detail & Related papers  (2020-10-19T13:53:38Z)
• Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent [1.1168121941015012]
  We study how code retrieval systems can be improved by leveraging descriptions to better capture the intents of code snippets. Building on recent progress in transfer learning and natural language processing, we create a domain-specific retrieval model for code annotated with a natural language description.
  arXiv  Detail & Related papers  (2020-08-27T15:39:09Z)
• A Multi-Perspective Architecture for Semantic Code Search [58.73778219645548]
  We propose a novel multi-perspective cross-lingual neural framework for code--text matching. Our experiments on the CoNaLa dataset show that our proposed model yields better performance than previous approaches.
  arXiv  Detail & Related papers  (2020-05-06T04:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.