Tackling Long Code Search with Splitting, Encoding, and Aggregating

• URL: http://arxiv.org/abs/2208.11271v3
• Date: Tue, 26 Mar 2024 14:51:38 GMT
• Title: Tackling Long Code Search with Splitting, Encoding, and Aggregating
• Authors: Fan Hu, Yanlin Wang, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, Xirong Li,
• Abstract summary: We propose a new baseline SEA (Split, Encode and Aggregate) for long code search. It splits long code into code blocks, encodes these blocks into embeddings, and aggregates them to obtain a comprehensive long code representation. With GraphCodeBERT as the encoder, SEA achieves an overall mean reciprocal ranking score of 0.785, which is 10.1% higher than GraphCodeBERT on the CodeSearchNet benchmark.
• Score: 67.02322603435628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code search with natural language helps us reuse existing code snippets. Thanks to the Transformer-based pretraining models, the performance of code search has been improved significantly. However, due to the quadratic complexity of multi-head self-attention, there is a limit on the input token length. For efficient training on standard GPUs like V100, existing pretrained code models, including GraphCodeBERT, CodeBERT, RoBERTa (code), take the first 256 tokens by default, which makes them unable to represent the complete information of long code that is greater than 256 tokens. To tackle the long code problem, we propose a new baseline SEA (Split, Encode and Aggregate), which splits long code into code blocks, encodes these blocks into embeddings, and aggregates them to obtain a comprehensive long code representation. With SEA, we could directly use Transformer-based pretraining models to model long code without changing their internal structure and re-pretraining. We also compare SEA with sparse Trasnformer methods. With GraphCodeBERT as the encoder, SEA achieves an overall mean reciprocal ranking score of 0.785, which is 10.1% higher than GraphCodeBERT on the CodeSearchNet benchmark, justifying SEA as a strong baseline for long code search. Our source code and experimental data are available at: https://github.com/fly-dragon211/SEA.

Related papers

• Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach [66.51005288743153]
  We investigate the legal and ethical issues of current neural code completion models. We tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks. We evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models.
  arXiv  Detail & Related papers  (2024-04-22T15:54:53Z)
• LongCoder: A Long-Range Pre-trained Language Model for Code Completion [56.813974784131624]
  LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction. memory tokens are included to highlight important statements that may be invoked later and need to be memorized.
  arXiv  Detail & Related papers  (2023-06-26T17:59:24Z)
• CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code [75.08995072899594]
  We propose CodeBERTScore: an evaluation metric for code generation. CodeBERTScore encodes the natural language input preceding the generated code. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics.
  arXiv  Detail & Related papers  (2023-02-10T22:12:05Z)
• CodeRetriever: Unimodal and Bimodal Contrastive Learning [128.06072658302165]
  We propose the CodeRetriever model, which combines the unimodal and bimodal contrastive learning to train function-level code semantic representations. For unimodal contrastive learning, we design a semantic-guided method to build positive code pairs based on the documentation and function name. For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build text-code pairs.
  arXiv  Detail & Related papers  (2022-01-26T10:54:30Z)
• DeSkew-LSH based Code-to-Code Recommendation Engine [3.7011129410662558]
  We present emphSenatus, a new code-to-code recommendation engine for machine learning on source code. At the core of Senatus is emphDe-Skew LSH, a new locality sensitive hashing algorithm that indexes the data for fast (sub-linear time) retrieval. We show Senatus improves performance by 6.7% F1 and query time 16x is faster compared to Facebook Aroma on the task of code-to-code recommendation.
  arXiv  Detail & Related papers  (2021-11-05T16:56:28Z)
• AugmentedCode: Examining the Effects of Natural Language Resources in Code Retrieval Models [5.112140303263898]
  We introduce Augmented Code (AugmentedCode) retrieval which takes advantage of existing information within the code. We showcased the the results of augmented programming language which outperforms on CodeSearchNet and CodeBERT with a Mean Reciprocal Rank (MRR) of 0.73 and 0.96.
  arXiv  Detail & Related papers  (2021-10-16T08:44:48Z)
• What do pre-trained code models know about code? [9.60966128833701]
  We use diagnostic tasks called probes to investigate pre-trained code models. BERT (pre-trained on English), CodeBERT and CodeBERTa (pre-trained on source code, and natural language documentation), and GraphCodeBERT (pre-trained on source code with dataflow) are investigated.
  arXiv  Detail & Related papers  (2021-08-25T16:20:17Z)
• GraphCodeBERT: Pre-training Code Representations with Data Flow [97.00641522327699]
  We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. We use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement.
  arXiv  Detail & Related papers  (2020-09-17T15:25:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.