CoSQA+: Enhancing Code Search Dataset with Matching Code

• URL: http://arxiv.org/abs/2406.11589v2
• Date: Fri, 23 Aug 2024 19:55:52 GMT
• Title: CoSQA+: Enhancing Code Search Dataset with Matching Code
• Authors: Jing Gong, Yanghui Wu, Linxi Liang, Zibin Zheng, Yanlin Wang,
• Abstract summary: CoSQA+ pairs high-quality queries with multiple suitable codes. CoSQA+ has demonstrated superior quality over CoSQA. We propose a new metric to assess one-to-N code search performance.
• Score: 27.10957318333608
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets are problematic: either using unrealistic queries, or with mismatched codes, and typically using one-to-one query-code pairing, which fails to reflect the reality that a query might have multiple valid code matches. This paper introduces CoSQA+, pairing high-quality queries (reused from CoSQA) with multiple suitable codes. We collect code candidates from diverse sources and form candidate pairs by pairing queries with these codes. Utilizing the power of large language models (LLMs), we automate pair annotation, filtering, and code generation for queries without suitable matches. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. Furthermore, we propose a new metric Mean Multi-choice Reciprocal Rank (MMRR), to assess one-to-N code search performance. We provide the code and data at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus.

Related papers

• KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding [49.56049319037421]
  KodCode is a synthetic dataset that addresses the persistent challenge of acquiring high-quality, verifiable training data. It comprises question-solution-test triplets that are systematically validated via a self-verification procedure. This pipeline yields a large-scale, robust and diverse coding dataset.
  arXiv  Detail & Related papers  (2025-03-04T19:17:36Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• You Augment Me: Exploring ChatGPT-based Data Augmentation for Semantic Code Search [47.54163552754051]
  Code search plays a crucial role in software development, enabling developers to retrieve and reuse code using natural language queries. Recently, large language models (LLMs) have made remarkable progress in both natural and programming language understanding and generation. We propose a novel approach ChatDANCE, which utilizes high-quality and diverse augmented data generated by a large language model.
  arXiv  Detail & Related papers  (2024-08-10T12:51:21Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering [16.394601658945625]
  Code Community Question Answering (CCQA) seeks to tackle programming-related issues, thereby boosting productivity in software engineering and academic research. Recent advancements in Reinforcement Learning from Human Feedback (RLHF) have transformed the fine-tuning process of Large Language Models (LLMs) to produce responses that closely mimic human behavior. We propose a novel framework called Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering (ALMupQA) to create user-focused responses.
  arXiv  Detail & Related papers  (2024-05-27T14:21:31Z)
• ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search [8.700556381819267]
  We introduce ProCQA, a large-scale programming question answering dataset extracted from the StackOverflow community. We propose a modality-agnostic contrastive pre-training approach to improve the alignment of text and code representations of current code language models.
  arXiv  Detail & Related papers  (2024-03-25T12:34:33Z)
• Enhancing Code Intelligence Tasks with ChatGPT [17.712126698173535]
  ChatGPT-generated comments demonstrate superior semantic consistency with the code compared to human references. We rebuild the widely used dataset, CodeSearchNet, with ChatGPT-generated comments. Results show that the model pre-trained by ChatGPT-enhanced data outperforms its counterpart on code summarization, code generation, and code translation tasks.
  arXiv  Detail & Related papers  (2023-12-23T09:01:08Z)
• SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References [73.67707138779245]
  We propose a new evaluation metric: SQuArE (Sentence-level QUestion AnsweRing Evaluation) We evaluate SQuArE on both sentence-level extractive (Answer Selection) and generative (GenQA) QA systems.
  arXiv  Detail & Related papers  (2023-09-21T16:51:30Z)
• Synthesizing Conjunctive Queries for Code Search [9.146394499214672]
  Squid is a new conjunctive query algorithm for searching code with target patterns. Squid successfully synthesizes the conjunctive queries for all the tasks, taking only 2.56 seconds on average.
  arXiv  Detail & Related papers  (2023-05-07T15:54:10Z)
• Generation-Augmented Query Expansion For Code Retrieval [51.20943646688115]
  We propose a generation-augmented query expansion framework. Inspired by the human retrieval process - sketching an answer before searching. We achieve new state-of-the-art results on the CodeSearchNet benchmark.
  arXiv  Detail & Related papers  (2022-12-20T23:49:37Z)
• RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering [87.18962441714976]
  We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA) We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
  arXiv  Detail & Related papers  (2022-10-25T21:39:36Z)
• Enhancing Semantic Code Search with Multimodal Contrastive Learning and Soft Data Augmentation [50.14232079160476]
  We propose a new approach with multimodal contrastive learning and soft data augmentation for code search. We conduct extensive experiments to evaluate the effectiveness of our approach on a large-scale dataset with six programming languages.
  arXiv  Detail & Related papers  (2022-04-07T08:49:27Z)
• Learning Deep Semantic Model for Code Search using CodeSearchNet Corpus [17.6095840480926]
  We propose a novel deep semantic model which makes use of the utilities of multi-modal sources. We apply the proposed model to tackle the CodeSearchNet challenge about semantic code search. Our model is trained on CodeSearchNet corpus and evaluated on the held-out data, the final model achieves 0.384 NDCG and won the first place in this benchmark.
  arXiv  Detail & Related papers  (2022-01-27T04:15:59Z)
• CodeRetriever: Unimodal and Bimodal Contrastive Learning [128.06072658302165]
  We propose the CodeRetriever model, which combines the unimodal and bimodal contrastive learning to train function-level code semantic representations. For unimodal contrastive learning, we design a semantic-guided method to build positive code pairs based on the documentation and function name. For bimodal contrastive learning, we leverage the documentation and in-line comments of code to build text-code pairs.
  arXiv  Detail & Related papers  (2022-01-26T10:54:30Z)
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering [63.92224685262063]
  CoSQA dataset includes 20,604 labels for pairs of natural language queries and codes. We introduce a contrastive learning method dubbed CoCLR to enhance query-code matching. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%.
  arXiv  Detail & Related papers  (2021-05-27T15:37:21Z)
• COSEA: Convolutional Code Search with Layer-wise Attention [90.35777733464354]
  We propose a new deep learning architecture, COSEA, which leverages convolutional neural networks with layer-wise attention to capture the code's intrinsic structural logic. COSEA can achieve significant improvements over state-of-the-art methods on code search tasks.
  arXiv  Detail & Related papers  (2020-10-19T13:53:38Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.