CoSQA+: Enhancing Code Search Dataset with Matching Code

• URL: http://arxiv.org/abs/2406.11589v2
• Date: Fri, 23 Aug 2024 19:55:52 GMT
• Title: CoSQA+: Enhancing Code Search Dataset with Matching Code
• Authors: Jing Gong, Yanghui Wu, Linxi Liang, Zibin Zheng, Yanlin Wang,
• Abstract summary: CoSQA+ pairs high-quality queries with multiple suitable codes. CoSQA+ has demonstrated superior quality over CoSQA. We propose a new metric to assess one-to-N code search performance.
• Score: 27.10957318333608
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets are problematic: either using unrealistic queries, or with mismatched codes, and typically using one-to-one query-code pairing, which fails to reflect the reality that a query might have multiple valid code matches. This paper introduces CoSQA+, pairing high-quality queries (reused from CoSQA) with multiple suitable codes. We collect code candidates from diverse sources and form candidate pairs by pairing queries with these codes. Utilizing the power of large language models (LLMs), we automate pair annotation, filtering, and code generation for queries without suitable matches. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. Furthermore, we propose a new metric Mean Multi-choice Reciprocal Rank (MMRR), to assess one-to-N code search performance. We provide the code and data at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus.

Related papers

• UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance [65.01483640267885]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet code generation remains a major challenge. We introduce UnitCoder, a systematic pipeline leveraging model-generated unit tests to guide and validate the code generation process. Our work presents a scalable approach that leverages model-generated unit tests to guide the synthesis of high-quality code data from pre-training corpora.
  arXiv  Detail & Related papers  (2025-02-17T05:37:02Z)
• CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering [12.431784613373523]
  We introduce CoReQA, a benchmark for Code Repository-level question answering. CoReQA was constructed from GitHub issues and comments from 176 popular repositories across four programming languages. We show that state-of-the-art proprietary and long-context models struggle to address repository-level questions effectively.
  arXiv  Detail & Related papers  (2025-01-07T00:24:07Z)
• You Augment Me: Exploring ChatGPT-based Data Augmentation for Semantic Code Search [47.54163552754051]
  Code search plays a crucial role in software development, enabling developers to retrieve and reuse code using natural language queries. Recently, large language models (LLMs) have made remarkable progress in both natural and programming language understanding and generation. We propose a novel approach ChatDANCE, which utilizes high-quality and diverse augmented data generated by a large language model.
  arXiv  Detail & Related papers  (2024-08-10T12:51:21Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering [16.394601658945625]
  Code Community Question Answering (CCQA) seeks to tackle programming-related issues, thereby boosting productivity in software engineering and academic research. Recent advancements in Reinforcement Learning from Human Feedback (RLHF) have transformed the fine-tuning process of Large Language Models (LLMs) to produce responses that closely mimic human behavior. We propose a novel framework called Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering (ALMupQA) to create user-focused responses.
  arXiv  Detail & Related papers  (2024-05-27T14:21:31Z)
• ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search [8.700556381819267]
  We introduce ProCQA, a large-scale programming question answering dataset extracted from the StackOverflow community. We propose a modality-agnostic contrastive pre-training approach to improve the alignment of text and code representations of current code language models.
  arXiv  Detail & Related papers  (2024-03-25T12:34:33Z)
• SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References [73.67707138779245]
  We propose a new evaluation metric: SQuArE (Sentence-level QUestion AnsweRing Evaluation) We evaluate SQuArE on both sentence-level extractive (Answer Selection) and generative (GenQA) QA systems.
  arXiv  Detail & Related papers  (2023-09-21T16:51:30Z)
• RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering [87.18962441714976]
  We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA) We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
  arXiv  Detail & Related papers  (2022-10-25T21:39:36Z)
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering [63.92224685262063]
  CoSQA dataset includes 20,604 labels for pairs of natural language queries and codes. We introduce a contrastive learning method dubbed CoCLR to enhance query-code matching. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%.
  arXiv  Detail & Related papers  (2021-05-27T15:37:21Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.