How the Misuse of a Dataset Harmed Semantic Clone Detection

• URL: http://arxiv.org/abs/2505.04311v1
• Date: Wed, 07 May 2025 10:52:28 GMT
• Title: How the Misuse of a Dataset Harmed Semantic Clone Detection
• Abstract summary: This paper demonstrates that BigCloneBench is problematic to use as ground truth for learning or evaluating semantic code similarity.<n>In a literature review of 179 papers that use BigCloneBench as a dataset, we found 139 papers that used BigCloneBench to evaluate semantic clone detection.<n>We emphasise that using BigCloneBench remains valid for the intended purpose of evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones.
• Score: 0.9361474110798144
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: BigCloneBench is a well-known and widely used large-scale dataset for the evaluation of recall of clone detection tools. It has been beneficial for research on clone detection and has become a standard in evaluating the performance of clone detection tools. More recently, it has also been widely used as a dataset to evaluate machine learning approaches to semantic clone detection or code similarity detection for functional or semantic similarity. This paper demonstrates that BigCloneBench is problematic to use as ground truth for learning or evaluating semantic code similarity, and highlights the aspects of BigCloneBench that affect the ground truth quality. A manual investigation of a statistically significant random sample of 406 Weak Type-3/Type-4 clone pairs revealed that 93% of them do not have a similar functionality and are therefore mislabelled. In a literature review of 179 papers that use BigCloneBench as a dataset, we found 139 papers that used BigCloneBench to evaluate semantic clone detection and where the results are threatened in their validity by the mislabelling. As such, these papers often report high F1 scores (e.g., above 0.9), which indicates overfitting to dataset-specific artefacts rather than genuine semantic similarity detection. We emphasise that using BigCloneBench remains valid for the intended purpose of evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge the important contributions of BigCloneBench to two decades of traditional clone detection research. However, the usage of BigCloneBench beyond the intended purpose without careful consideration of its limitations has led to misleading results and conclusions, and potentially harmed the field of semantic clone detection.

Related papers

• Semantic Code Clone Detection: Are We There Yet? [30.160018956984853]
  Code clone detection has been extensively studied for decades.<n>Recent approaches have reported remarkably high performance for semantic (Type-4) clones on benchmark datasets.<n>We present the first systematic empirical study investigating the generalizability of state-of-the-art semantic code clone detectors.
  arXiv  Detail & Related papers  (2026-06-24T01:11:40Z)
• Identifying and Characterizing Semantic Clones of Solidity Functions [4.407571766292332]
  We present and empirically assess a scalable methodology, based on analyzing code and comments, to spot semantically equivalent Solidity functions.<n>Our results show that LLM-generated summaries, combined with sentence transformers like BERT, can bridge the documentation gap.<n>This work establishes a modern benchmark for Solidity clone detection and provides a foundation for the automated discovery of secure and efficient code alternatives.
  arXiv  Detail & Related papers  (2026-04-29T10:49:20Z)
• Are Classical Clone Detectors Good Enough For the AI Era? [5.2308433103996625]
  Increasing adoption of AI-generated code has reshaped modern software development.<n> AI-generated clones exhibit systematic syntactic patterns and semantic differences learned from large-scale training data.<n>Classical code clone detection tools have historically been validated primarily on human-authoreds.
  arXiv  Detail & Related papers  (2025-09-30T04:19:17Z)
• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property.<n>We propose a novel dataset ownership verification method.<n>Our approach introduces a clustering-based backdoor watermark (CBW)<n>We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• Fuzzy Granule Density-Based Outlier Detection with Multi-Scale Granular Balls [65.44462297594308]
  Outlier detection refers to the identification of anomalous samples that deviate significantly from the distribution of normal data.<n>Most unsupervised outlier detection methods are carefully designed to detect specified outliers.<n>We propose a fuzzy rough sets-based multi-scale outlier detection method to identify various types of outliers.
  arXiv  Detail & Related papers  (2025-01-06T12:35:51Z)
• On the Use of Deep Learning Models for Semantic Clone Detection [4.796947520072581]
  We propose a multi-step evaluation approach for five state-of-the-art clone detection models leveraging existing benchmark datasets.<n>Specifically, we examine three highly-performing single-language models (ASTNN, GMN, CodeBERT) on BigCloneBench, SemanticCloneBench, and GPTCloneBench.<n>While single-language models show high F1 scores for BigCloneBench, their performance on SemanticCloneBench varies (up to 20%)<n>Interestingly, the cross-language model (C4) shows superior performance (around 7%) on SemanticCloneBench over other models.
  arXiv  Detail & Related papers  (2024-12-19T11:15:02Z)
• C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection [98.34703790782254]
  We introduce Category Common Prompt CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder.<n>Our method achieves a 12.41% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing.
  arXiv  Detail & Related papers  (2024-08-19T02:14:25Z)
• SimClone: Detecting Tabular Data Clones using Value Similarity [37.85935189975307]
  Presence of data clones between datasets can cause issues when using datasets with clones to build AI software. We propose a novel method called SimClone for data clone detection in tabular datasets without relying on structural information. Our results show that our SimClone outperforms the current state-of-the-art method by at least 20% in terms of both F1-score and AUC.
  arXiv  Detail & Related papers  (2024-06-24T04:16:32Z)
• Using a Nearest-Neighbour, BERT-Based Approach for Scalable Clone Detection [0.0]
  SSCD is a BERT-based clone detection approach that targets high recall of Type 3 and Type 4 clones at scale. It does so by computing a representative embedding for each code fragment and finding similar fragments using a nearest neighbour search. This paper details the approach and an empirical assessment towards configuring and evaluating that approach in industrial setting.
  arXiv  Detail & Related papers  (2023-09-05T12:38:55Z)
• GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBench [1.8687918300580921]
  We present a comprehensive semantic clone and cross-language clone benchmark, GPTCloneBench by exploiting SemanticCloneBench and OpenAI's GPT-3 model. From 79,928 clone pairs of GPT-3 output, we created a benchmark with 37,149 true semantic clone pairs, 19,288 false semantic pairs(Type-1/Type-2), and 20,770 cross-language clones across four languages (Java, C, C#, and Python)
  arXiv  Detail & Related papers  (2023-08-26T21:50:34Z)
• Evaluation of Contrastive Learning with Various Code Representations for Code Clone Detection [3.699097874146491]
  We evaluate contrastive learning for detecting semantic clones of code snippets. We use CodeTransformator to create a dataset that mimics plagiarised code based on competitive programming solutions. The results of our evaluation show that proposed models perform diversely in each task, however the performance of the graph-based models is generally above the others.
  arXiv  Detail & Related papers  (2022-06-17T12:25:44Z)
• ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning [52.831894583501395]
  Continual learning assumes the incoming data are fully labeled, which might not be applicable in real applications. We propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN) We show ORDisCo achieves significant performance improvement on various semi-supervised learning benchmark datasets for SSCL.
  arXiv  Detail & Related papers  (2021-01-02T09:04:14Z)
• Detection of Adversarial Supports in Few-shot Classifiers Using Feature Preserving Autoencoders and Self-Similarity [89.26308254637702]
  We propose a detection strategy to highlight adversarial support sets. We make use of feature preserving autoencoder filtering and also the concept of self-similarity of a support set to perform this detection. Our method is attack-agnostic and also the first to explore detection for few-shot classifiers to the best of our knowledge.
  arXiv  Detail & Related papers  (2020-12-09T14:13:41Z)
• Semantic Clone Detection via Probabilistic Software Modeling [69.43451204725324]
  This article contributes a semantic clone detection approach that detects clones that have 0% syntactic similarity. We present SCD-PSM as a stable and precise solution to semantic clone detection.
  arXiv  Detail & Related papers  (2020-08-11T17:54:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.