TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs

• URL: http://arxiv.org/abs/2605.24079v1
• Date: Fri, 22 May 2026 17:30:20 GMT
• Title: TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs
• Abstract summary: We present TRACER, a semantic-aware framework for fine-grained code contamination detection.<n> TRACER models contamination using three levels of semantic overlap - Functionally Identical, Nearly Identical, and Shared Logic.<n>We also introduce the first benchmark for fine-grained code contamination detection, spanning three widely used benchmarks and three representative post-training datasets.
• Score: 5.739404573725206
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data contamination is a known threat to the reliability of model evaluation. However, it remains underexplored in code large language models (LLMs), where contamination often goes beyond exact duplication. We present TRACER, a semantic-aware framework for fine-grained code contamination detection. TRACER models contamination using three levels of semantic overlap - Functionally Identical, Nearly Identical, and Shared Logic - and detects them through a coarse-to-fine pipeline. We also introduce the first benchmark for fine-grained code contamination detection, spanning three widely used benchmarks and three representative post-training datasets. TRACER achieves strong and consistent performance across multiple LLM backbones, with GPT-5 reaching an F1 score of 0.91 in fine-grained detection. In the binary setting, TRACER attains an F1 of 0.92, outperforming existing methods by 42%-217%. We further conduct ablation studies and error analysis to assess the contributions of individual components in TRACER.

Related papers

• Robustness of Anomaly Detection Models for Industrial Control Systems under Training-Time Data Contamination [3.7391021139771325]
  This paper evaluates the robustness of offline ICS anomaly-detection pipelines on the Secure Water Treatment benchmark under training-time contamination.<n>We assess 11 heterogeneous anomaly detectors under three contamination strategies: random injection, similarity-targeted injection, and feature-noise injection.<n>The results show that robustness is strongly model-dependent and cannot be predicted from clean-data performance alone.
  arXiv  Detail & Related papers  (2026-08-24T17:49:36Z)
• The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation [28.17279842216215]
  Large language models (LLMs) have demonstrated impressive reasoning abilities across a wide range of tasks.<n>Data contamination undermines the objective evaluation of these capabilities.<n>Current approaches struggle to reliably detect such stealthy contamination.<n>We propose the Zero-CoT Probe (ZCP), a novel black-box detection method.
  arXiv  Detail & Related papers  (2026-05-21T01:06:19Z)
• LLM-Enhanced Log Anomaly Detection: A Comprehensive Benchmark of Large Language Models for Automated System Diagnostics [0.0]
  Large Language Models (LLMs) offer promising new approaches to log understanding.<n>LLMs demonstrate remarkablezero-shot capabilities without requiring any labeled training data.<n>Our findings provide actionable guidelines for choosing log anomaly detection methods.
  arXiv  Detail & Related papers  (2026-04-14T02:51:48Z)
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation [73.76465227729818]
  Open-source Vision-Language Models (VLMs) have achieved state-of-the-art performance on benchmark tasks.<n>Pretraining corpora raise a critical concern for both practitioners and users: inflated performance due to test-set leakage.<n>We show that existing detection approaches either fail outright or exhibit inconsistent behavior.<n>We propose a novel simple yet effective detection method based on multi-modal semantic perturbation.
  arXiv  Detail & Related papers  (2025-11-05T18:59:52Z)
• CLEANet: Robust and Efficient Anomaly Detection in Contaminated Multivariate Time Series [17.992163690121565]
  CLEANet is a robust and efficient anomaly detection framework in contaminated time series.<n>Cleanet introduces a Contamination-Resilient Training Framework (CRTF)<n>Cleanet mitigates higher F1 and 81.28% lower runtime compared with ten state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-26T10:34:19Z)
• VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination [15.52442661491358]
  Large Language Models (LLMs) have revolutionized code generation, achieving exceptional results on various established benchmarking frameworks.<n>However, concerns about data contamination raise questions about the validity of these evaluations.<n>We analyze state-of-the-art (SOTA) evaluation frameworks for Verilog code generation.
  arXiv  Detail & Related papers  (2025-03-17T12:26:49Z)
• Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM [53.05486269607166]
  multimodal large language models (MLLMs) have significantly enhanced performance across benchmarks.<n>Existing detection methods for unimodal large language models (LLMs) are inadequate for MLLMs due to multimodal data complexity and multi-phase training.<n>We analyze multimodal data contamination using our analytical framework, MM-Detect, which defines two contamination categories-unimodal and cross-modal.
  arXiv  Detail & Related papers  (2024-11-06T10:44:15Z)
• Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection [59.41026558455904]
  We focus on multi-modal anomaly detection. Specifically, we investigate early multi-modal approaches that attempted to utilize models pre-trained on large-scale visual datasets. We propose a Local-to-global Self-supervised Feature Adaptation (LSFA) method to finetune the adaptors and learn task-oriented representation toward anomaly detection.
  arXiv  Detail & Related papers  (2024-01-06T07:30:41Z)
• Rethinking Benchmark and Contamination for Language Models with Rephrased Samples [49.18977581962162]
  Large language models are increasingly trained on all the data ever produced by humans. Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets.
  arXiv  Detail & Related papers  (2023-11-08T17:35:20Z)
• Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection [81.07346419422605]
  Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
  arXiv  Detail & Related papers  (2022-07-24T18:49:26Z)
• Disentangle Your Dense Object Detector [82.22771433419727]
  Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. We propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art detectors.
  arXiv  Detail & Related papers  (2021-07-07T00:52:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.