Towards Benchmarking Design Pattern Detection Under Obfuscation: Reproducing and Evaluating Attention-Based Detection Method

• URL: http://arxiv.org/abs/2512.07193v1
• Date: Mon, 08 Dec 2025 06:10:34 GMT
• Title: Towards Benchmarking Design Pattern Detection Under Obfuscation: Reproducing and Evaluating Attention-Based Detection Method
• Authors: Manthan Shenoy, Andreas Rausch,
• Abstract summary: We reproduce the DPDAtt, an attention-based design pattern detection approach using learning-based classifiers, and evaluate its performance under obfuscation.<n>Our findings reveal that these trained classifiers depend significantly on superficial syntactic features, leading to substantial misclassification when such cues are removed.<n>This work highlights the need for more robust detection tools capable of capturing deeper semantic meanings in source code.
• Score: 2.1843439591862333
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper investigates the semantic robustness of attention-based classifiers for design pattern detection, particularly focusing on their reliance on structural and behavioral semantics. We reproduce the DPDAtt, an attention-based design pattern detection approach using learning-based classifiers, and evaluate its performance under obfuscation. To this end, we curate an obfuscated version of the DPDAtt Corpus, where the name identifiers in code such as class names, method names, etc., and string literals like print statements and comment blocks are replaced while preserving control flow, inheritance, and logic. Our findings reveal that these trained classifiers in DPDAtt depend significantly on superficial syntactic features, leading to substantial misclassification when such cues are removed through obfuscation. This work highlights the need for more robust detection tools capable of capturing deeper semantic meanings in source code. We propose our curated Obfuscated corpus (containing 34 Java source files) as a reusable proof-of-concept benchmark for evaluating state-of-the-art design pattern detectors on their true semantic generalization capabilities.

Related papers

• PADBen: A Comprehensive Benchmark for Evaluating AI Text Detectors Against Paraphrase Attacks [2.540711742769252]
  We investigate why iteratively-paraphrased text evades detection systems designed for AIGT identification.<n>We introduce PADBen, the first benchmark systematically evaluating detector robustness against paraphrase attack scenarios.
  arXiv  Detail & Related papers  (2025-11-01T05:59:46Z)
• When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection [64.23509202768945]
  We introduce dataset, the first benchmark for evaluating detector robustness in personalized settings.<n>Our experimental results demonstrate large performance gaps across detectors in personalized settings.<n>We propose method, a simple and reliable way to predict detector performance changes in personalized settings.
  arXiv  Detail & Related papers  (2025-10-14T13:10:23Z)
• When Names Disappear: Revealing What LLMs Actually Understand About Code [7.691597373321699]
  Large Language Models (LLMs) achieve strong results on code tasks, but how they derive program meaning remains unclear.<n>We argue that code communicates through two channels: structural semantics, which define formal behavior, and human-interpretable naming, which conveys intent.<n>Removing the naming channel severely degrades intent-level tasks such as summarization, where models regress to line-by-line descriptions.
  arXiv  Detail & Related papers  (2025-10-03T16:53:13Z)
• Command-line Obfuscation Detection using Small Language Models [0.7373617024876725]
  adversaries often use command-line obfuscation to avoid detection. We have implemented a scalable NLP-based detection method that leverages a custom-trained, small transformer language model. We show the model's superiority to signatures on established malware and showcase previously unseen obfuscated samples detected by our model.
  arXiv  Detail & Related papers  (2024-08-05T17:01:33Z)
• EditSum: A Retrieve-and-Edit Framework for Source Code Summarization [46.84628094508991]
  Existing studies show that code summaries help developers understand and maintain source code. Code summarization aims to generate natural language descriptions automatically for source code. This paper proposes a novel retrieve-and-edit approach named EditSum for code summarization.
  arXiv  Detail & Related papers  (2023-08-26T05:48:57Z)
• Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos [63.94040814459116]
  Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence. We propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps. We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations.
  arXiv  Detail & Related papers  (2023-08-19T09:12:13Z)
• ConTextual Mask Auto-Encoder for Dense Passage Retrieval [49.49460769701308]
  CoT-MAE is a simple yet effective generative pre-training method for dense passage retrieval. It learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding. We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines.
  arXiv  Detail & Related papers  (2022-08-16T11:17:22Z)
• Span Classification with Structured Information for Disfluency Detection in Spoken Utterances [47.05113261111054]
  We propose a novel architecture for detecting disfluencies in transcripts from spoken utterances. Our proposed model achieves state-of-the-art results on the widely used English Switchboard for disfluency detection.
  arXiv  Detail & Related papers  (2022-03-30T03:22:29Z)
• MASKER: Masked Keyword Regularization for Reliable Text Classification [73.90326322794803]
  We propose a fine-tuning method, coined masked keyword regularization (MASKER), that facilitates context-based prediction. MASKER regularizes the model to reconstruct the keywords from the rest of the words and make low-confidence predictions without enough context. We demonstrate that MASKER improves OOD detection and cross-domain generalization without degrading classification accuracy.
  arXiv  Detail & Related papers  (2020-12-17T04:54:16Z)
• Detection of Adversarial Supports in Few-shot Classifiers Using Feature Preserving Autoencoders and Self-Similarity [89.26308254637702]
  We propose a detection strategy to highlight adversarial support sets. We make use of feature preserving autoencoder filtering and also the concept of self-similarity of a support set to perform this detection. Our method is attack-agnostic and also the first to explore detection for few-shot classifiers to the best of our knowledge.
  arXiv  Detail & Related papers  (2020-12-09T14:13:41Z)
• Adversarial Semantic Collisions [129.55896108684433]
  We study semantic collisions: texts that are semantically unrelated but judged as similar by NLP models. We develop gradient-based approaches for generating semantic collisions. We show how to generate semantic collisions that evade perplexity-based filtering.
  arXiv  Detail & Related papers  (2020-11-09T20:42:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.