Do System Prompts Leave Behavioral Fingerprints? A Large-Scale Empirical Study of Clone Detection via Output Similarity

• URL: http://arxiv.org/abs/2608.24461v1
• Date: Tue, 25 Aug 2026 12:12:22 GMT
• Title: Do System Prompts Leave Behavioral Fingerprints? A Large-Scale Empirical Study of Clone Detection via Output Similarity
• Abstract summary: We propose Black-Box Behavioral Fingerprinting (BBF) for system prompts.<n>BBF registers a behavioral signature from model outputs and later tests whether a suspect deployment matches that signature.<n>BBF resists non-adaptive prompt paraphrasing (AUC $geq 0.889$) and is robust to imperfect extraction.
• Score: 12.32748486531306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: System prompts can be extracted from commercial LLMs with over 80\% success and redeployed at zero cost, yet a prompt owner has no way to verify whether a suspected deployment is a clone. We propose Black-Box Behavioral Fingerprinting (BBF): the prompt owner registers a behavioral signature from model outputs and later tests whether a suspect deployment matches that signature more closely than an unrelated baseline. BBF requires only black-box API access. Through a large-scale study (4 model families, 8 benchmarks, 288{,}000 responses), we find that prompt choice explains 24.4\% of output variance and same-model detection reaches AUC 0.876. Cross-model performance is bounded by detector identity, with off-diagonal AUC ranging from 0.845 (Claude as detector) down to 0.665 (Qwen) and overall mean 0.725. BBF resists non-adaptive prompt paraphrasing (AUC $\geq 0.889$) and is robust to imperfect extraction, but a single-sentence formal-tone prefix can collapse detection on short structured outputs (MNLI 0.978 $\to$ 0.547), isolating style-invariant detection as the key open problem. Diagnostic Query Optimization, a zero-cost query selection rule, adds $+0.120$ to cross-model AUC.

Related papers

• The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics [0.0]
  Large language models are increasingly used in blockchain forensic investigations.<n> adversarial deception reduces drain detection by 20.0 percentage points.<n>Models correctly describe the hidden drain mechanism but accept the contract's framing and dismiss it as benign.
  arXiv  Detail & Related papers  (2026-09-12T18:45:24Z)
• AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes [38.37599802008238]
  Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped.<n>All existing audits decide backbone identity from the text-output channel.<n>We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs.
  arXiv  Detail & Related papers  (2026-08-30T08:22:12Z)
• Selective Disclosure of Hidden Directives in Reasoning Models: Behavioral Asymmetry and Steering [0.9474649136535703]
  Chain-of-thought (CoT) reasoning traces are increasingly proposed as a mechanism for AI oversight.<n>This assumes CoT surfaces what a model is instructed to do regardless of instructions given.<n>We test this assumption along two axes.
  arXiv  Detail & Related papers  (2026-08-29T05:50:54Z)
• A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use [28.57781785937644]
  We introduce PRISMS (Probing Representations In Support of Monitoring and Steering), a closed-loop framework that shares a failure-specific neuron basis between sparse detection and activation.<n>Across six models, PRISMS reduces pooled over-calling rate by 80% while increasing tool-required accuracy by 14.2 percentage points.
  arXiv  Detail & Related papers  (2026-07-31T19:03:44Z)
• Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference [22.335515110971475]
  Visual-token pruning is judged by answer quality at a fixed retention budget.<n>We turn this blind spot into an evidence-risk audit that couples answer behavior with geometric token-origin provenance.<n>We show that favorable target-verification points do not imply task-general compression.
  arXiv  Detail & Related papers  (2026-07-29T13:45:31Z)
• Level, Sharpness, and Corpus: Why Zero-Shot OOD Detector Rankings Do Not Transfer [36.02174685737611]
  We find that detector rankings reverse across deployments, every detector exceeds $80%$ FPR95 on at least one domain, and the preferred detector depends on both the in-distribution data and the underlying VLM.<n>We introduce the Complementary Evidence Guard (CEG), a detector-agnostic wrapper that preserves complementary evidence through a non-compensatory fusion of the base detector, level, and sharpness.
  arXiv  Detail & Related papers  (2026-07-29T08:00:49Z)
• Sensitivity to Subjective Expected Utility Maximization: A Methodological Study, with an Illustrative Application to LLM Decision-Making [0.0]
  Vehicle is a softmax choice model with a sensitivity parameter $$ on SEU-valued alternatives.<n>The contribution is a sequence of identifiability results for $$ and for belief and utility parameters $(, )$, validated in Stansberg.<n>A two-by-two application runs end-to-end on real choice data, detecting a structured comparative $$ effect sampling in two of four cells.
  arXiv  Detail & Related papers  (2026-07-08T00:29:18Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing [29.035258104995204]
  We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
  arXiv  Detail & Related papers  (2026-05-13T18:01:05Z)
• SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio [0.038379177968040606]
  We propose SELFDOUBT, a single-pass uncertainty framework for reasoning language models.<n>Our key signal, the Hedge-to-Verify Ratio (HVR), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit selfchecking behavior.<n> SELFDOUBT operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API.
  arXiv  Detail & Related papers  (2026-04-07T19:19:29Z)
• AttnDiff: Attention-based Differential Fingerprinting for Large Language Models [17.46604955122771]
  textscAttnDiff extracts fingerprints from models via intrinsic information-routing behavior.<n>textscAttnDiff probes minimally edited prompt pairs that induce controlled semantic conflicts.
  arXiv  Detail & Related papers  (2026-04-07T06:57:47Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature [33.84718903572477]
  Large language models (LLMs) have shown the ability to produce fluent and cogent content.<n>To build trustworthy AI systems, it is imperative to distinguish between machine-generated and human-authored content.<n>Fast-DetectGPT is an optimized zero-shot detector that substitutes DetectGPT's perturbation step with a more efficient sampling step.
  arXiv  Detail & Related papers  (2023-10-08T11:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.