CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs

• URL: http://arxiv.org/abs/2609.01161v1
• Date: Tue, 01 Sep 2026 12:43:02 GMT
• Title: CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs
• Abstract summary: We introduce CopyShield, a benchmark comparing three representative defenses at distinct intervention levels.<n>We evaluate CopyShield on two model families, LLaMA-3.1-8B and Mistral-7B-v0.3, using controlled memorization over five public-domain books.
• Score: 10.232931203118614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models can reproduce memorized text verbatim, yet copyright defenses are usually evaluated under incompatible protocols. We introduce CopyShield, a controlled benchmark comparing three representative defenses at distinct intervention levels: contrastive decoding (output), Direct Preference Optimization (behavioral), and activation intervention (representation). We evaluate CopyShield on two model families, LLaMA-3.1-8B and Mistral-7B-v0.3, using controlled memorization over five public-domain books and a shared protocol measuring literal leakage, calibrated non-literal leakage, utility, and degeneracy. Across these methods, intervention level is associated with distinct compliance-utility trade-offs. On LLaMA-3.1-8B, contrastive decoding remains near-degeneracy-free (0-2%) but reaches a literal-suppression floor at NV-Recall 0.192-0.203. DPO nearly eliminates literal leakage (0.263 to 0.002) but induces paraphrase-loop degeneracy in 58% of QA outputs, with no utility gain over the SFT baseline. Activation intervention attains the lowest non-literal flagging rate (1/200) by blocking 84% of non-literal queries before generation. Human evaluation confirms that DPO has low coherence, whereas activation lowers perceived copyright risk through broad refusal. On Mistral-7B-v0.3, the output- and representation-level patterns persist, while DPO degeneracy falls to 10-14%, showing that its severity is model-dependent. Together, CopyShield provides cross-level reference baselines and identifies targeted non-literal suppression as an open challenge. The code is available at https://github.com/spotai-mbzuai/CopyShield.git.

Related papers

• LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation [48.83701310501069]
  Large language models (LLMs) can generate neural-network modifications, but unrestricted generation is often invalid or harmful.<n>This paper studies a narrower setting: improving a weak target model using a stronger same-family source model from a neural-network database.<n>We propose a source-guided candidate-generation protocol with non-source controls, source-conditioned candidates, and a no-LLM hp_copy ablation under equal evaluation budgets.
  arXiv  Detail & Related papers  (2026-07-06T23:59:18Z)
• A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots [1.948261185683419]
  We present a three-layer framework that intercepts both direct and indirect prompt injection throughout the inference pipeline.<n> Evaluation on 5,080 samples across GPT-4o, Llama 3, and Mistral 7B shows that the framework reduces Attack Success Rate (ASR) from 71.4% to 11.3%.
  arXiv  Detail & Related papers  (2026-06-17T23:59:57Z)
• Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs [52.149036302760386]
  Large language models can be fine-tuned to encode prompt-borne secrets into fluent, seemingly benign outputs.<n>Recent work proposes detection using linear probes that recover the secret from internal activations.<n>We show that this defense can be systematically evaded, but that detectability can be recovered through a targeted data-level intervention.
  arXiv  Detail & Related papers  (2026-06-08T12:27:11Z)
• Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration [3.9458538778648964]
  This paper is a preliminary study of abliteration, a low-rank weight edit that replicates out the refusal stream as a tool to remove this barrier.<n> Vulnerability verdicts are produced by a three-tool detector ensemble (QLCode, Sem, Bandit) followed by manual adjudication.
  arXiv  Detail & Related papers  (2026-06-03T20:06:52Z)
• SilentRetrieval: Hijacking Retrieval-Augmented Generation via Semantically-Preserving Adversarial Data Poisoning [1.0998907972211756]
  We present SilentRetrieval, a two-stage data poisoning attack that hijacks RAG systems through adversarially crafted documents.<n>SilentRetrieval achieves 84.6%/81.3% HR@10 and 57.5%/54.8% ASR-LLM on Natural Questions and MS MARCO, while maintaining near-benign perplexity.
  arXiv  Detail & Related papers  (2026-05-27T07:30:30Z)
• LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests [0.0]
  Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed.<n>We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests.<n>We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API.
  arXiv  Detail & Related papers  (2026-04-13T21:05:42Z)
• CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attacks [48.54598003197356]
  State space models (SSMs) like Mamba have gained significant traction as efficient alternatives to Transformers.<n>HiSPAs, a recently discovered vulnerability that corrupts SSM memory through adversarial strings, pose a critical threat to these architectures.<n>We introduce the CLASP model to defend against this threat.
  arXiv  Detail & Related papers  (2026-03-12T17:29:55Z)
• Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model [99.16364381244445]
  Modern language models (LMs) tend to memorize portions of their training data and emit verbatim spans.<n>We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying.<n>We evaluate our methods across six model pairs on long-form evaluations of copyright risk and utility.
  arXiv  Detail & Related papers  (2026-02-06T19:00:14Z)
• SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking [58.475471437150674]
  We propose sequential watermarking for soft prompts (SWAP)<n>SWAP encodes watermarks through a specific order of defender-specified out-of-distribution classes.<n>Experiments on 11 datasets demonstrate SWAP's effectiveness, harmlessness, and robustness against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-11-05T13:48:48Z)
• Obliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Models [2.7174461714624805]
  We introduce Obliviate, a lightweight method that surgically suppresses exact reproduction of specified sequences.<n>Obliviate first identifies memorized passages and then, for each target token, minimally adjusts the model's output distribution.<n>We evaluate Obliviate on four popular 6-8B- parameter models (LLaMA-3.1, LLaMA-3.1-Instruct, Qwen-2.5, and Yi-1.5) using synthetic benchmarks and organic copyrighted excerpts.
  arXiv  Detail & Related papers  (2025-02-20T20:02:56Z)
• CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation [132.00910067533982]
  We introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters.
  arXiv  Detail & Related papers  (2024-07-09T17:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.