Asking Back: Interaction-Layer Antidistillation Watermarks

• URL: http://arxiv.org/abs/2605.16462v1
• Date: Fri, 15 May 2026 08:28:35 GMT
• Title: Asking Back: Interaction-Layer Antidistillation Watermarks
• Abstract summary: Existing defenses operate on the teacher's output tokens.<n>Recent work shows a oblivious attacker can strip these signals without losing the underlying knowledge.<n>We propose interaction-layer antidistillation watermarks.
• Score: 7.826668598190874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting unauthorized knowledge distillation from a deployed LLM API is hard because the defender controls neither the attacker's training pipeline nor the next-token logits. Existing defenses operate on the teacher's output tokens -- biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or rewriting outputs after generation. Recent work shows a paraphrasing attacker can strip these signals without losing the underlying knowledge. We propose interaction-layer antidistillation watermarks, which move the trace one layer higher, into the teacher's interaction behavior: the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker -- an explicit follow-up question, a low-frequency variant, or a declarative restatement. An oblivious distiller inherits the behavior, and the defender audits via black-box queries with a human-validated LLM-as-judge (Cohen's kappa = 0.84/0.78 on strong/style rubrics). Across 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher (35,343 judged samples), behavioral watermarks transfer at 88.9% (Gemma) / 80.9% (OLMo) / 45.2% (Qwen) relative fidelity (H1, H2). Under non-adaptive DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling (about 66.4%) and student-relative retention of 21-112%, with OLMo preserving the watermark above the teacher itself (H3, F-Amp). Low-density (about 20%) explicit and implicit declarative variants transfer above per-family baseline (H4, F-Style). An N=20 in-lab study (pre-registered Latin-square) shows all marker variants within 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon support H5. The interaction layer is a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses.

Related papers

• Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement [27.122890248991556]
  On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR)<n>We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale.<n>We find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones.
  arXiv  Detail & Related papers  (2026-08-31T16:22:10Z)
• Learning to Follow In-Context Watermark Instructions via Self-Distillation [117.19153619258644]
  In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response.<n>We introduce $mathsfICWBench$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality.<n>We propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability.
  arXiv  Detail & Related papers  (2026-08-29T03:42:28Z)
• Mismatch Matters: On-Policy Distillation Beyond Token Agreement [27.396104103072844]
  We show that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens.<n>We propose TIDE, which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass.
  arXiv  Detail & Related papers  (2026-08-10T16:53:14Z)
• Visual Contrastive Self-Distillation [61.298159957622886]
  We propose Visual Contrastive Self-Distillation, which converts image-content removal into an on-policy self-distillation signal.<n>We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student.
  arXiv  Detail & Related papers  (2026-07-23T17:37:37Z)
• ADS-C: Antidistillation Sampling for Classification [1.1458853556386797]
  Antidistillation sampling is proposed for large language models.<n>We show its behavior is governed by the distribution of the teacher's per-input confidence margins.<n>Our method, ADS-C, is the first antidistillation defense for classification whose utility cost is exactly zero.
  arXiv  Detail & Related papers  (2026-07-16T21:33:37Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification [5.584060970507507]
  We present a benchmark jointly evaluating federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification.<n>It spans 13 FL algorithms and 10 KD objectives (a 130-pair cross-product) across 504 training runs, evaluated on ModelNet40 and a clinical craniosynostosis dataset.
  arXiv  Detail & Related papers  (2026-06-30T20:12:27Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information [22.436966302243565]
  We propose Anti-Self-Distillation, which ascends a divergence between student and teacher rather than descending it.<n>AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points.
  arXiv  Detail & Related papers  (2026-05-12T06:40:43Z)
• DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation [5.647839536820347]
  We introduce a taxonomy of three defense categories -- output perturbation, data poisoning, and information throttling.<n>We evaluate nine defense configurations using a standardized pipeline with Qwen3-14B as teacher and Qwen2.5-7B-Instruct as student.<n>Our results reveal that, in a same-family distillation setting against a naive attacker, most output-level defenses are surprisingly ineffective.
  arXiv  Detail & Related papers  (2026-03-08T22:38:30Z)
• Entropy-Aware On-Policy Distillation of Language Models [36.60992451188347]
  We introduce Entropy-Aware On-Policy Distillation.<n>Our key idea is augmenting the standard reverse KL objective with forward KL when teacher entropy is high.<n>It balances mode-seeking precision with mode-covering without sacrificing on-policy training efficiency.
  arXiv  Detail & Related papers  (2026-03-07T07:26:18Z)
• Character-Level Perturbations Disrupt LLM Watermarks [64.60090923837701]
  We formalize the system model for Large Language Model (LLM) watermarking.<n>We characterize two realistic threat models constrained on limited access to the watermark detector.<n>We demonstrate character-level perturbations are significantly more effective for watermark removal under the most restrictive threat model.<n> Experiments confirm the superiority of character-level perturbations and the effectiveness of the Genetic Algorithm (GA) in removing watermarks under realistic constraints.
  arXiv  Detail & Related papers  (2025-09-11T02:50:07Z)
• Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation? [75.99961894619986]
  This paper investigates whether student models can acquire the capabilities of teacher models through knowledge distillation while avoiding watermark inheritance.<n>We propose two categories of watermark removal approaches: pre-distillation removal through untargeted and targeted training data paraphrasing (UP and TP), and post-distillation removal through inference-time watermark neutralization (WN)
  arXiv  Detail & Related papers  (2025-02-17T09:34:19Z)
• Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable [70.77600345240867]
  A novel arbitrary-in-arbitrary-out (AIAO) strategy makes watermarks resilient to fine-tuning-based removal. Unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO.
  arXiv  Detail & Related papers  (2024-05-01T12:03:39Z)
• Deep Partition Aggregation: Provable Defense against General Poisoning Attacks [136.79415677706612]
  Adrial poisoning attacks distort training data in order to corrupt the test-time behavior of a classifier. We propose two novel provable defenses against poisoning attacks. DPA is a certified defense against a general poisoning threat model. SS-DPA is a certified defense against label-flipping attacks.
  arXiv  Detail & Related papers  (2020-06-26T03:16:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.