ADS-C: Antidistillation Sampling for Classification

• URL: http://arxiv.org/abs/2607.15467v1
• Date: Thu, 16 Jul 2026 21:33:37 GMT
• Title: ADS-C: Antidistillation Sampling for Classification
• Abstract summary: Antidistillation sampling is proposed for large language models.<n>We show its behavior is governed by the distribution of the teacher's per-input confidence margins.<n>Our method, ADS-C, is the first antidistillation defense for classification whose utility cost is exactly zero.
• Score: 1.1458853556386797
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge distillation enables an adversary to replicate a proprietary classifier by querying its prediction interface and training a surrogate on the returned probability vectors. Antidistillation sampling, proposed for large language models, counters this threat with an input-dependent, gradient-directed perturbation of the served distribution; its transfer to classification has not been studied. Adapting the defense to classification, we show its behavior is governed by the distribution of the teacher's per-input confidence margins. Because well-trained classifiers are severely overconfident, the direct transfer exhibits an inert window: below a closed-form-predictable threshold, it affects neither attacker nor defender; beyond it, the defense undergoes a phase transition and degrades the teacher faster than the attacker's student. Temperature softening rescales the transition in closed form, and every temperature configuration lies on the same unfavorable trade-off curve. Our method, ADS-C, composes the perturbation under a closed-form, per-input margin budget that provably preserves every served top-1 prediction, so the defended teacher's accuracy equals the undefended teacher's identically. Under this guarantee the distilled student still loses 17.4 percentage points on CIFAR-100, 29.6 on CIFAR-10, and 13.3 on Tiny-ImageNet; matching this degradation with the unmodified defense costs 27.5, 32.9, and 22.2 points of teacher accuracy. Because served labels are unchanged, a hard-label attacker gains nothing, while the defended soft output trains a student up to 29.7 points below that floor: the incentive to distill served probabilities is not merely removed but reversed. To our knowledge, ADS-C is the first antidistillation defense for classification whose utility cost is exactly zero.

Related papers

• Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall [79.59596652272923]
  We propose Switch Distillation, a simple mid-training objective that distills on tokens where the teacher is confident.<n>Relative to standard NTP, it achieves 1.61-1.71x the reasoning performance and 1.13-1.19x the knowledge and commonsense performance.
  arXiv  Detail & Related papers  (2026-09-01T17:00:30Z)
• CW-BASS v2: Saturation-Aware Pseudo-Label Selection for Semi-Supervised Segmentation under Foundation-Model Teachers [0.0]
  CW-BASS v2, a saturation-aware pseudo-label selection method, reads the teacher's confidence regime rather than committing to one rule.<n>It pairs held-out calibration, an unbiased per-class noise estimate, with a self-adaptive confidence floor that provably bounds retention away from 1.
  arXiv  Detail & Related papers  (2026-08-13T03:31:31Z)
• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation [14.183839885490372]
  On-policy distillation (OPD) adapts diffusion models by querying a teacher along trajectories generated by the current student.<n>Existing OPD methods naturally extend velocity matching to the CFG-composed prediction.<n>We show that this objective is under-identified at the branch level.<n>We introduce Positive--Direction Matching (PDM), a branch-aware OPD objective that separately constrains the positive prediction and the CFG conditional direction.
  arXiv  Detail & Related papers  (2026-07-27T17:57:02Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Prediction-Only Distillation in Linear and Logistic Regression [61.07540493350384]
  Self-distillation is typically studied when the student is retrained on the teacher's original training inputs.<n>We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme.<n>For binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled.
  arXiv  Detail & Related papers  (2026-07-16T20:45:56Z)
• Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions [4.672326975246762]
  Knowledge Distillation and mixup have proven effective at inducing smoothness in class boundaries.<n>Their interaction remains poorly understood, particularly when mixup is applied only during student training.<n>We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion.
  arXiv  Detail & Related papers  (2026-06-10T14:59:26Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Asking Back: Interaction-Layer Antidistillation Watermarks [7.826668598190874]
  Existing defenses operate on the teacher's output tokens.<n>Recent work shows a oblivious attacker can strip these signals without losing the underlying knowledge.<n>We propose interaction-layer antidistillation watermarks.
  arXiv  Detail & Related papers  (2026-05-15T08:28:35Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• PLD: A Choice-Theoretic List-Wise Knowledge Distillation [12.52282972328311]
  We introduce "Plackett-Luce Distillation (PLD)", a weighted list-wise ranking loss.<n>PLD achieves consistent gains across diverse architectures and distillation objectives.
  arXiv  Detail & Related papers  (2025-06-14T15:31:54Z)
• Distilling Adversarial Robustness Using Heterogeneous Teachers [9.404102810698202]
  robustness can be transferred from an adversarially trained teacher to a student model using knowledge distillation. We develop a defense framework against adversarial attacks by distilling robustness using heterogeneous teachers. Experiments on classification tasks in both white-box and black-box scenarios demonstrate that DARHT achieves state-of-the-art clean and robust accuracies.
  arXiv  Detail & Related papers  (2024-02-23T19:55:13Z)
• Towards Fair Classification against Poisoning Attacks [52.57443558122475]
  We study the poisoning scenario where the attacker can insert a small fraction of samples into training data. We propose a general and theoretically guaranteed framework which accommodates traditional defense methods to fair classification against poisoning attacks.
  arXiv  Detail & Related papers  (2022-10-18T00:49:58Z)
• Adversarial Training with Rectified Rejection [114.83821848791206]
  We propose to use true confidence (T-Con) as a certainty oracle, and learn to predict T-Con by rectifying confidence. We prove that under mild conditions, a rectified confidence (R-Con) rejector and a confidence rejector can be coupled to distinguish any wrongly classified input from correctly classified ones.
  arXiv  Detail & Related papers  (2021-05-31T08:24:53Z)
• Deep Partition Aggregation: Provable Defense against General Poisoning Attacks [136.79415677706612]
  Adrial poisoning attacks distort training data in order to corrupt the test-time behavior of a classifier. We propose two novel provable defenses against poisoning attacks. DPA is a certified defense against a general poisoning threat model. SS-DPA is a certified defense against label-flipping attacks.
  arXiv  Detail & Related papers  (2020-06-26T03:16:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.