Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility

• URL: http://arxiv.org/abs/2607.20440v1
• Date: Tue, 12 May 2026 04:36:20 GMT
• Title: Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility
• Abstract summary: Anti-distillation (AD) has been proposed to generate defensive outputs that hinder distillation effectiveness.<n>We propose textbfunderlineSkeleton-textbfunderlineGuided textbfunderlineReasoning textbfunderlineEditing (SGRE)<n>In the answer stage, the teacher model first generates clean reasoning traces, preserving the original reasoning accuracy.<n>In the editing stage, we introduce a three-stage strategy consisting of reasoning skeleton extraction, skeleton graph coarsening, and skeleton verbal
• Score: 13.916452483499185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Proprietary large language models (LLMs) entail substantial intellectual and financial investment, making them valuable intellectual property (IP). However, even when deployed via black-box APIs, these models remain vulnerable to unauthorized knowledge distillation, which allows adversaries to cheaply extract and replicate model capabilities. To address this issue, anti-distillation (AD) has been proposed to generate defensive outputs that hinder distillation effectiveness, overcoming the limitation of watermarking-based approaches that rely on post-hoc verification. However, existing AD methods based on internal model perturbations struggle to balance anti-distillability and utility (e.g., answer accuracy and naturalness) of reasoning traces, with stronger defenses often causing significant utility loss. To fill this gap, we propose \textbf{\underline{S}}keleton-\textbf{\underline{G}}uided \textbf{\underline{R}}easoning \textbf{\underline{E}}diting (SGRE), an \textit{Answer-then-Edit} framework that performs post-hoc trace modification for anti-distillation. In the answer stage, the teacher model first generates clean reasoning traces, preserving the original reasoning accuracy while enabling more flexible control over trace naturalness. In the editing stage, we draw inspiration from Cognitive Load Theory (CLT) and introduce a three-stage strategy consisting of reasoning skeleton extraction, skeleton graph coarsening, and skeleton verbalization. These operations jointly perturb reasoning structures and augment textual complexity to amplify extraneous load on student models, hindering their acquisition of underlying reasoning patterns. Extensive experiments across diverse LLMs demonstrate that SGRE achieves state-of-the-art performance in reducing distillation effectiveness, while maintaining lossless reasoning accuracy and superior trace naturalness.

Related papers

• Robust Reasoning via Dynamic Token Selection for Distribution-Aligned Self-Distillation [10.71966126081564]
  Self-distillation improves learning efficiency by rewriting reference answers as training data that better matches the model's own distribution.<n>But reference answers also introduce strong stylistic biases, causing the generative model to imitate surface forms rather than learn useful reasoning patterns.<n>We propose Distribution-Aligned Self-Distillation (DASD), which uses an answer-aware reference model to generate candidate tokens and dynamically filters them according to the base model's confidence.
  arXiv  Detail & Related papers  (2026-05-30T09:03:03Z)
• Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective [52.25797439810419]
  Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored.<n>We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels.<n>We derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility.
  arXiv  Detail & Related papers  (2026-02-03T11:16:59Z)
• ReGuLaR: Variational Latent Reasoning Guided by Rendered Chain-of-Thought [49.203970812338916]
  Explicit reasoning chains introduce substantial computational redundancy.<n>Recent latent reasoning methods attempt to mitigate this by compressing reasoning processes into latent space.<n>We propose Rendered CoT-Guided variational Latent Reasoning (ReGuLaR)
  arXiv  Detail & Related papers  (2026-01-30T17:08:06Z)
• Copyright Laundering Through the AI Ouroboros: Adapting the 'Fruit of the Poisonous Tree' Doctrine to Recursive AI Training [0.31061678033205636]
  Copyright enforcement rests on an evidentiary bargain.<n>It comes under strain when AI systems are trained through multi-generational pipelines.<n>This Article adapts the "fruit of the poisonous tree" (FOPT) principle to propose a AI-FOPT standard.
  arXiv  Detail & Related papers  (2026-01-06T01:02:50Z)
• Information-Preserving Reformulation of Reasoning Traces for Antidistillation [80.99448418790125]
  We propose an information-preserving antidistillation reformulation of reasoning traces.<n>A small auxiliary model is trained to perform this reformulation, incurring minimal computational overhead.<n>Experiments demonstrate that PART consistently disrupts distillation across student models of different sizes and types.
  arXiv  Detail & Related papers  (2025-10-13T15:42:11Z)
• KaVa: Latent Reasoning via Compressed KV-Cache Distillation [8.058130367727005]
  KaVa is a framework that distills knowledge directly from a compressed KV-cache of a teacher into a latent-reasoning student via self-distillation.<n>We show that the abstract, unstructured knowledge within compressed KV-cache, which lacks direct token correspondence, can serve as a rich supervisory signal for a latent reasoning student.
  arXiv  Detail & Related papers  (2025-10-02T17:59:51Z)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models [62.70575022567081]
  We propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning.<n>Our work establishes a new direction for building more robust and reliable reasoning models.
  arXiv  Detail & Related papers  (2025-09-29T04:27:23Z)
• ConciseHint: Boosting Efficient Reasoning via Continuous Concise Hints during Generation [74.37307916314407]
  We propose a framework dubbed ConciseHint, which continuously encourages the reasoning model to speak concisely.<n>Experiments on the state-of-the-art LRMs, including DeepSeek-R1 and Qwen-3 series, demonstrate that our method can effectively produce concise reasoning.
  arXiv  Detail & Related papers  (2025-06-23T16:20:44Z)
• DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation [49.58082402742583]
  Large Language Models (LLMs) represent substantial intellectual and economic investments.<n>LLMs can inadvertently facilitate model imitation via knowledge distillation (KD)<n>This paper introduces an effective and efficient Defensive Output Generation (DOGe) strategy.
  arXiv  Detail & Related papers  (2025-05-26T04:31:38Z)
• Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective [65.10019978876863]
  Diffusion-Based Purification (DBP) has emerged as an effective defense mechanism against adversarial attacks.<n>In this paper, we propose that the intrinsicity in the DBP process is the primary factor driving robustness.
  arXiv  Detail & Related papers  (2024-04-22T16:10:38Z)
• Confidence Preservation Property in Knowledge Distillation Abstractions [2.9370710299422598]
  Social media platforms prevent malicious activities by detecting harmful content of posts and comments. They employ large-scale deep neural network language models for sentiment analysis and content understanding. Some models, like BERT, are complex, and have numerous parameters, which makes them expensive to operate and maintain. Industry experts employ a knowledge distillation compression technique, where a distilled model is trained to reproduce the classification behavior of the original model.
  arXiv  Detail & Related papers  (2024-01-21T01:37:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.