Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs

• URL: http://arxiv.org/abs/2606.21638v1
• Date: Fri, 19 Jun 2026 17:44:22 GMT
• Title: Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs
• Abstract summary: Tiered Language Models (TLMs) support multiple capability levels.<n>We show that keyed configuration can acquire a new language, gain instruction-following ability, and memorize private factual knowledge.<n>Because authorization operates on the model's weight structure rather than in the input space, the mechanism resists fine-tuning-based extraction and partial key compromise.
• Score: 27.546429876997873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-weight Large Language Models (LLMs) enable scientific progress and broad deployment. However, they make it difficult to control access to sensitive capabilities. Current practice either suppresses dangerous capabilities before release or mediates access through closed services that use specialized model variants, input/output monitors, and API permissions. The former is susceptible to jailbreaks while sacrificing capability for all users to mitigate the risks posed by a few, and the latter is fundamentally incompatible with open-weight release. In this paper, we propose Tiered Language Models (TLMs), where a single set of released weights supports multiple capability levels. In its default public configuration, a TLM behaves as a conventional LLM. A compact secret key specifies a permutation over a small parameter subset, inducing an alternative computation graph over the same weights that exposes additional capabilities. We develop a training protocol that jointly pretrains both configurations from scratch, then fine-tunes the keyed configuration on private data with regularization to preserve the public model's behavior. We pretrain 180M- and 650M-parameter TLMs and demonstrate that the keyed configuration can acquire a new language, gain instruction-following ability, and memorize private factual knowledge, whereas the public configuration exhibits none of these capabilities. Moreover, we show that our approach extends naturally to multiple hierarchical tiers. Because authorization operates on the model's weight structure rather than in the input space, the mechanism resists fine-tuning-based extraction and partial key compromise. In general, TLMs take a step toward reconciling open-weight release with selective capability control.

Related papers

• Privacy-Preserving Split Learning for Federated LLM Fine-Tuning [12.651705155064102]
  Fine-tuning large language models (LLMs) on domain-specific data is essential for downstream adaptation.<n>In many deployments, a participant cannot hold the complete model locally. Split Learning (SL) addresses this by partitioning the model between the participant and a server so that only a small portion runs locally.<n> Federated Learning (FL) enables collaborative training across participants by sharing only model updates instead of raw data.
  arXiv  Detail & Related papers  (2026-09-09T06:48:21Z)
• Unlocking Lossless Speedups in LLMs via Discrete Diffusion [67.4336730000437]
  Large Language Models (LLMs) owe much of their success to next-token prediction (NTP)<n>We introduce diffusion-augmented LLMs, a new class of models that defines an AR model distribution.<n>We decouple the parameters of these models into two sets: AR weights, trained using the standard NTP objective, and lightweight diffusion weights, trained to generate multiple tokens simultaneously.
  arXiv  Detail & Related papers  (2026-09-03T15:48:43Z)
• CLIFT: Turning Gemini Robotics On-Device into Humanoid Specialists via Non-Invasive Closed-Loop Iterative Fine-Tuning [45.60764233402256]
  An emerging access paradigm for closed-weight robot foundation models is the managed supervised fine-tuning (SFT) API.<n>We study how effective this managed-API regime is for humanoid adaptation.<n>We introduce CLIFT: Closed-Loop Iterative Fine-Tuning, which turns deployment-time reward feedback into API-compatible supervised data.
  arXiv  Detail & Related papers  (2026-07-31T08:50:25Z)
• Locking Pretrained Weights via Deep Low-Rank Residual Distillation [31.050349139080286]
  We exploit the inference-training of automatic differentiation as a novel defense against adaptive attackers.<n>We propose DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained parameter in their model with a deep low-rank residual network (DLR-Net) of comparable count.<n>Our defense succeeds in withstanding adaptive attackers with full knowledge of the defense strategy while preserving the original model's capabilities.
  arXiv  Detail & Related papers  (2026-05-11T16:09:32Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs [61.15237978606501]
  Large language models can infer private user attributes from user-generated text.<n>Existing anonymization-based defenses are coarse-grained, lacking word-level precision in anonymizing privacy-leaking elements.<n>We propose a unified defense framework that combines fine-grained anonymization (TRACE) with inference-preventing optimization (RPS)
  arXiv  Detail & Related papers  (2026-02-12T03:37:50Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Practical Secure Inference Algorithm for Fine-tuned Large Language Model Based on Fully Homomorphic Encryption [0.0]
  We combine Fully Homomorphic Encryption(FHE) and provable security theory with Fine-Tuning(PEFT) to propose an efficient and secure inference scheme for large language models.<n>In this paper, we use the open-source model ChatGLM2-6B as the base model which is fine-tuned by LoRA.<n> Experimental results show the inference efficiency of our scheme reaches 1.61s/ which displays that the scheme has good practicality.
  arXiv  Detail & Related papers  (2025-01-03T07:19:23Z)
• Instructional Fingerprinting of Large Language Models [57.72356846657551]
  We present a pilot study on fingerprinting Large language models (LLMs) as a form of very lightweight instruction tuning. Results on 11 popularly-used LLMs showed that this approach is lightweight and does not affect the normal behavior of the model. It also prevents publisher overclaim, maintains robustness against fingerprint guessing and parameter-efficient training, and supports multi-stage fingerprinting akin to MIT License.
  arXiv  Detail & Related papers  (2024-01-21T09:51:45Z)
• MatFormer: Nested Transformer for Elastic Inference [91.45687988953435]
  MatFormer is a novel Transformer architecture designed to provide elastic inference across diverse deployment constraints.<n>MatFormer achieves this by incorporating a nested Feed Forward Network (FFN) block structure within a standard Transformer model.<n>We show that a 850M decoder-only MatFormer language model (MatLM) allows us to extract multiple smaller models spanning from 582M to 850M parameters.
  arXiv  Detail & Related papers  (2023-10-11T17:57:14Z)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [54.692405042065815]
  We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ protects only 1% salient weights and achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. We also implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs.
  arXiv  Detail & Related papers  (2023-06-01T17:59:10Z)
• Differentially Private Decoding in Large Language Models [14.221692239892207]
  We propose a simple, easy to interpret, and computationally lightweight perturbation mechanism to be applied to an already trained model at the decoding stage. Our perturbation mechanism is model-agnostic and can be used in conjunction with any Large Language Model.
  arXiv  Detail & Related papers  (2022-05-26T20:50:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.