Towards Provable (In)Secure Model Weight Release Schemes

• URL: http://arxiv.org/abs/2506.19874v2
• Date: Thu, 26 Jun 2025 08:45:10 GMT
• Title: Towards Provable (In)Secure Model Weight Release Schemes
• Abstract summary: Recent secure weight release schemes claim to enable open-source model distribution while protecting model ownership and preventing misuse.<n>These approaches lack rigorous security foundations and provide only informal security guarantees.<n>Inspired by established works in cryptography, we formalize the security of weight release schemes by introducing several concrete security definitions.
• Score: 5.177249919642388
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent secure weight release schemes claim to enable open-source model distribution while protecting model ownership and preventing misuse. However, these approaches lack rigorous security foundations and provide only informal security guarantees. Inspired by established works in cryptography, we formalize the security of weight release schemes by introducing several concrete security definitions. We then demonstrate our definition's utility through a case study of TaylorMLP, a prominent secure weight release scheme. Our analysis reveals vulnerabilities that allow parameter extraction thus showing that TaylorMLP fails to achieve its informal security goals. We hope this work will advocate for rigorous research at the intersection of machine learning and security communities and provide a blueprint for how future weight release schemes should be designed and evaluated.

Related papers

• SecureBreak -- A dataset towards safe and secure models [3.797867929356259]
  This paper introduces SecureBreak, a safety-oriented dataset designed to support the development of AI-driven solutions for detecting harmful LLM outputs.<n>The dataset is highly reliable due to careful manual annotation, where labels are assigned conservatively to ensure safety.<n>Tests with pre-trained LLMs show improved results after fine-tuning on SecureBreak.
  arXiv  Detail & Related papers  (2026-03-23T13:41:05Z)
• Extending the Formalism and Theoretical Foundations of Cryptography to AI [18.724847875398435]
  Recent progress in (Large) Language Models has enabled the development of autonomous LM-based agents.<n>One emerging direction to mitigate security risks is to constrain agent behaviours via access control and permissioning mechanisms.<n>We first systematize the landscape by constructing an attack taxonomy tailored to language models.<n>We then develop a formal treatment of agentic access control by defining an AIOracle algorithmically and introducing a security-game framework.
  arXiv  Detail & Related papers  (2026-03-03T04:11:21Z)
• Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models [57.006252510102506]
  Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications.<n>We introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems.
  arXiv  Detail & Related papers  (2026-02-12T22:03:35Z)
• Towards Realistic Guarantees: A Probabilistic Certificate for SmoothLLM [0.7212939068975618]
  SmoothLLM provides a certification guarantee against jailbreaking attacks.<n>It relies on a strict k-unstable' assumption that rarely holds in practice.<n>We introduce a more realistic probabilistic framework, (k, $varepsilon$)-unstable,' to certify defenses against diverse jailbreaking attacks.
  arXiv  Detail & Related papers  (2025-11-24T03:25:16Z)
• Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models [63.54707418559388]
  We propose patching for large language models (LLMs) like software versions.<n>Our method enables rapid remediation by prepending a compact, learnable prefix to an existing model.
  arXiv  Detail & Related papers  (2025-11-11T17:25:44Z)
• Beyond Algorithmic Proofs: Towards Implementation-Level Provable Security [1.338174941551702]
  We present Implementation-Level Provable Security, a new paradigm that defines security in terms of structurally verifiable resilience against real-world attack surfaces during deployment.<n>We present SEER (Secure and Efficient Encryption-based Erasure via Ransomware), a file destruction system that repurposes and reinforces the encryption core of Babuk ransomware.
  arXiv  Detail & Related papers  (2025-08-02T01:58:06Z)
• Provably Secure Retrieval-Augmented Generation [7.412110686946628]
  This paper proposes the first provably secure framework for Retrieval-Augmented Generation (RAG) systems.<n>Our framework employs a pre-storage full-encryption scheme to ensure dual protection of both retrieved content and vector embeddings.
  arXiv  Detail & Related papers  (2025-08-01T21:37:16Z)
• Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention [65.47632669243657]
  A dishonest institution can exploit mechanisms to discriminate or unjustly deny services under the guise of uncertainty.<n>We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage.<n>We propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence.
  arXiv  Detail & Related papers  (2025-05-29T19:47:50Z)
• Probabilistic Shielding for Safe Reinforcement Learning [51.35559820893218]
  In real-life scenarios, a Reinforcement Learning (RL) agent must often also behave in a safe manner, including at training time.<n>We present a new, scalable method, which enjoys strict formal guarantees for Safe RL.<n>We show that our approach provides a strict formal safety guarantee that the agent stays safe at training and test time.
  arXiv  Detail & Related papers  (2025-03-09T17:54:33Z)
• Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models [8.024771725860127]
  Large Language Models (LLMs) remain vulnerable to jailbreak attacks that bypass their safety mechanisms. We introduce a novel scalable jailbreak attack that preempts the activation of an LLM's safety policies by occupying its computational resources.
  arXiv  Detail & Related papers  (2024-10-05T15:10:01Z)
• Safety Layers in Aligned Large Language Models: The Key to LLM Security [43.805905164456846]
  Internal parameters in aligned LLMs can be vulnerable to security degradation when subjected to fine-tuning attacks.<n>Our work uncovers the mechanism behind security in aligned LLMs at the parameter level, identifying a small set of contiguous layers in the middle of the model.<n>We propose a novel fine-tuning approach, Safely Partial- Fine-Tuning (SPPFT), that fixes the gradient of the safety layers during fine-tuning to address the security degradation.
  arXiv  Detail & Related papers  (2024-08-30T04:35:59Z)
• Tamper-Resistant Safeguards for Open-Weight LLMs [57.90526233549399]
  We develop a method for building tamper-resistant safeguards into open-weight LLMs.<n>We find that our method greatly improves tamper-resistance while preserving benign capabilities.
  arXiv  Detail & Related papers  (2024-08-01T17:59:12Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• Jailbreaking as a Reward Misspecification Problem [80.52431374743998]
  We propose a novel perspective that attributes this vulnerability to reward misspecification during the alignment process.<n>We introduce a metric ReGap to quantify the extent of reward misspecification and demonstrate its effectiveness.<n>We present ReMiss, a system for automated red teaming that generates adversarial prompts in a reward-misspecified space.
  arXiv  Detail & Related papers  (2024-06-20T15:12:27Z)
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? [52.238883592674696]
  Ring-A-Bell is a model-agnostic red-teaming tool for T2I diffusion models. It identifies problematic prompts for diffusion models with the corresponding generation of inappropriate content. Our results show that Ring-A-Bell, by manipulating safe prompting benchmarks, can transform prompts that were originally regarded as safe to evade existing safety mechanisms.
  arXiv  Detail & Related papers  (2023-10-16T02:11:20Z)
• Source-Replacement Model for Phase-Matching Quantum Key Distribution [0.40964539027092917]
  We study the security of the phase-matching protocol using an intuitive source-replacement model. As an application of this approach, we introduce a beam-splitting attack scheme.
  arXiv  Detail & Related papers  (2023-09-29T15:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.