PRISM: A Design Framework for Open-Source Foundation Model Safety

• URL: http://arxiv.org/abs/2406.10415v1
• Date: Fri, 14 Jun 2024 21:26:15 GMT
• Title: PRISM: A Design Framework for Open-Source Foundation Model Safety
• Authors: Terrence Neumann, Bryan Jones,
• Abstract summary: This paper addresses the question of how open foundation model developers should approach model safety. We introduce PRISM, a design framework for open-source foundation model safety that emphasizes Private, Robust, Independent Safety measures. PRISM aims to create a safer open-source ecosystem that maximizes the potential of these powerful technologies while minimizing the risks to individuals and society as a whole.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of open-source foundation models has brought transparency and accessibility to this groundbreaking technology. However, this openness has also enabled the development of highly-capable, unsafe models, as exemplified by recent instances such as WormGPT and FraudGPT, which are specifically designed to facilitate criminal activity. As the capabilities of open foundation models continue to grow, potentially outpacing those of closed-source models, the risk of misuse by bad actors poses an increasingly serious threat to society. This paper addresses the critical question of how open foundation model developers should approach model safety in light of these challenges. Our analysis reveals that open-source foundation model companies often provide less restrictive acceptable use policies (AUPs) compared to their closed-source counterparts, likely due to the inherent difficulties in enforcing such policies once the models are released. To tackle this issue, we introduce PRISM, a design framework for open-source foundation model safety that emphasizes Private, Robust, Independent Safety measures, at Minimal marginal cost of compute. The PRISM framework proposes the use of modular functions that moderate prompts and outputs independently of the core language model, offering a more adaptable and resilient approach to safety compared to the brittle reinforcement learning methods currently used for value alignment. By focusing on identifying AUP violations and engaging the developer community in establishing consensus around safety design decisions, PRISM aims to create a safer open-source ecosystem that maximizes the potential of these powerful technologies while minimizing the risks to individuals and society as a whole.

Related papers

• OpenAI o1 System Card [274.83891368890977]
  The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. This report outlines the safety work carried out for the OpenAI o1 and OpenAI o1-mini models, including safety evaluations, external red teaming, and Preparedness Framework evaluations.
  arXiv  Detail & Related papers  (2024-12-21T18:04:31Z)
• On Evaluating the Durability of Safeguards for Open-Weight LLMs [80.36750298080275]
  We discuss whether technical safeguards can impede the misuse of large language models (LLMs) We show that even evaluating these defenses is exceedingly difficult and can easily mislead audiences into thinking that safeguards are more durable than they really are. We suggest future research carefully cabin claims to more constrained, well-defined, and rigorously examined threat models.
  arXiv  Detail & Related papers  (2024-12-10T01:30:32Z)
• Position: On-Premises LLM Deployment Demands a Middle Path: Preserving Privacy Without Sacrificing Model Confidentiality [18.575663556525864]
  We argue that deploying closed-source LLMs within user-controlled infrastructure enhances data privacy and mitigates misuse risks. A well-designed on-premises deployment must ensure model confidentiality -- by preventing model theft -- and offer privacy-preserving customization. Our findings demonstrate that privacy and confidentiality can coexist, paving the way for secure on-premises AI deployment.
  arXiv  Detail & Related papers  (2024-10-15T02:00:36Z)
• Towards Secure and Private AI: A Framework for Decentralized Inference [14.526663289437584]
  Large multimodal foundational models present challenges in scalability, reliability, and potential misuse. Decentralized systems offer a solution by distributing workload and mitigating central points of failure. We address these challenges with a comprehensive framework designed for responsible AI development.
  arXiv  Detail & Related papers  (2024-07-28T05:09:17Z)
• Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning [61.2224355547598]
  Open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. Our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions.
  arXiv  Detail & Related papers  (2024-04-16T13:22:54Z)
• Here's a Free Lunch: Sanitizing Backdoored Models with Model Merge [17.3048898399324]
  democratization of pre-trained language models through open-source initiatives has rapidly advanced innovation and expanded access to cutting-edge technologies. backdoor attacks, where hidden malicious behaviors are triggered by specific inputs, compromising natural language processing (NLP) system integrity and reliability. This paper suggests that merging a backdoored model with other homogeneous models can significantly remediate backdoor vulnerabilities.
  arXiv  Detail & Related papers  (2024-02-29T16:37:08Z)
• On the Societal Impact of Open Foundation Models [93.67389739906561]
  We focus on open foundation models, defined here as those with broadly available model weights. We identify five distinctive properties of open foundation models that lead to both their benefits and risks.
  arXiv  Detail & Related papers  (2024-02-27T16:49:53Z)
• Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives [6.575445633821399]
  Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate. This paper offers an examination of the risks and benefits of open-sourcing highly capable foundation models.
  arXiv  Detail & Related papers  (2023-09-29T17:03:45Z)
• Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models [103.71308117592963]
  We present an algorithm for training self-destructing models leveraging techniques from meta-learning and adversarial learning. In a small-scale experiment, we show MLAC can largely prevent a BERT-style model from being re-purposed to perform gender identification.
  arXiv  Detail & Related papers  (2022-11-27T21:43:45Z)
• Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments [84.3830478851369]
  We propose a safe reinforcement learning approach that can jointly learn the environment and optimize the control policy. Our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.
  arXiv  Detail & Related papers  (2022-09-29T20:49:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.