Adversaries Can Misuse Combinations of Safe Models

• URL: http://arxiv.org/abs/2406.14595v2
• Date: Mon, 1 Jul 2024 19:58:00 GMT
• Title: Adversaries Can Misuse Combinations of Safe Models
• Authors: Erik Jones, Anca Dragan, Jacob Steinhardt,
• Abstract summary: Developers try to evaluate whether an AI system can be misused by adversaries before releasing it. We show that adversaries can misuse combinations of models even when each individual model is safe. Our work suggests that even perfectly-aligned frontier systems can enable misuse without ever producing malicious outputs.
• Score: 36.863895028598336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developers try to evaluate whether an AI system can be misused by adversaries before releasing it; for example, they might test whether a model enables cyberoffense, user manipulation, or bioterrorism. In this work, we show that individually testing models for misuse is inadequate; adversaries can misuse combinations of models even when each individual model is safe. The adversary accomplishes this by first decomposing tasks into subtasks, then solving each subtask with the best-suited model. For example, an adversary might solve challenging-but-benign subtasks with an aligned frontier model, and easy-but-malicious subtasks with a weaker misaligned model. We study two decomposition methods: manual decomposition where a human identifies a natural decomposition of a task, and automated decomposition where a weak model generates benign tasks for a frontier model to solve, then uses the solutions in-context to solve the original task. Using these decompositions, we empirically show that adversaries can create vulnerable code, explicit images, python scripts for hacking, and manipulative tweets at much higher rates with combinations of models than either individual model. Our work suggests that even perfectly-aligned frontier systems can enable misuse without ever producing malicious outputs, and that red-teaming efforts should extend beyond single models in isolation.

Related papers

• Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging [83.5273168208788]
  Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training.<n>The merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors.<n>We propose Purifying TAsk Vectors (PAVE) in knowledge-aware subspace to overcome these challenges.
  arXiv  Detail & Related papers  (2025-10-16T14:02:57Z)
• Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent [74.02034188307857]
  Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data. We find existing methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance. Our approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.
  arXiv  Detail & Related papers  (2025-01-02T12:45:21Z)
• Exploring Query Efficient Data Generation towards Data-free Model Stealing in Hard Label Setting [38.755154033324374]
  Data-free model stealing involves replicating the functionality of a target model into a substitute model without accessing the target model's structure, parameters, or training data.<n>This paper presents a new data-free model stealing approach called Query Efficient Data Generation (textbfQEDG)<n>We introduce two distinct loss functions to ensure the generation of sufficient samples that closely and uniformly align with the target model's decision boundary.
  arXiv  Detail & Related papers  (2024-12-18T03:03:15Z)
• NegMerge: Consensual Weight Negation for Strong Machine Unlearning [21.081262106431506]
  Machine unlearning aims to selectively remove specific knowledge from a model. Current methods rely on fine-tuning models on the forget set, generating a task vector, and subtracting it from the original model. We propose a novel method that leverages all given fine-tuned models rather than selecting a single one.
  arXiv  Detail & Related papers  (2024-10-08T00:50:54Z)
• Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networks [86.55317144826179]
  Previous methods always leverage the transferable adversarial examples as the model fingerprint. We propose a novel yet simple model stealing detection method based on SAmple Correlation (SAC) SAC successfully defends against various model stealing attacks, even including adversarial training or transfer learning.
  arXiv  Detail & Related papers  (2022-10-21T02:07:50Z)
• Composing Ensembles of Pre-trained Models via Iterative Consensus [95.10641301155232]
  We propose a unified framework for composing ensembles of different pre-trained models. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer.
  arXiv  Detail & Related papers  (2022-10-20T18:46:31Z)
• Careful What You Wish For: on the Extraction of Adversarially Trained Models [2.707154152696381]
  Recent attacks on Machine Learning (ML) models pose several security and privacy threats. We propose a framework to assess extraction attacks on adversarially trained models. We show that adversarially trained models are more vulnerable to extraction attacks than models obtained under natural training circumstances.
  arXiv  Detail & Related papers  (2022-07-21T16:04:37Z)
• MEGA: Model Stealing via Collaborative Generator-Substitute Networks [4.065949099860426]
  Recent data-free model stealingmethods are shown effective to extract the knowledge of thetarget model without using real query examples. We propose a data-free model stealing frame-work,MEGA, which is based on collaborative generator-substitute networks. Our results show that theaccuracy of our trained substitute model and the adversarialattack success rate over it can be up to 33% and 40% higherthan state-of-the-art data-free black-box attacks.
  arXiv  Detail & Related papers  (2022-01-31T09:34:28Z)
• Training Meta-Surrogate Model for Transferable Adversarial Attack [98.13178217557193]
  We consider adversarial attacks to a black-box model when no queries are allowed. In this setting, many methods directly attack surrogate models and transfer the obtained adversarial examples to fool the target model. We show we can obtain a Meta-Surrogate Model (MSM) such that attacks to this model can be easier transferred to other models.
  arXiv  Detail & Related papers  (2021-09-05T03:27:46Z)
• BODAME: Bilevel Optimization for Defense Against Model Extraction [10.877450596327407]
  We consider an adversarial setting to prevent model extraction under the assumption that will make best guess on the service provider's attacker. We formulate a surrogate model using the predictions of the true model. We give a tractable transformation and an algorithm for more complicated models that are learned by using gradient descent-based algorithms.
  arXiv  Detail & Related papers  (2021-03-11T17:08:31Z)
• Differentiable Language Model Adversarial Attacks on Categorical Sequence Classifiers [0.0]
  An adversarial attack paradigm explores various scenarios for the vulnerability of deep learning models. We use a fine-tuning of a language model for adversarial attacks as a generator of adversarial examples. Our model works for diverse datasets on bank transactions, electronic health records, and NLP datasets.
  arXiv  Detail & Related papers  (2020-06-19T11:25:36Z)
• DaST: Data-free Substitute Training for Adversarial Attacks [55.76371274622313]
  We propose a data-free substitute training method (DaST) to obtain substitute models for adversarial black-box attacks. To achieve this, DaST utilizes specially designed generative adversarial networks (GANs) to train the substitute models. Experiments demonstrate the substitute models can achieve competitive performance compared with the baseline models.
  arXiv  Detail & Related papers  (2020-03-28T04:28:13Z)
• Regularizers for Single-step Adversarial Training [49.65499307547198]
  We propose three types of regularizers that help to learn robust models using single-step adversarial training methods. Regularizers mitigate the effect of gradient masking by harnessing on properties that differentiate a robust model from that of a pseudo robust model.
  arXiv  Detail & Related papers  (2020-02-03T09:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.