Who's asking? User personas and the mechanics of latent misalignment

• URL: http://arxiv.org/abs/2406.12094v2
• Date: Tue, 13 Aug 2024 14:02:13 GMT
• Title: Who's asking? User personas and the mechanics of latent misalignment
• Authors: Asma Ghandeharioun, Ann Yuan, Marius Guerard, Emily Reif, Michael A. Lepori, Lucas Dixon,
• Abstract summary: misaligned capabilities remain latent in safety-tuned models. We show that even when model generations are safe, harmful content can persist in hidden representations. We investigate why certain personas break model safeguards and find that they enable the model to form more charitable interpretations.
• Score: 12.92431783194089
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite investments in improving model safety, studies show that misaligned capabilities remain latent in safety-tuned models. In this work, we shed light on the mechanics of this phenomenon. First, we show that even when model generations are safe, harmful content can persist in hidden representations and can be extracted by decoding from earlier layers. Then, we show that whether the model divulges such content depends significantly on its perception of who it is talking to, which we refer to as user persona. In fact, we find manipulating user persona to be even more effective for eliciting harmful content than direct attempts to control model refusal. We study both natural language prompting and activation steering as control methods and show that activation steering is significantly more effective at bypassing safety filters. We investigate why certain personas break model safeguards and find that they enable the model to form more charitable interpretations of otherwise dangerous queries. Finally, we show we can predict a persona's effect on refusal given only the geometry of its steering vector.

Related papers

• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• QUEEN: Query Unlearning against Model Extraction [22.434812818540966]
  Model extraction attacks pose a non-negligible threat to the security and privacy of deep learning models. We propose QUEEN (QUEry unlEarNing) that proactively launches counterattacks on potential model extraction attacks.
  arXiv  Detail & Related papers  (2024-07-01T13:01:41Z)
• Steering Without Side Effects: Improving Post-Deployment Control of Language Models [61.99293520621248]
  Language models (LMs) have been shown to behave unexpectedly post-deployment. We present KL-then-steer (KTS), a technique that decreases the side effects of steering while retaining its benefits. Our best method prevents 44% of jailbreak attacks compared to the original Llama-2-chat-7B model.
  arXiv  Detail & Related papers  (2024-06-21T01:37:39Z)
• Refusal in Language Models Is Mediated by a Single Direction [4.532520427311685]
  We show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. We propose a novel white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities.
  arXiv  Detail & Related papers  (2024-06-17T16:36:12Z)
• On Prompt-Driven Safeguarding for Large Language Models [172.13943777203377]
  We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO. Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness.
  arXiv  Detail & Related papers  (2024-01-31T17:28:24Z)
• Navigating the OverKill in Large Language Models [84.62340510027042]
  We investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to an over-attention of harmful words like 'kill' and prompts emphasizing safety will exacerbate overkill. We introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon.
  arXiv  Detail & Related papers  (2024-01-31T07:26:47Z)
• MOVE: Effective and Harmless Ownership Verification via Embedded External Features [109.19238806106426]
  We propose an effective and harmless model ownership verification (MOVE) to defend against different types of model stealing simultaneously. We conduct the ownership verification by verifying whether a suspicious model contains the knowledge of defender-specified external features. In particular, we develop our MOVE method under both white-box and black-box settings to provide comprehensive model protection.
  arXiv  Detail & Related papers  (2022-08-04T02:22:29Z)
• Initiative Defense against Facial Manipulation [82.96864888025797]
  We propose a novel framework of initiative defense to degrade the performance of facial manipulation models controlled by malicious users. We first imitate the target manipulation model with a surrogate model, and then devise a poison perturbation generator to obtain the desired venom.
  arXiv  Detail & Related papers  (2021-12-19T09:42:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.