Auditing language models for hidden objectives

• URL: http://arxiv.org/abs/2503.10965v2
• Date: Fri, 28 Mar 2025 01:48:40 GMT
• Title: Auditing language models for hidden objectives
• Authors: Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura-Gasparian, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, Evan Hubinger,
• Abstract summary: We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives.<n>As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploit some of these errors.<n>We leverage this model to study alignment audits in two ways. First, we conduct a blind auditing game where four teams, unaware of the model's hidden objective or training, investigate it for concerning behaviors and their causes.<n>Three teams successfully uncovered the model's hidden objective using techniques including
• Score: 26.85568746300155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploit some of these errors. We verify via out-of-distribution evaluations that the model generalizes to exhibit whatever behaviors it believes RMs rate highly, including ones not reinforced during training. We leverage this model to study alignment audits in two ways. First, we conduct a blind auditing game where four teams, unaware of the model's hidden objective or training, investigate it for concerning behaviors and their causes. Three teams successfully uncovered the model's hidden objective using techniques including interpretability with sparse autoencoders (SAEs), behavioral attacks, and training data analysis. Second, we conduct an unblinded follow-up study of eight techniques for auditing the model, analyzing their strengths and limitations. Overall, our work provides a concrete example of using alignment audits to discover a model's hidden objective and proposes a methodology for practicing and validating progress in alignment auditing.

Related papers

• Self-supervised Analogical Learning using Language Models [59.64260218737556]
  We propose SAL, a self-supervised analogical learning framework.<n> SAL mimics the human analogy process and trains models to explicitly transfer high-quality symbolic solutions.<n>We show that the resulting models outperform base language models on a wide range of reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-02-03T02:31:26Z)
• Frontier Models are Capable of In-context Scheming [41.30527987937867]
  One safety concern is that AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives. We evaluate frontier models on a suite of six agentic evaluations where models are instructed to pursue goals. We find that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate in-context scheming capabilities.
  arXiv  Detail & Related papers  (2024-12-06T12:09:50Z)
• Complementary Learning for Real-World Model Failure Detection [15.779651238128562]
  We introduce complementary learning, where we use learned characteristics from different training paradigms to detect model errors. We demonstrate our approach by learning semantic and predictive motion labels in point clouds in a supervised and self-supervised manner. We perform a large-scale qualitative analysis and present LidarCODA, the first dataset with labeled anomalies in lidar point clouds.
  arXiv  Detail & Related papers  (2024-07-19T13:36:35Z)
• How Close are Other Computer Vision Tasks to Deepfake Detection? [42.79190870582115]
  We present a new measurement, "model separability," for assessing a model's raw capacity to separate data in an unsupervised manner. Our analysis shows that pre-trained face recognition models are more closely related to deepfake detection than other models. We found that self-supervised models deliver the best results, but there is a risk of overfitting.
  arXiv  Detail & Related papers  (2023-10-02T06:32:35Z)
• A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
  We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check. Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models. The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
  arXiv  Detail & Related papers  (2023-07-25T17:02:38Z)
• An Empirical Study of Deep Learning Models for Vulnerability Detection [4.243592852049963]
  We surveyed and reproduced 9 state-of-the-art deep learning models on 2 widely used vulnerability detection datasets. We investigated model capabilities, training data, and model interpretation. Our findings can help better understand model results, provide guidance on preparing training data, and improve the robustness of the models.
  arXiv  Detail & Related papers  (2022-12-15T19:49:34Z)
• Improving In-Context Few-Shot Learning via Self-Supervised Training [48.801037246764935]
  We propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage. We find that the intermediate self-supervision stage produces models that outperform strong baselines.
  arXiv  Detail & Related papers  (2022-05-03T18:01:07Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Defending against Model Stealing via Verifying Embedded External Features [90.29429679125508]
  adversaries can steal' deployed models even when they have no training samples and can not get access to the model parameters or structures. We explore the defense from another angle by verifying whether a suspicious model contains the knowledge of defender-specified emphexternal features. Our method is effective in detecting different types of model stealing simultaneously, even if the stolen model is obtained via a multi-stage stealing process.
  arXiv  Detail & Related papers  (2021-12-07T03:51:54Z)
• Paired Examples as Indirect Supervision in Latent Decision Models [109.76417071249945]
  We introduce a way to leverage paired examples that provide stronger cues for learning latent decisions. We apply our method to improve compositional question answering using neural module networks on the DROP dataset.
  arXiv  Detail & Related papers  (2021-04-05T03:58:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.