Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models

• URL: http://arxiv.org/abs/2412.01784v1
• Date: Mon, 02 Dec 2024 18:34:51 GMT
• Title: Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models
• Authors: Cameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani, Fedor Ryzhenkov, Jacob Haimes, Felix Hofstätter, Teun van der Weij,
• Abstract summary: We present a novel model-agnostic method for detecting sandbagging behavior using noise injection.<n>We test this technique across a range of model sizes and multiple-choice question benchmarks (MMLU, AI2, WMDP)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Capability evaluations play a critical role in ensuring the safe deployment of frontier AI systems, but this role may be undermined by intentional underperformance or ``sandbagging.'' We present a novel model-agnostic method for detecting sandbagging behavior using noise injection. Our approach is founded on the observation that introducing Gaussian noise into the weights of models either prompted or fine-tuned to sandbag can considerably improve their performance. We test this technique across a range of model sizes and multiple-choice question benchmarks (MMLU, AI2, WMDP). Our results demonstrate that noise injected sandbagging models show performance improvements compared to standard models. Leveraging this effect, we develop a classifier that consistently identifies sandbagging behavior. Our unsupervised technique can be immediately implemented by frontier labs or regulatory bodies with access to weights to improve the trustworthiness of capability evaluations.

Related papers

• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition [72.24180896265192]
  We introduce AgentNoiseBench, a framework for evaluating robustness of agentic models under noisy environments.<n>We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios.<n>We then categorize environmental noise into two primary types: user-noise and tool-noise.<n>Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T20:33:10Z)
• World Model Robustness via Surprise Recognition [2.019328356430632]
  We develop an algorithm that reduces the impact of noise in world model-based reinforcement learning agents.<n>We show that our techniques preserve performance relative to baselines under varying types and levels of noise.<n>We demonstrate that our methods enhance the stability of two state-of-the-art world models with markedly different underlying architectures.
  arXiv  Detail & Related papers  (2025-11-30T22:25:45Z)
• Transformer-Based Indirect Structural Health Monitoring of Rail Infrastructure with Attention-Driven Detection and Localization of Transient Defects [1.1782896991259]
  We introduce an incremental synthetic data benchmark designed to evaluate model robustness against progressively complex challenges.<n>We evaluate several established unsupervised models alongside our proposed Attention-Focused Transformer.<n>Our proposed model achieves accuracy comparable to the state-of-the-art solution while demonstrating better inference speed.
  arXiv  Detail & Related papers  (2025-10-08T23:01:53Z)
• Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models [57.49136894315871]
  New paradigm of test-time scaling has yielded remarkable breakthroughs in reasoning models and generative vision models.<n>We propose one solution to the problem of integrating test-time scaling knowledge into a model during post-training.<n>We replace reward guided test-time noise optimization in diffusion models with a Noise Hypernetwork that modulates initial input noise.
  arXiv  Detail & Related papers  (2025-08-13T17:33:37Z)
• CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shifts [67.48102304531734]
  We introduce CNS-Bench, a Continuous Nuisance Shift Benchmark to quantify robustness of image classifiers for continuous and realistic nuisance shifts.<n>We propose a filtering mechanism that outperforms previous methods, thereby enabling reliable benchmarking with generative models.
  arXiv  Detail & Related papers  (2025-07-23T16:15:48Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Reliable Few-shot Learning under Dual Noises [166.53173694689693]
  We propose DEnoised Task Adaptation (DETA++) for reliable few-shot learning.<n>DETA++ employs a memory bank to store and refine clean regions for each inner-task class, based on which a Local Nearestid (LocalNCC) is devised to yield noise-robust predictions on query samples.<n>Extensive experiments demonstrate the effectiveness and flexibility of DETA++.
  arXiv  Detail & Related papers  (2025-06-19T14:05:57Z)
• Leveraging LLM Inconsistency to Boost Pass@k Performance [3.797421474324735]
  Large language models (LLMs) achieve impressive abilities in numerous domains, but exhibit inconsistent performance in response to minor input changes.<n>We introduce a novel method for leveraging models' inconsistency to boost Pass@k performance.<n>Specifically, we present a "Variator" agent that generates k variants of a given task and submits one candidate solution for each one.
  arXiv  Detail & Related papers  (2025-05-19T10:22:04Z)
• Meta Learning-Driven Iterative Refinement for Robust Anomaly Detection in Industrial Inspection [9.132399905884364]
  We propose to leverage the adaptation ability of meta learning approaches to identify and reject noisy training data to improve the learning process. In our model, we employ Model Agnostic Meta Learning (MAML) and an iterative refinement process through an Inter-Quartile Range rejection scheme to enhance their adaptability and robustness.
  arXiv  Detail & Related papers  (2025-03-03T14:11:41Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Robust VAEs via Generating Process of Noise Augmented Data [9.366139389037489]
  This paper introduces a novel framework that enhances robustness by regularizing the latent space divergence between original and noise-augmented data. Our empirical evaluations demonstrate that this approach, termed Robust Augmented Variational Auto-ENcoder (RAVEN), yields superior performance in resisting adversarial inputs.
  arXiv  Detail & Related papers  (2024-07-26T09:55:34Z)
• Advancing the Robustness of Large Language Models through Self-Denoised Smoothing [50.54276872204319]
  Large language models (LLMs) have achieved significant success, but their vulnerability to adversarial perturbations has raised considerable concerns. We propose to leverage the multitasking nature of LLMs to first denoise the noisy inputs and then to make predictions based on these denoised versions. Unlike previous denoised smoothing techniques in computer vision, which require training a separate model to enhance the robustness of LLMs, our method offers significantly better efficiency and flexibility.
  arXiv  Detail & Related papers  (2024-04-18T15:47:00Z)
• Provable Robustness for Streaming Models with a Sliding Window [51.85182389861261]
  In deep learning applications such as online content recommendation and stock market analysis, models use historical data to make predictions. We derive robustness certificates for models that use a fixed-size sliding window over the input stream. Our guarantees hold for the average model performance across the entire stream and are independent of stream size, making them suitable for large data streams.
  arXiv  Detail & Related papers  (2023-03-28T21:02:35Z)
• Confidence-aware Training of Smoothed Classifiers for Certified Robustness [75.95332266383417]
  We use "accuracy under Gaussian noise" as an easy-to-compute proxy of adversarial robustness for an input. Our experiments show that the proposed method consistently exhibits improved certified robustness upon state-of-the-art training methods.
  arXiv  Detail & Related papers  (2022-12-18T03:57:12Z)
• Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection [67.49587673594276]
  We introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem. We demonstrate that the diffusion denoising process (DDP) of DMs serves as a novel form of asymmetric, which is well-suited to enhance the input and mitigate the overconfidence problem. Our experiments on CIFAR10, CIFAR100, and ImageNet show that our method outperforms SOTA approaches.
  arXiv  Detail & Related papers  (2022-11-21T08:45:08Z)
• Bayesian NVH metamodels to assess interior cabin noise using measurement databases [0.0]
  This research work proposes a global NVH metamodeling technique for broadband noises such as aerodynamic and rolling noises. Generalized additive models (GAMs) with bootstraps and Gaussian basis functions are used to model the dependency of sound pressure level (SPL) on predictor variables. Probabilistic modelling is carried out using an open-source library PyMC3.
  arXiv  Detail & Related papers  (2022-06-12T19:48:24Z)
• Adaptive Model Predictive Control by Learning Classifiers [26.052368583196426]
  We propose an adaptive MPC variant that automatically estimates control and model parameters. We leverage recent results showing that BO can be formulated as a density ratio estimation. This is then integrated into a model predictive path integral control framework yielding robust controllers for a variety of challenging robotics tasks.
  arXiv  Detail & Related papers  (2022-03-13T23:22:12Z)
• Certified Adversarial Defenses Meet Out-of-Distribution Corruptions: Benchmarking Robustness and Simple Baselines [65.0803400763215]
  This work critically examines how adversarial robustness guarantees change when state-of-the-art certifiably robust models encounter out-of-distribution data. We propose a novel data augmentation scheme, FourierMix, that produces augmentations to improve the spectral coverage of the training data. We find that FourierMix augmentations help eliminate the spectral bias of certifiably robust models enabling them to achieve significantly better robustness guarantees on a range of OOD benchmarks.
  arXiv  Detail & Related papers  (2021-12-01T17:11:22Z)
• Probabilistic robust linear quadratic regulators with Gaussian processes [73.0364959221845]
  Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design. We present a novel controller synthesis for linearized GP dynamics that yields robust controllers with respect to a probabilistic stability margin.
  arXiv  Detail & Related papers  (2021-05-17T08:36:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.