BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing

• URL: http://arxiv.org/abs/2608.31105v1
• Date: Mon, 31 Aug 2026 17:10:59 GMT
• Title: BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
• Abstract summary: BLOOM-WILT elicits natural multi-turn instances of rare behaviours without training cost or access beyond the target's next-token distribution.<n>We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings.
• Score: 0.898478290216692
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.

Related papers

• How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method [4.37919186746582]
  We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent.<n>We extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective.<n>SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly.
  arXiv  Detail & Related papers  (2026-09-04T15:30:22Z)
• Prediction of Prediction (PoP): Inter-Layer Activation Fusion for Single-Pass Hallucination Detection in Large Language Models [0.0]
  This work evaluates whether internal hidden-state transition dynamics during generation can signal factual errors without auxiliary decoding calls.<n>We introduce Prediction of Prediction (PoP), a mechanism that captures layer-transition uncertainty.<n>PoP achieves an area under the receiver operating characteristic curve (AUROC) of 75.5% for factual-correctness classification.
  arXiv  Detail & Related papers  (2026-08-27T14:17:14Z)
• On-Policy Self-Distillation in Diffusion Models [89.32656931795293]
  Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change.<n>We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries.<n>Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators.
  arXiv  Detail & Related papers  (2026-08-25T14:55:24Z)
• Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks [1.9800951131982487]
  We use Speculative Refinement (SpecRef) to warm-start a masked diffusion language model from an AR draft.<n>We evaluate SpecRef across six benchmarks (HumanEval, MBPP, GSM8K, BBH, ARC-Challenge, HellaSwag) with three distinct evaluation protocols.<n>These observations apply to any multi-stage or non-autoregressive generation pipeline and point toward more diagnostic evaluation practices.
  arXiv  Detail & Related papers  (2026-06-25T18:52:24Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic [16.151531550259993]
  Large language models (LLMs) are increasingly deployed through hosted APIs, making model extraction a practical threat to model ownership and service security.<n>We formulate model extraction monitoring as benign-calibrated traffic-window distribution testing.<n>We show that an embarrassingly simple detector is effective: embed incoming queries into a semantic space and test whether their aggregate distribution deviates from historical benign traffic.
  arXiv  Detail & Related papers  (2026-06-04T05:33:49Z)
• When Individually Calibrated Models Become Collectively Miscalibrated [3.556355987197792]
  We show that individually calibrated predictors can become collectively miscalibrated when their predictions interact strategically.<n>We prove that under Brier-score-based aggregation with positively correlated beliefs, each agent's individually optimal report systematically underestimates the positive-class probability.<n>In contrast, VCG-based aggregation aligns incentives by rewarding marginal contribution, achieving dominant-strategy incentive compatibility and near-optimal performance.
  arXiv  Detail & Related papers  (2026-05-14T05:25:16Z)
• MANTA: Multi-turn Assessment for Nonhuman Thinking & Alignment [0.1452394725421793]
  MANTA is a dynamic multi-turn evaluation framework built on the Inspect AI platform.<n>We present preliminary results from evaluations of claude-sonnet-4-20250514 and openai/gpt-4o.<n>We additionally present STYLEJUDGE, a controlled four-judge study demonstrating systematic format bias in LLM-as-judge evaluation.
  arXiv  Detail & Related papers  (2026-04-18T19:51:32Z)
• Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions [50.1404916337174]
  We present the first large-scale, statistically powered audit of native probabilistic sampling in large language models (LLMs)<n>We show that batch generation achieves only modest statistical validity, with a 13% median pass rate, while independent requests collapse almost entirely.<n>We conclude that current LLMs lack a functional internal sampler, necessitating the use of external tools for applications requiring statistical guarantees.
  arXiv  Detail & Related papers  (2026-01-08T22:33:12Z)
• Test-Time Alignment via Hypothesis Reweighting [56.71167047381817]
  Large pretrained models often struggle with underspecified tasks.<n>We propose a novel framework to address the challenge of aligning models to test-time user intent.
  arXiv  Detail & Related papers  (2024-12-11T23:02:26Z)
• Semantic Self-adaptation: Enhancing Generalization with a Single Sample [45.111358665370524]
  We propose a self-adaptive approach for semantic segmentation. It fine-tunes the parameters of convolutional layers to the input image using consistency regularization. Our empirical study suggests that self-adaptation may complement the established practice of model regularization at training time.
  arXiv  Detail & Related papers  (2022-08-10T12:29:01Z)
• MEMO: Test Time Robustness via Adaptation and Augmentation [131.28104376280197]
  We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions. We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable.
  arXiv  Detail & Related papers  (2021-10-18T17:55:11Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.