Linear Probe Accuracy Scales with Model Size and Benefits from Multi-Layer Ensembling

• URL: http://arxiv.org/abs/2604.13386v1
• Date: Wed, 15 Apr 2026 01:21:52 GMT
• Title: Linear Probe Accuracy Scales with Model Size and Benefits from Multi-Layer Ensembling
• Abstract summary: Linear probes can detect when language models produce outputs they "know" are wrong.<n>We show that combining probes from multiple layers into an ensemble recovers strong performance even where single-layer probes fail.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear probes can detect when language models produce outputs they "know" are wrong, a capability relevant to both deception and reward hacking. However, single-layer probes are fragile: the best layer varies across models and tasks, and probes fail entirely on some deception types. We show that combining probes from multiple layers into an ensemble recovers strong performance even where single-layer probes fail, improving AUROC by +29% on Insider Trading and +78% on Harm-Pressure Knowledge. Across 12 models (0.5B--176B parameters), we find probe accuracy improves with scale: ~5% AUROC per 10x parameters (R=0.81). Geometrically, deception directions rotate gradually across layers rather than appearing at one location, explaining both why single-layer probes are brittle and why multi-layer ensembles succeed.

Related papers

• The Hallucination Signal Is a Mean Shift: Why Simple Probes Suffice [24.66711854680014]
  Hidden-state probes effectively detect LLM hallucinations, but the geometry of the signal remains poorly characterized.<n>Across three 7B-scale models and three datasets in a paired-example paradigm, we find the signal overwhelmingly dominated by a single mean-shift component.<n> Shrinkage linear discriminant analysis closes about 73% of the gap between 1D and full-dimensional classifiers.<n>Because the signal spans a contiguous layer band, LayerMix aggregates it to match oracle-layer performance without oracle access.
  arXiv  Detail & Related papers  (2026-08-28T23:01:21Z)
• Flow Map Denoisers: Traversing the Distortion-Perception Plane for Inverse Problems [48.501415601682815]
  Image restoration faces a fundamental tradeoff: methods that minimize error produce blurry reconstructions, while those that maximize perceptual quality yield sharp but less faithful images.<n>We show that flow map models implicitly define a one- parameter family of denoisers that continuously spans the distortion perception frontier.<n>Within a Plug-and-Play solver, the same mechanism extends to general inverse problems, where it controls a tradeoff between perceptual alignment and data consistency.
  arXiv  Detail & Related papers  (2026-06-18T05:15:43Z)
• Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations [5.218766876318545]
  Linear probes trained on LLM activations are increasingly proposed as deception-detection metrics, yet report AUROC exceeding 0.96 on clean benchmarks while collapsing under distributional shift.<n>This paper systematically pressure-tests probe-based metrics across the Gemma 3 model family (1B-27B parameters)<n>We test four hypotheses about deception encoding: (1) single linear direction, (2) multi-dimensional subspace, (3) convex conic hull, (4) entropy proxy.
  arXiv  Detail & Related papers  (2026-05-27T04:51:55Z)
• Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations [55.251494694783894]
  Disaster Operational Response Agent benchmark (DORA) is the first agentic benchmark for end-to-end disaster response.<n>Tasks span five dimensions that cover the operational disaster-response pipeline: disaster perception, spatial relational analysis, rescue and evacuation planning, temporal evolution reasoning, and multi-modal report synthesis.<n>DORA establishes a rigorous testbed for operationally reliable disaster-response agents.
  arXiv  Detail & Related papers  (2026-05-12T06:57:41Z)
• Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement [0.0]
  Vision-language models encode geometry that their text pathway fails to express.<n>LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees.<n>These findings enable a single frozen backbone to function as a multi-task geometric sensor.
  arXiv  Detail & Related papers  (2026-03-06T16:48:27Z)
• Building Production-Ready Probes For Gemini [21.610660065701243]
  We find that the shift from short-context to long-context inputs is difficult for existing probe architectures.<n>We propose several new probe architectures that handle this long-context distribution shift.<n>We find early positive results using AlphaEvolve to automate improvements in both probe architecture search and adaptive red teaming.
  arXiv  Detail & Related papers  (2026-01-16T18:54:29Z)
• Caught in the Act: a mechanistic approach to detecting deception [0.1013295809149289]
  We show that linear probes on LLMs can detect deception in their responses with extremely high accuracy.<n>We observe that probes on smaller models (1.5B) achieve chance accuracy at detecting deception, while larger models (greater than 7B) reach 70-80%.<n>We find multitudes of linear directions that encode deception, ranging from 20 in Qwen 3B to nearly 100 in DeepSeek 7B and Qwen 14B models.
  arXiv  Detail & Related papers  (2025-08-27T01:29:52Z)
• Distilling Parallel Gradients for Fast ODE Solvers of Diffusion Models [53.087070073434845]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face image quality degradation under a low-latency budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as ours), a novel ODE solver that mitigates truncation errors by incorporating multiple parallel gradient evaluations in each ODE step.
  arXiv  Detail & Related papers  (2025-07-20T03:08:06Z)
• A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detection [13.109309606764754]
  We introduce a plug-in detection framework that leverages internal layer-wise inconsistencies within the target model itself.<n>Our method achieves state-of-the-art detection performance with negligible computational overhead.
  arXiv  Detail & Related papers  (2025-05-19T00:48:53Z)
• Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable [70.77600345240867]
  A novel arbitrary-in-arbitrary-out (AIAO) strategy makes watermarks resilient to fine-tuning-based removal. Unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO.
  arXiv  Detail & Related papers  (2024-05-01T12:03:39Z)
• Global Context Aggregation Network for Lightweight Saliency Detection of Surface Defects [70.48554424894728]
  We develop a Global Context Aggregation Network (GCANet) for lightweight saliency detection of surface defects on the encoder-decoder structure. First, we introduce a novel transformer encoder on the top layer of the lightweight backbone, which captures global context information through a novel Depth-wise Self-Attention (DSA) module. The experimental results on three public defect datasets demonstrate that the proposed network achieves a better trade-off between accuracy and running efficiency compared with other 17 state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-09-22T06:19:11Z)
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty [58.144520501201995]
  Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
  arXiv  Detail & Related papers  (2021-10-12T22:04:19Z)
• Low-Complexity Probing via Finding Subnetworks [101.43611844196064]
  We propose a subtractive-based probe for probing neural networks for linguistic properties. We find that the subnetwork probe achieves both higher accuracy on pre-trained models and lower accuracy on random models. By varying the complexity of each probe, we show that subnetwork probing-dominates probing in that it higher accuracy given any budget of probe complexity.
  arXiv  Detail & Related papers  (2021-04-08T05:11:21Z)
• Hard-label Manifolds: Unexpected Advantages of Query Efficiency for Finding On-manifold Adversarial Examples [67.23103682776049]
  Recent zeroth order hard-label attacks on image classification models have shown comparable performance to their first-order, gradient-level alternatives. It was recently shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors. We propose an information-theoretic argument based on a noisy manifold distance oracle, which leaks manifold information through the adversary's gradient estimate.
  arXiv  Detail & Related papers  (2021-03-04T20:53:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.