Speculative Probing: LLM Monitoring at Speculative-Decoding Cost

• URL: http://arxiv.org/abs/2608.28099v1
• Date: Fri, 28 Aug 2026 09:07:58 GMT
• Title: Speculative Probing: LLM Monitoring at Speculative-Decoding Cost
• Abstract summary: Real-time classification during language model inference is valuable for safety filtering, behavioral analysis, and model monitoring.<n>Current approaches force a trade-off between accuracy and efficiency.<n> speculative-decoding module in recent LLMs can be repurposed for efficient high-quality classification.
• Score: 8.480625921580154
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time classification during language model inference is valuable for safety filtering, behavioral analysis, and model monitoring, but current approaches force a trade-off between accuracy and efficiency. Hidden-state probes are fast but limited: they are either not context-aware: operating on a single vector and cannot model interactions across positions; or they are very costly: having dedicated classifier models (Llama Guard, Qwen Guard, LLM-as-judge) or performing computation on hidden states for all tokens and then pooling the results (MultiMax). This shows an intrinsic trade-off between efficiency and accuracy. However, we find that the speculative-decoding module in recent LLMs can be repurposed for efficient high-quality classification. By appending a trained soft prompt at the end of the target sequence, we can repurpose the speculative-decoding module into a sequence classifier. At inference time in a speculative-decoding pipeline, the KV cache is already in GPU memory, so classification adds negligible overhead. We evaluate on four classification tasks across four models (Qwen3.5-4B, 9B, 27B, MiniCPM4.1-8B). Our small probes consistently outperform zero-shot GPT-5.4-mini and, on multilingual prompt safety, match or beat specialized 8B safety classifiers (Qwen3Guard-Gen-8B, Llama-Guard-3-8B) without running a full LLM.

Related papers

• GLiGuard: Schema-Conditioned Classification for LLM Safeguard [5.918280835312565]
  We introduce textbfGLiGuard, a bidirectional encoder adapted from GLiNER2 for LLM content moderation.<n>The key idea is to encode task definitions and label semantics directly into the input sequence as structured token schemas.<n>GLiGuard achieves F1 scores competitive with 7B--27B decoder-based guards despite being 23--90$times$ smaller.
  arXiv  Detail & Related papers  (2026-05-08T16:44:07Z)
• Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs [12.31163751026309]
  We evaluate 8 open-source code LLMs on 19 benchmarks across four task families.<n> Sensitivity patterns vary widely across models and tasks.<n>For these datasets, models may rely more on learned generalization than direct memorization.
  arXiv  Detail & Related papers  (2026-04-15T15:43:10Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• Evaluating Large Language Models for Gait Classification Using Text-Encoded Kinematic Waveforms [1.1635287453977645]
  Large Language Models (LLMs) may offer explanatory capabilities and confidence-aware outputs when applied to structured kinematic data.<n>This study evaluated whether general-purpose LLMs can classify continuous gait kinematics when represented as textual numeric sequences.
  arXiv  Detail & Related papers  (2026-03-04T10:23:35Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification [2.0069888187253615]
  Production LLM systems often rely on separate models for safety and other classification-heavy steps.<n>We instead reuse computation already paid for by the serving LLM: we train lightweight probes on its hidden states and predict labels in the same forward pass used for generation.
  arXiv  Detail & Related papers  (2026-01-19T18:40:29Z)
• Beyond Linear Probes: Dynamic Safety Monitoring for Language Models [67.15793594651609]
  Traditional safety monitors require the same amount of compute for every query.<n>We introduce Truncated Polynomials (TPCs), a natural extension of linear probes for dynamic activation monitoring.<n>Our key insight is that TPCs can be trained and evaluated progressively, term-by-term.
  arXiv  Detail & Related papers  (2025-09-30T13:32:59Z)
• GLiClass: Generalist Lightweight Model for Sequence Classification Tasks [49.2639069781367]
  We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks.<n>Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios.
  arXiv  Detail & Related papers  (2025-08-11T06:22:25Z)
• Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributions [60.43398881149664]
  We introduce LOS-Net, a lightweight attention-based architecture trained on an efficient encoding of the LLM Output Signature.<n>It achieves superior performance across diverse benchmarks and LLMs, while maintaining extremely low detection latency.
  arXiv  Detail & Related papers  (2025-03-18T09:04:37Z)
• Do LLMs Understand the Safety of Their Inputs? Training-Free Moderation via Latent Prototypes [1.0779346838250028]
  Latent Prototype Moderator (LPM) is a training-free moderation method that uses Mahalanobis distance in latent space to assess input safety.<n>LPM matches or exceeds state-of-the-art guard models across multiple safety benchmarks.
  arXiv  Detail & Related papers  (2025-02-22T10:31:50Z)
• A Little Confidence Goes a Long Way [3.6371715211657243]
  We introduce a group of related methods for binary classification tasks using probes of the hidden state activations in large language models (LLMs) Performance is on par with the largest and most advanced LLMs currently available, but requiring orders of magnitude fewer computational resources and not requiring labeled data.
  arXiv  Detail & Related papers  (2024-08-20T23:36:00Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.