MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference

• URL: http://arxiv.org/abs/2605.30218v1
• Date: Thu, 28 May 2026 16:50:19 GMT
• Title: MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference
• Abstract summary: MarginGate restores 100% sequence-level deterministic decoding on Llama-3.1-8B and Qwen2.5-14B with 18.56%/15.05% verifier trigger rates.<n>We ask whether verification can be applied exclusively to flipped tokens.
• Score: 7.261960634383562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can emit different tokens when decoded alone or inside a larger batch. Existing fixes use batch-invariant operators or LLM-42's per-token verification, incurring cost even when most steps are stable. We ask whether verification can be applied exclusively to flipped tokens. Across five models, batch-induced token flips are sparse on the flip-rate benchmarks: on MATH500, Llama-3.1-8B flips on $0.48\%$ of synchronous decode steps, and all tested models stay within the 0.3-1.3% range on MATH500, GSM8K, and HumanEval. K/V perturbations remain flat before flips, while low top-1/top-2 logit margins expose much of the flip risk. MarginGate turns these observations into a verifier policy: it keeps BF16 decoding on high-margin steps, verifies only low-margin steps, and repairs confirmed mismatches by replacing the current K/V column. We evaluate on four datasets, calibrating on MATH500 and transferring to GSM8K, SharedGPT, and HumanEval. MarginGate restores 100% sequence-level deterministic decoding on Llama-3.1-8B and Qwen2.5-14B with 18.56%/15.05% verifier trigger rates, reducing LLM-42's latency increment by 2.23x/1.99x relative to always-on verification. On DSR1-Distill-Qwen-7B, the same policy reaches determinism in a harder regime at 49.50% triggers.

Related papers

• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models [29.673427534201654]
  We introduce Facet-Probe, a five-facet audit of multimodal large language models.<n>We find that none of the 18 MLLMs we audit are order-invariant.<n>We propose cross-ordering flip rate as a standard reporting axis for MLLMs.
  arXiv  Detail & Related papers  (2026-06-24T17:53:26Z)
• Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation [0.0]
  Per-token counterfactual credit estimation asks which token in a language-model rollout caused the final answer to be right or wrong.<n>We measure what that assumption costs on a stock inference engine, with a three-pass design.<n>Across six configurations and three models, re-feeding changes the credit estimate at rates 14-28 percentage points above the replica floor.
  arXiv  Detail & Related papers  (2026-06-14T06:09:16Z)
• A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving [0.0]
  Safety evaluations of language models often treat serving configuration as fixed background infrastructure.<n>We synthesize four artifact-backed studies into a paired testing protocol.<n>Standard vLLM reproduces 22/55 label flips on current score-flip candidates, while enabling VLLM_BATCH_INIANT=1 reduces the same test to 0/55 flips.
  arXiv  Detail & Related papers  (2026-05-26T23:22:55Z)
• Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection [0.0]
  Multi-agent debate improves factuality and reasoning, but most recipes pick a fixed round count.<n>We adapt Wald's Sequential Probability Ratio Test (SPRT) as a plug-in compute governor for LLM debates.<n>On GSM8K the rule stops in 1.01 average rounds (4.06 LLM calls) at 97.0% accuracy vs 99.0% for fixed-5 debate at 15 calls: a 3.7x call reduction at -2pp accuracy.<n>On MMLU the calibrated KL collapses to about 0 and the rule caps on 99.5% of items at 2.1x cost.
  arXiv  Detail & Related papers  (2026-05-18T23:43:12Z)
• Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steering [4.032680910442999]
  Large language models commit unrecoverable reasoning errors mid-generation.<n>We introduce $textbfLatent Phase-Shift Rollback$ (LPSR)<n>At each generation step, we monitor the residual stream at a critical layer lcrit, detect abrupt directional reversals (phase shifts) via a cosine-similarity $+$ entropy dual gate.<n>No fine-tuning, gradient computation, or additional forward passes are required.
  arXiv  Detail & Related papers  (2026-04-20T17:53:33Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime [0.0]
  Most cited calibration result in deep learning -- post-temperature-scaling ECE of 0.012 on CIFAR-100 -- is below the statistical noise floor.<n>We prove this is not a failure of the experiment but a law: the minimax rate for estimating calibration error with model error rate epsilon is Theta((Lepsilon/m)2/3), and no estimator can beat it.
  arXiv  Detail & Related papers  (2026-04-14T16:48:24Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks [12.396822247035578]
  We present exMT, a benchmark for objective extraction and metacognition.<n>Given a multi-turn transcript, a model must output a one-sentence base objective and a self-reported confidence.<n> Accuracy is scored by similarity to gold objectives, then thresholded once on 300 calibration items.
  arXiv  Detail & Related papers  (2025-08-23T03:32:04Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• MarginMatch: Improving Semi-Supervised Learning with Pseudo-Margins [73.17295479535161]
  MarginMatch is a new SSL approach combining consistency regularization and pseudo-labeling. We analyze the behavior of the model on the pseudo-labeled examples as the training progresses to ensure low quality predictions are masked out. We obtain an improvement in error rate over the state-of-the-art of 3.25% on CIFAR-100 with only 25 labels per class and of 3.78% on STL-10 using as few as 4 labels per class.
  arXiv  Detail & Related papers  (2023-08-17T15:19:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.