Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels

• URL: http://arxiv.org/abs/2607.16228v2
• Date: Tue, 21 Jul 2026 16:12:17 GMT
• Title: Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels
• Abstract summary: tensor- kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances.<n>We ask what absolute tolerance would the kernel itself, observed under its correct implementation, justify?<n>The answer is much tighter than the current hand-picked atol.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Most tensor-kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances. The thresholds are copied across the corpus and rarely revisited. We mine the element-wise error distribution of every test case from accumulated cloud GPU runs across the 26-entry gpuemu corpus and 2 dtypes (8,076 result rows). We then ask one empirical question: what absolute tolerance would the kernel itself, observed under its correct implementation, justify? The answer is much tighter than the current hand-picked atol. The largest tightening is attention_triton fp16 at $2{,}184\times$. Restricted to the seven LLM-style buggy variants for which the corpus ships a paired correct counterpart, calibrated per-(op, dtype) tolerances raise bug-detection recall from 73.2% (1,805 of 2,467) to 82.4% (2,034 of 2,467), an absolute gain of 9.3 percentage points (+229 new detections). The control false-positive count rises from 0 to 20 out of 1,882 correct-control cases (+1.1 percentage points).

Related papers

• Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance [0.10152838128195468]
  Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance.<n>We find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing.<n>Four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales.
  arXiv  Detail & Related papers  (2026-08-25T03:05:54Z)
• The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference [0.10152838128195468]
  Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.<n>We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel inside vLLM.<n>At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64).
  arXiv  Detail & Related papers  (2026-08-13T20:34:36Z)
• A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family [0.9740025522928777]
  Systems that generate GPU kernels with language models report high correctness rates.<n>Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference.<n>We build a contract-grade verifier of twelve adversarial gates, each a property a correct kernel must satisfy, several of them tolerance-free.
  arXiv  Detail & Related papers  (2026-08-13T01:25:56Z)
• Bimanual Manipulation Within an 8 GB Budget: Zero-Copy Sensing and Quantized ACT on an Entry-Level Jetson [35.18016233072556]
  We present a bimanual SO-101 system running entirely on an NVIDIA Jetson Orin Nano Super (8 GB)<n>We build a GStreamer capture pipeline backed by NVMM buffers that removes redundant host-device copies from three-camera sensing.<n>We train ACT and Diffusion Policy on identical demonstrations, each at its own reference budget.
  arXiv  Detail & Related papers  (2026-08-04T17:09:41Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy [17.48959208263029]
  We introduce emphmean-to-score (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map.<n>For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly.
  arXiv  Detail & Related papers  (2026-07-23T14:37:30Z)
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary [0.0]
  Can a language model read the quality of its ongoing computation?<n>Can an external intervention turn that readout into better outcomes?<n>We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT.
  arXiv  Detail & Related papers  (2026-07-20T22:40:36Z)
• Active Quantum Kernel Acquisition for Gaussian Process Regression [30.032230630492155]
  Recent work on quantum kernel classification has shown that allocating shots non-uniformly across kernel entries can reduce the shot budget required to reach a target accuracy.<n>We extend this idea to Gaussian process (GP) regression, a setting whose downstream quantities (full-spectrum posterior variance, log-determinant, marginal likelihood) couple to kernel error more tightly than the sign-only outputs of classification.<n>We derive three closed-form pair-level sensitivities predictive coupling $|_i_j|$, leave-one-out residual, and marginal-likelihood gradient and plug them into a Neyman-style minimum
  arXiv  Detail & Related papers  (2026-06-27T09:40:20Z)
• Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs [0.0]
  We evaluate seven test-generation strategies across a closeop corpus.<n>We report each strategy on two axes: bug recall and control false-positive (FP) rate.
  arXiv  Detail & Related papers  (2026-06-23T21:43:30Z)
• The Correctness Illusion in LLM-Generated GPU Kernels [0.0]
  Benchmarks for LLM-generated GPU kernels score correctness through fixed-shape, small-sample allclose-style checks.<n>We construct a controlled corpus of 24 Triton and CPU stand-in kernels.<n>We re-run the same protocol on five GPU classes.
  arXiv  Detail & Related papers  (2026-06-18T11:52:46Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection [0.0]
  Multi-agent debate improves factuality and reasoning, but most recipes pick a fixed round count.<n>We adapt Wald's Sequential Probability Ratio Test (SPRT) as a plug-in compute governor for LLM debates.<n>On GSM8K the rule stops in 1.01 average rounds (4.06 LLM calls) at 97.0% accuracy vs 99.0% for fixed-5 debate at 15 calls: a 3.7x call reduction at -2pp accuracy.<n>On MMLU the calibrated KL collapses to about 0 and the rule caps on 99.5% of items at 2.1x cost.
  arXiv  Detail & Related papers  (2026-05-18T23:43:12Z)
• Adaptive Calibration in Non-Stationary Environments [44.81344039432424]
  We develop a suite of algorithms that achieve adaptive calibration guarantees under multiple calibration measures.<n>Our approach builds on and extends prior work [Hu et al., 2026, Luo et al., 2025], introducing a novel non-uniform partition of the prediction space that allocates finer resolution near the underlying ground truth.
  arXiv  Detail & Related papers  (2026-05-12T04:06:40Z)
• KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels [41.03500441875287]
  LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered.<n>We present KernelBenchX, a benchmark designed to answer this question through category-aware evaluation of correctness and hardware efficiency.
  arXiv  Detail & Related papers  (2026-05-06T14:18:36Z)
• The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime [0.0]
  Most cited calibration result in deep learning -- post-temperature-scaling ECE of 0.012 on CIFAR-100 -- is below the statistical noise floor.<n>We prove this is not a failure of the experiment but a law: the minimax rate for estimating calibration error with model error rate epsilon is Theta((Lepsilon/m)2/3), and no estimator can beat it.
  arXiv  Detail & Related papers  (2026-04-14T16:48:24Z)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference [31.2331188304598]
  Changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses.<n>We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision.<n>Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32.
  arXiv  Detail & Related papers  (2025-06-11T08:23:53Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)
• KSD Aggregated Goodness-of-fit Test [38.45086141837479]
  We introduce a strategy to construct a test, called KSDAgg, which aggregates multiple tests with different kernels. We provide non-asymptotic guarantees on the power of KSDAgg. We find that KSDAgg outperforms other state-of-the-art adaptive KSD-based goodness-of-fit testing procedures.
  arXiv  Detail & Related papers  (2022-02-02T00:33:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.