A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family

• URL: http://arxiv.org/abs/2608.12700v2
• Date: Mon, 17 Aug 2026 20:55:05 GMT
• Title: A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family
• Abstract summary: Systems that generate GPU kernels with language models report high correctness rates.<n>Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference.<n>We build a contract-grade verifier of twelve adversarial gates, each a property a correct kernel must satisfy, several of them tolerance-free.
• Score: 0.9740025522928777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Systems that generate GPU kernels with language models report high correctness rates. Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference. A kernel can pass that test and still be silently wrong. It can return an ordinary number where the true answer is a NaN or an infinity, differ from run to run, break when the shape changes, or accumulate in fp16 where the reference keeps an fp32 total. We build the instrument that checks correctness properly: a contract-grade verifier of twelve adversarial gates, each a property a correct kernel must satisfy, several of them tolerance-free, so no choice of threshold can explain a failure away. Aimed outward, the verifier audits 2,638 machine-generated kernels that a public system's own harness had already accepted as correct. It finds 39.5% broken beyond any tolerance argument and 62.1% carrying at least one violation. The field's standard test accepts 1,487 kernels the verifier rejects, against only 14 the other way. We defend the finding four independent ways: a 7/7 positive control, a threshold-calibration sweep, 98.5% agreement with the reference benchmark's own correctness code, and a stratified hand-audit. Aimed inward, the verifier judges a kernel of our own: the first native Blackwell tcgen05 training backward for the gated-linear-recurrence (GDN) family, including the reverse-state stage the field still runs on a fallback. We establish its correctness independently, against a double-precision oracle, and train five family members through it. The correctness signal behind reported progress in kernel generation is far weaker than the numbers suggest, and a set of tolerance-free contracts would close most of the gap.

Related papers

• Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR [0.0]
  Reinforcement learning with rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer into a binary reward.<n>Prior work reports that one evaluation harness accepts only about 94% of its own ground truth answers, blaming parsing.<n>We supply the decomposition. We apply metamorphic testing to the verifier rather than the model, generating certified equivalent answer variants.<n>We measure rejection per answer category across four widely used verifiers over 307,420 verdicts.
  arXiv  Detail & Related papers  (2026-09-01T14:57:59Z)
• Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets [13.85834524293015]
  We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with.<n>Models judge whether a candidate belongs far better than they author the set itself.
  arXiv  Detail & Related papers  (2026-08-02T05:00:44Z)
• Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs [0.0]
  We evaluate seven test-generation strategies across a closeop corpus.<n>We report each strategy on two axes: bug recall and control false-positive (FP) rate.
  arXiv  Detail & Related papers  (2026-06-23T21:43:30Z)
• Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels [0.0]
  tensor- kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances.<n>We ask what absolute tolerance would the kernel itself, observed under its correct implementation, justify?<n>The answer is much tighter than the current hand-picked atol.
  arXiv  Detail & Related papers  (2026-06-23T20:50:36Z)
• The Correctness Illusion in LLM-Generated GPU Kernels [0.0]
  Benchmarks for LLM-generated GPU kernels score correctness through fixed-shape, small-sample allclose-style checks.<n>We construct a controlled corpus of 24 Triton and CPU stand-in kernels.<n>We re-run the same protocol on five GPU classes.
  arXiv  Detail & Related papers  (2026-06-18T11:52:46Z)
• DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models [6.628401122676601]
  A rule-based logic solver resolves every instance in our benchmark in under 50 microseconds with 100% accuracy.<n>We introduce DeFAb (Defeasible Abduction Benchmark), a dataset and generation pipeline.
  arXiv  Detail & Related papers  (2026-06-17T00:13:40Z)
• Verifiable Rewards Beyond Math and Code: Lightweight Corpus-Grounded Process Supervision for Factual Question Answering [44.82662196757139]
  We propose a lightweight, plug-in-ready process reward that replaces neural verifiers with a corpus-grounded signal from Wikipedia.<n>Across 30 (model, benchmark) cells spanning six instruction-tuned models, CorVer improves over the raw baseline for every cell.<n>It also outperforms four neural-verifier baselines in 18 of 20 cells under their feasible configurations, while training 4.8 to 8.4x faster.
  arXiv  Detail & Related papers  (2026-05-28T09:14:37Z)
• CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution [52.691495954442985]
  CoVerRL is a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other.<n> Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks.<n>Self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
  arXiv  Detail & Related papers  (2026-03-18T14:38:55Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• PRover: Proof Generation for Interpretable Reasoning over Rules [81.40404921232192]
  We propose a transformer-based model that answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation.
  arXiv  Detail & Related papers  (2020-10-06T15:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.