AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis

• URL: http://arxiv.org/abs/2606.09682v1
• Date: Mon, 08 Jun 2026 16:02:03 GMT
• Title: AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis
• Abstract summary: AutoMegaresearchKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative kernel that runs the whole forward pass in one launch.<n>AMK trails cuBLAS on the high-bandwidth training-class A100/H100, where the harness localizes the cross-SM-sync bottleneck.<n>This is a precision-asymmetric (W8A16 vs bf16) comparison at decode position 0; the largest real checkpoint is TinyLlama-1.1B.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed. A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent-proposed schedule is rejected before launch: across 7,160 adversarial schedules (6,091 unsafe) it had zero false-accepts and accepted all 360 real lowerings. The same source retargets sm_80/sm_90/sm_120 from one codebase, auto-generates correct megakernels for 10 of 10 supported models, and on a real SmolLM2-135M checkpoint reproduces HuggingFace greedy decode token-for-token (perplexity match 2.5e-7). An unattended, agent-drivable autoresearch loop self-improves the megakernel over its own baseline (1.25-1.72x). A search-found int8 (W8A16) megakernel beats CUDA-graphed cuBLAS bf16 at batch-1 decode across NVIDIA's datacenter inference fleet: L4 up to 1.33x, the current-gen L40S 1.25-1.27x, A10G up to 1.08x at scale, and the consumer RTX 5090 1.19-1.23x. The ordering is not a clean function of bandwidth (the 864 GB/s L40S beats the 600 GB/s A10G); the divide is inference-class vs training-class. AMK trails cuBLAS on the high-bandwidth training-class A100/H100, where the harness localizes the cross-SM-sync bottleneck; we report the gap plainly. This is a precision-asymmetric (W8A16 vs bf16) comparison at decode position 0; the largest real checkpoint is TinyLlama-1.1B. Code and the harness: https://github.com/RightNow-AI/AutoMegaKernel

Related papers

• What actually runs: a measurement study of language model placement and decode speed on the Apple Neural Engine [0.0]
  We train matched models across size and precision, with quantized checkpoints byte-identical in structure to their fp16 counterparts.<n>We read the ANE's memory-controller byte counters during inference, establishing what actually ran rather than what the compiler intended.
  arXiv  Detail & Related papers  (2026-08-22T21:22:24Z)
• FusionML: Prefill, Not Decode - Mechanism and Boundaries of CPU+GPU Co-Execution on Unified-Memory Apple Silicon [0.0]
  Previous attempts, including our own, failed or produced precision-confounded wins.<n>We identify the cause: MLX's lazy-graph scheduler emphserializes cross-stream workWhenever a CPU-stream operation consumes an unmaterialized GPU result inside one evaluation graph, a row-split matmul runs x1.38 faster with materialized inputs.<n>We characterize the boundaries equally carefully: decode cannot benefit, bound by shared bandwidth co-execution does not add.
  arXiv  Detail & Related papers  (2026-07-24T11:42:17Z)
• A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi [0.0]
  This report keeps the hardware and asks what a model that fits can do.<n>We deploy Mini-V-4.6, a modern multimodal assistant pairing a SigLIP2 encoder and window-attention merger.<n>The system answers an image question end-to-end in 1.7s.
  arXiv  Detail & Related papers  (2026-07-16T04:48:44Z)
• Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks [75.92297551160692]
  General-purpose agents such as OpenClaw are increasingly used as autonomous tool users.<n>We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol.<n>Claw-SWE-Bench treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation.
  arXiv  Detail & Related papers  (2026-06-10T17:16:23Z)
• Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode [0.0]
  Physical AI systems run a different workload from cloud cloud LLM serving.<n>We measure batch-1 decode for three 7 to 8B-class GQA transformers across four NVIDIA GPUs.<n>The achieved fraction of peak bandwidth falls as peak bandwidth rises.
  arXiv  Detail & Related papers  (2026-05-28T21:03:14Z)
• Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference [17.23666143382396]
  Large language models (LLMs) serve real-time inference in commercial online advertising systems.<n>MegaKernel eliminates launch overhead and interoperator round-trips by fusing multiple operators into a single persistent kernel.<n>Ada-MK: a three-dimensional shared-memory constraint model combined with K-dimension splitting that reduces peak shared memory usage by 50%.<n>On an NVIDIA L20, Ada-MK improves single-batch throughput by up to 23.6% over vanillaRT-LLM and 50.2% over vLLM.
  arXiv  Detail & Related papers  (2026-05-12T06:04:28Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs [75.62489398666644]
  We show that an emphautoresearch-style pipeline citepkarpathy2026autoresearch powered by Claude Code discovers novel white-box adversarial attack.<n>textbfsignificantly outperform all existing (30+ methods) in jailbreaking and prompt injection evaluations.
  arXiv  Detail & Related papers  (2026-03-25T16:50:56Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference [0.0]
  TIDE is a post-training system that attaches tiny learned at periodic checkpoint layers and routers, at inference time, selects the earliest layer whose hidden state has converged for each token.<n>On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining layer at 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%.
  arXiv  Detail & Related papers  (2026-03-22T18:58:07Z)
• Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference [0.0]
  Mamba-2's state space duality algorithm maps cleanly onto what XLA's fusion and tiling passes actually optimise.<n>We implement the full inference path (prefill, cached autoregressive decoding) as shaped standard primitives under XLA.<n>The implementation runs unmodified on CPU, NVIDIA GPU, and Google Cloud TPU from a single JAX source.
  arXiv  Detail & Related papers  (2026-03-10T12:03:00Z)
• Chronicals: A High-Performance Framework for LLM Fine-Tuning with 3.51x Speedup over Unsloth [0.0]
  We present Chronicals, an open-source training framework achieving 3.5x speedup over Unsloth.<n>We provide complete mathematical foundations: online softmax correctness, FlashAttention IO complexity O(N2 d2 M-1), LoRA+ learning rate gradient approximations.
  arXiv  Detail & Related papers  (2026-01-06T00:00:55Z)
• dInfer: An Efficient Inference Framework for Diffusion Language Models [54.80918957287927]
  Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs.<n>We present dInfer, an efficient and framework for dLLM inference.
  arXiv  Detail & Related papers  (2025-10-09T16:19:42Z)
• DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training [82.06732962485754]
  FlashAttention effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. We introduce DISTFLASHATTN, a memory-efficient attention mechanism optimized for long-context LLMs training. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses.
  arXiv  Detail & Related papers  (2023-10-05T03:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.