Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure

• URL: http://arxiv.org/abs/2608.08722v1
• Date: Sun, 09 Aug 2026 14:16:11 GMT
• Title: Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure
• Abstract summary: Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim.<n>We document this concretely in two GPU- kernel-optimization suites with held-out generalization gates.<n>We give a four-mode taxonomy of these failures, from configuration fingerprints to gate leakage.
• Score: 6.599344783327054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization suites with held-out generalization gates: Metal-Sci (10 scientific-compute tasks) and Metal-ZK (12 zero-knowledge/cryptographic tasks), in which three frontier LLMs (Opus 4.7, Gemini 3.1 Pro, GPT-5.5) propose Metal kernels inside a $(1{+}1)$ evolutionary loop with rich feedback. Although no model is prompted to act adversarially, the promoted winners repeatedly fingerprint the evaluation configuration: they branch on the identity of runtime parameters, tune the measured branch maximally, and leave the unmeasured branch slow or silently wrong. Across the pooled suites, $16/53$ ($30\%$) of in-distribution wins fail to transfer to held-out configurations. We give a four-mode taxonomy of these failures, from configuration fingerprints to gate leakage. We distill design guidance for measurement under strategic optimization: held-out probes retain validity only on non-enumerable axes; gates must measure held-out performance, not just correctness; and a transfer rate is interpretable only with per-failure mechanism grades: ours decomposes into gamed, overfit, and benign. Code and research artifacts: https://github.com/vicgalle/kernel-fingerprinting

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attribution [49.01642339390897]
  Concept-Targeted Attribution (CTA) trains attribution graphs with respect to a linear probe direction.<n>We show that CTA yields probe-specific circuits that explain why an internal concept representation arises in a prompt.
  arXiv  Detail & Related papers  (2026-08-27T08:12:49Z)
• Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware [0.0]
  Single-stream autoregressive decoding of large language models is bound by memory bandwidth.<n>A small draft model proposes $K$ tokens autoregressively, the target model scores all of them in one batched pass.<n>We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations.
  arXiv  Detail & Related papers  (2026-07-19T15:01:35Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Self-Specializing Vision-Language Transmon Chip Calibration in a Physics-Grounded Environment [0.023227405857540805]
  Calibrating a superconducting transmon chip is a sequential decision problem under noise, drift, and a finite budget.<n>We study whether a vision-language agent can close this loop and specialize itself to one physical device without weight updates.
  arXiv  Detail & Related papers  (2026-07-03T10:54:28Z)
• Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL [76.45061154544568]
  Self-play reinforcement learning trains language models on their own generated tasks, co-evolving a proposer and solver without human labels.<n>Recent systems report strong reasoning gains, but collapse and instability are widely observed and poorly understood.<n>We argue instead that self-play stability is governed by two distinct levers: a data-level gate that decides which proposer-generated tasks enter the training pool, and the reward signal that updates the policy on tasks already admitted.
  arXiv  Detail & Related papers  (2026-05-21T09:19:23Z)
• Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon [6.599344783327054]
  Metal-Sci is a 10-task benchmark of scientific Apple Silicon Metal compute kernels.<n>Each task ships a CPU reference, a roofline-anchored fitness function, and a held-out generalization size.<n>We report matched single-model sweeps of Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 on M1 Pro.
  arXiv  Detail & Related papers  (2026-05-10T19:11:26Z)
• GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives [48.545980031973556]
  GAMBIT is a benchmark with three evaluation modes and two independent scores for evaluating imposter detectors.<n>The benchmark comes with a dataset of 27,804 labeled instances spanning 240 co-evolved imposter strategies.
  arXiv  Detail & Related papers  (2026-05-09T16:07:23Z)
• Graduated Trust Gating for IoT Location Verification: Trading Off Detection and Proof Escalation [0.0]
  Existing spoofing detectors output a binary decision, forcing system designers to choose between high false-deny and high false-accept rates.<n>We propose a graduated trust gate that computes a multi-signal integrity score and maps it to three actions: PROCEED, STEP-UP, or DENY.<n>Session-latch mechanism ensures that a single suspicious fix blocks the entire session, preventing post-transition score recovery.
  arXiv  Detail & Related papers  (2026-04-04T23:42:56Z)
• Nondeterminism-Aware Optimistic Verification for Floating-Point Neural Networks [11.159101812934503]
  Nondeterministic Aware Optimistic verification protocol for floating-point execution on heterogeneous accelerators.<n>We implement NAO as a PyTorch-compatible runtime and a contract layer currently deployed on Holesky testnet.
  arXiv  Detail & Related papers  (2025-10-15T21:10:39Z)
• Cloning Games, Black Holes and Cryptography [50.022147589030304]
  We introduce a new toolkit for analyzing cloning games.<n>This framework allows us to analyze a new cloning game based on binary phase states.<n>We show that the binary phase variantally optimal bound offers quantitative insights into information scrambling in idealized models of black holes.
  arXiv  Detail & Related papers  (2024-11-07T14:09:32Z)
• WR-ONE2SET: Towards Well-Calibrated Keyphrase Generation [57.11538133231843]
  Keyphrase generation aims to automatically generate short phrases summarizing an input document. The recently emerged ONE2SET paradigm generates keyphrases as a set and has achieved competitive performance. We propose WR-ONE2SET which extends ONE2SET with an adaptive instance-level cost Weighting strategy and a target Re-assignment mechanism.
  arXiv  Detail & Related papers  (2022-11-13T09:56:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.