The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

• URL: http://arxiv.org/abs/2605.18840v1
• Date: Wed, 13 May 2026 03:19:38 GMT
• Title: The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
• Abstract summary: Leaderboards rank frontier models on independent axes but do not reveal whether capabilities reinforce or trade off across releases.<n>We decompose paired SWE-bench and GPQA Diamond scores into a population coupling trend and per-release residual.<n>We provide a three-level playbook (locate, diagnose, rotate), a per-lab measurement-priority table, and seven falsifiable predictions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Leaderboards rank frontier models on independent axes but do not reveal whether capabilities reinforce or trade off across releases -- and at the frontier, this interaction is the more informative signal. We decompose paired SWE-bench and GPQA Diamond scores into a population coupling trend and per-release residual ($h$-field) that diagnoses capability emphasis and identifies which measurement or stress test is most informative next. Across 34 models from 10 labs (2024--2026), capabilities cooperate ($r = +0.72$, $p < 10^{-6}$), but cooperation varies by lab and over time: DeepSeek reversed from reasoning-rich to coding-first ($h$: $+11.2 \to -4.7$, 15.9-pp swing); Google maintains consistent reasoning emphasis; Anthropic oscillates between coding excursions and recovery. Cooperation is not static -- it cascades. Six open-weight architectures confirm a second capability transition at 30--72B, and SWE-bench is now saturating while HLE and instruction-following retain discriminatory spread -- signaling the next axis rotation. We provide a three-level playbook (locate, diagnose, rotate), a per-lab measurement-priority table, and seven falsifiable predictions with timestamped criteria for the next 12 months of frontier releases. Per-lab coupling slopes vary $5\times$ (Google $1.15$ vs. DeepSeek $0.23$), quantifying how efficiently each recipe converts coding gains into reasoning. Five April 2026 releases confirm the diagnostic out of sample ($r$ rises from $+0.72$ to $+0.75$). An interactive dashboard provides phase classification with actionable recommendations, $h$-field diagnostics, per-lab coupling trajectories, ODE-based scaling predictions, benchmark rotation guidance, self-steering demo, and live tracking of all seven predictions: https://zehenlabs.com/cape/.

Related papers

• Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning [62.24941741019998]
  We formalize fork placement as locating the emphss of the chain's value curve, where the expected outcome turns.<n>Three instantiations, none needing step-level supervision, span access levels: a black-box probe, a logit-lens depth profile, and a learned activation direction.
  arXiv  Detail & Related papers  (2026-09-10T04:06:45Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification [0.8803840644459077]
  Cov-LLE restores function-space diversity that weight-orthonormality cannot, and at matched $K$ recovers much of the diversity and calibration of a deep ensemble.<n> Viewing OC as a last-layer ensemble also organizes detectors into a two-axis taxonomy.
  arXiv  Detail & Related papers  (2026-07-26T21:56:02Z)
• A Spectral Phase Diagram for Binary Few-Shot Classification: Intrinsic Dimensionality, Geometric Saturation, and Representational Diagnosis [0.0]
  The saturation index $S(K) = operatornameerank(widehat_W(K)) / K$ measures the ratio of the effective rank of the pooled within-class sample covariance to the shot count.<n>The index is computable in $O(d3)$ time from support features alone, requiring no test labels or trained classifier.
  arXiv  Detail & Related papers  (2026-06-12T16:46:24Z)
• Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling [0.0]
  We measure the coupling between reasoning and truthfulness across 63 base models from 16 families.<n>We find a regime change invisible to loss curves below a family-dependent critical scale $N_c$, capabilities anticorrelate; above it, they cooperate.
  arXiv  Detail & Related papers  (2026-05-13T03:14:09Z)
• What Do EEG Foundation Models Capture from Human Brain Signals? [64.48249643001402]
  Modern EEG foundation models learn directly from raw signals via self-supervised pretraining.<n>We decompose into three sub-questions: emphwhat does the model learn, emphwhat does the model use, and emphhow much can be explained<n>The audit covers three foundation models (CSBrain, CBraMod, LaBraM), five clinical tasks (MDD, Stress, ISRUC-Sleep, TUSL, Siena), and a 6-family 63-feature lexicon.
  arXiv  Detail & Related papers  (2026-05-12T01:57:53Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race [3.9508043303559828]
  Reproducing an empirical NLP study used to take weeks.<n>All code, $648$ mimic drafts, trained detectors, diagnostics, and adversarial trajectories are released.
  arXiv  Detail & Related papers  (2026-05-04T14:10:41Z)
• Bayesian X-Learner: Calibrated Posterior Inference for Heterogeneous Treatment Effects under Heavy-Tailed Outcomes [0.0]
  We present X-Learner, an X-Learner built on cross-fitted doubly robust pseudo-outcomes.<n>On contaminated "whale" DGPs with up to 20-25% tail density, a one-flag extension (contamination_severity) that selects a Huber-$$ loss per Huber's minimax-$$ relation recovers $(x)$ with tight credible intervals.
  arXiv  Detail & Related papers  (2026-04-30T04:07:11Z)
• When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors [66.18091962164219]
  Existing metrics fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences.<n>We propose two complementary metrics to isolate non-mandatory behavioral patterns: textbfResponse Pattern Similarity (RPS) for verbal alignment and textbfAction Graph Similarity (AGS) for tool-use habits modeled as directed graphs.
  arXiv  Detail & Related papers  (2026-04-23T03:48:56Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Hardware Validation of DAGI via a Modular "Ridge" Signature and High-Order Synergistic Information [0.0]
  DAGI (Directed Acyclic Graph Information) framework on IBM Quantum hardware.<n>Small, controlled experiment whose ideal output distribution is constrained to a low-dimensional modular manifold (a "ridge)<n>Key recovery exceeds chance: per-shot accuracy 0.1689 (chance 0.125, 95% Wilson CI [0.1610, 0.1772])<n>These results support the claim that DAGI detects and quantifies non-trivial, hardware-resilient information structure.
  arXiv  Detail & Related papers  (2026-04-16T14:16:59Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.