Submodular Benchmark Selection

• URL: http://arxiv.org/abs/2605.02209v1
• Date: Mon, 04 May 2026 04:17:02 GMT
• Title: Submodular Benchmark Selection
• Abstract summary: Entropy and mutual information between selected and remaining benchmarks arise as natural objectives.<n>Experiments on three matrices from ten public leaderboards show that mutual information selection outperforms entropy for imputation at small subsets.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating large language models across many benchmarks is expensive, yet many benchmarks are highly correlated. We formalize the selection of a small, informative subset as submodular maximization under a multivariate Gaussian model. Entropy (log-determinant covariance) and mutual information between selected and remaining benchmarks arise as natural objectives. Both are submodular; entropy selection coincides with pivoted Cholesky and has spectral residual bounds, while mutual information is non-monotone in general but empirically monotone for small subsets, so we optimize it greedily. Experiments on three matrices from ten public leaderboards show that mutual information selection outperforms entropy for imputation at small subsets.

Related papers

• Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks [31.891185002244146]
  In evaluation-unsupervised benchmark coreset selection, the selection algorithm uses no model evaluation outcomes.<n>We develop and evaluate many different submodular functions for this purpose, including determinantal point process (DPP) based approaches.<n>We show our proposed objective is not limited to the evaluation-unsupervised regime: in the setting where only a handful of whole benchmarks must be selected, the same objective matches or outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-07-02T18:37:18Z)
• Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications [79.53938312089308]
  The MIDX-Sampler is a novel adaptive sampling strategy based on an inverted multi-index approach.<n>Our method is backed by rigorous theoretical analysis, addressing key concerns such as sampling bias, gradient bias, convergence rates, and generalization error bounds.
  arXiv  Detail & Related papers  (2025-01-15T04:09:21Z)
• Generalized Grade-of-Membership Estimation for High-dimensional Locally Dependent Data [6.626575011678484]
  Mixed membership models are widely used for analyzing survey responses and population genetics data.<n>Existing approaches, such as Bayesian MCMC inference, are not scalable and lack theoretical guarantees in high-dimensional settings.<n>We introduce a novel and simple approach that flattens the three-way quasi-tensor into a "fat" matrix, and then perform a singular value decomposition of it to estimate parameters.
  arXiv  Detail & Related papers  (2024-12-27T18:51:15Z)
• How many classifiers do we need? [50.69951049206484]
  We provide a detailed analysis of how the disagreement and the polarization among classifiers relate to the performance gain achieved by aggregating individual classifiers. We prove results for the behavior of the disagreement in terms of the number of classifiers. Our theories and claims are supported by empirical results on several image classification tasks with various types of neural networks.
  arXiv  Detail & Related papers  (2024-11-01T02:59:56Z)
• A Consistent and Scalable Algorithm for Best Subset Selection in Single Index Models [1.233738817269826]
  Best-subset selection in high-dimensional models is known to be computationally intractable.<n>Existing proxy algorithms are appealing but do not yield the bestsubset solution.<n>We propose a provably scalable algorithm for the best-subset selection in high-dimensional SIMs.
  arXiv  Detail & Related papers  (2023-09-12T13:48:06Z)
• Variable Selection for Kernel Two-Sample Tests [9.581064100940381]
  We consider the variable selection problem for two-sample tests.<n>We propose a novel framework based on the kernel maximum mean discrepancy (MMD)<n>Our approach seeks a subset of variables with a pre-specified size that maximizes the variance-regularized kernel MMD statistic.
  arXiv  Detail & Related papers  (2023-02-15T00:39:56Z)
• Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation [64.49871502193477]
  We propose a novel semi-supervised subspace clustering method, which is able to simultaneously augment the initial supervisory information and construct a discriminative affinity matrix. Comprehensive experimental results on six commonly-used benchmark datasets demonstrate the superiority of our method over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-05-21T01:47:17Z)
• Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences [52.6022911513076]
  Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. We propose Linformer and Informer to reduce the quadratic complexity to linear (modulo logarithmic factors) via low-dimensional projection and row selection. Based on the theoretical analysis, we propose Skeinformer to accelerate self-attention and further improve the accuracy of matrix approximation to self-attention.
  arXiv  Detail & Related papers  (2021-12-10T06:58:05Z)
• Information-theoretic Feature Selection via Tensor Decomposition and Submodularity [38.05393186002834]
  We introduce a low-rank tensor model of the joint PMF of all variables and indirect targeting as a way of mitigating complexity and maximizing the classification performance for a given number of features. By indirectly aiming to predict the latent variable of the naive Bayes model instead of the original target variable, it is possible to formulate the feature selection problem as of a monotone submodular function subject to a cardinality constraint.
  arXiv  Detail & Related papers  (2020-10-30T10:36:46Z)
• Bayesian Attention Modules [65.52970388117923]
  We propose a scalable version of attention that is easy to implement and optimize. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.
  arXiv  Detail & Related papers  (2020-10-20T20:30:55Z)
• A Group Norm Regularized Factorization Model for Subspace Segmentation [4.926716472066594]
  This paper proposes a group norm regularized factorization model (GNRFM) inspired by the LRR model for subspace segmentation. Specifically, we adopt group norm regularization to make the columns of the factor matrix sparse, thereby achieving a purpose of low rank. Compared with traditional models and algorithms, the proposed method is faster and more robust to noise, so the final clustering results are better.
  arXiv  Detail & Related papers  (2020-01-08T15:20:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.