ModelLens: Finding the Best for Your Task from Myriads of Models

• URL: http://arxiv.org/abs/2605.07075v1
• Date: Fri, 08 May 2026 00:49:05 GMT
• Title: ModelLens: Finding the Best for Your Task from Myriads of Models
• Abstract summary: We introduce ModelLens, a unified framework for model recommendation in the wild.<n>Public leaderboard interactions, though scattered and noisy, collectively trace out an implicit atlas of model capabilities.<n>ModelLens ranks unseen models on unseen datasets without running candidates on the target dataset.
• Score: 25.958674884209557
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The open-source model ecosystem now contains hundreds of thousands of pretrained models, yet picking the best model for a new dataset is increasingly infeasible: new models and unbenchmarked datasets emerge continuously, leaving practitioners with no prior records on either side. Existing approaches handle only fragments of this in-the-wild setting: AutoML and transferability estimation select models from small predefined pools or require expensive per-model forward passes on the target dataset, while model routing presupposes a given candidate pool. We introduce ModelLens, a unified framework for model recommendation in the wild. Our key insight is that public leaderboard interactions, though scattered and noisy, collectively trace out an implicit atlas of model capabilities across heterogeneous evaluation settings, a signal rich enough to learn from directly. By learning a performance-aware latent space over model--dataset--metric tuples, ModelLens ranks unseen models on unseen datasets without running candidates on the target dataset. On a new benchmark of 1.62M evaluation records spanning 47K models and 9.6K datasets, ModelLens surpasses baselines that either rely on metadata alone or require running each candidate on the target dataset. Its recommended Top-K pools further improve multiple representative routing methods by up to 81% across diverse QA benchmarks. Case studies on recently released benchmarks further confirm generalization to both text and vision-language tasks.

Related papers

• LOCUS: Low-Dimensional Model Embeddings for Efficient Model Exploration, Comparison, and Selection [15.182368486530128]
  We propose LOCUS, a method that produces low-dimensional vector embeddings that compactly represent a language model's capabilities across queries.<n>LOCUS is an attention-based approach that generates embeddings by a deterministic forward pass over query encodings and evaluation scores via an encoder model.<n>We train a correctness predictor that uses model embeddings and query encodings to achieve state-of-the-art routing accuracy on unseen queries.
  arXiv  Detail & Related papers  (2026-01-28T22:09:42Z)
• Consensus-Driven Active Model Selection [29.150990754584978]
  We propose a method for active model selection using predictions from candidate models to prioritize the labeling of test data points.<n>Our method, CODA, performs consensus-driven active model selection by modeling relationships between categories, and data points.<n>We validate our approach by curating a collection of 26 benchmark tasks capturing a range of model selection scenarios.
  arXiv  Detail & Related papers  (2025-07-31T17:56:28Z)
• Approximating Language Model Training Data from Weights [70.08614275061689]
  We formalize the problem of data approximation from model weights and propose several baselines and metrics.<n>We develop a gradient-based approach that selects the highest-matching data from a large public text corpus.<n>Even when none of the true training data is known, our method is able to locate a small subset of public Web documents.
  arXiv  Detail & Related papers  (2025-06-18T15:26:43Z)
• All models are wrong, some are useful: Model Selection with Limited Labels [49.62984196182567]
  We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. We show that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model.
  arXiv  Detail & Related papers  (2024-10-17T14:45:56Z)
• Enabling Small Models for Zero-Shot Selection and Reuse through Model Label Learning [50.68074833512999]
  We introduce a novel paradigm, Model Label Learning (MLL), which bridges the gap between models and their functionalities.<n>Experiments on seven real-world datasets validate the effectiveness and efficiency of MLL.
  arXiv  Detail & Related papers  (2024-08-21T09:08:26Z)
• DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
  Standard practice is to filter for examples that match human notions of data quality. We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
  arXiv  Detail & Related papers  (2024-01-23T17:22:00Z)
• Large Language Model Routing with Benchmark Datasets [40.42044096089315]
  No single model typically achieves the best accuracy in all tasks and use cases. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this selection. We show that this problem can be reduced to a collection of binary classification tasks.
  arXiv  Detail & Related papers  (2023-09-27T17:08:40Z)
• Anchor Points: Benchmarking Models with Much Fewer Examples [88.02417913161356]
  In six popular language classification benchmarks, model confidence in the correct class on many pairs of points is strongly correlated across models. We propose Anchor Point Selection, a technique to select small subsets of datasets that capture model behavior across the entire dataset. Just several anchor points can be used to estimate model per-class predictions on all other points in a dataset with low mean absolute error.
  arXiv  Detail & Related papers  (2023-09-14T17:45:51Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [47.432215933099016]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.<n>This creates a barrier to fusing knowledge across individual models to yield a better single model.<n>We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Where to start? Analyzing the potential value of intermediate models [16.32982010228009]
  We perform a systematic analysis of the emphintertraining scheme over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed emphindependently for the target dataset. We leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings.
  arXiv  Detail & Related papers  (2022-10-31T19:24:02Z)
• A linearized framework and a new benchmark for model selection for fine-tuning [112.20527122513668]
  Fine-tuning from a collection of models pre-trained on different domains is emerging as a technique to improve test accuracy in the low-data regime. We introduce two new baselines for model selection -- Label-Gradient and Label-Feature Correlation. Our benchmark highlights accuracy gain with model zoo compared to fine-tuning Imagenet models.
  arXiv  Detail & Related papers  (2021-01-29T21:57:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.