Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

• URL: http://arxiv.org/abs/2605.17110v1
• Date: Sat, 16 May 2026 18:30:37 GMT
• Title: Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
• Abstract summary: We propose ECC, an algorithm that calibrates prior semantic embeddings to bridge the gap between surface-level semantics and latent capability requirements.<n> ECC characterizes each cluster through a capability profile parameterized by a Bradley-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands.
• Score: 12.71199389824453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability-aware LLM evaluation. Existing clustering methods, which primarily rely on semantic taxonomies or embeddings, often fail to capture such latent capability requirements due to a misalignment between surface-level semantics and actual model performance. We propose ECC, an algorithm that calibrates prior semantic embeddings using limited posterior model comparisons to bridge the gap between surface-level semantics and latent capability requirements. ECC characterizes each cluster through a capability profile parameterized by a Bradley-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands, jointly learning a flexible, capability-aware clustering structure that supports query-specific inference of LLM capabilities. Extensive quantitative and qualitative evaluations demonstrate that ECC significantly improves LLM capability ranking quality, outperforming human-labeled and embedding-based baselines by an average of 17.64 and 18.02 percentage points, respectively, and proves effective in downstream tasks such as query routing.

Related papers

• Unsupervised Transfer Clustering for Mitigating Cold Start in Active Prompt Learning [1.5539424546888112]
  Vision-Language Models (VLMs) are able to achieve impressive zero-shot classification performance by aligning visual and textual representations.<n>Each new task still demands handcrafted prompts.<n>Active Prompt Learning (APL) combines Active Learning (AL) and Prompt Learning (PL) into a single framework.
  arXiv  Detail & Related papers  (2026-09-04T18:14:44Z)
• Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents [81.79110139097297]
  General-purpose embedding models excel at recognizing semantic similarities but fail to capture the characteristics of texts specified by user instructions.<n>We reframe instruction-following clustering as a generative task and train large reasoning models as autonomous clustering agents.<n>Our reasoning-driven training pipeline enables LRMs to interpret high-level clustering instructions and then infer the corresponding latent groupings.
  arXiv  Detail & Related papers  (2026-03-06T10:15:54Z)
• ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering [79.69917150582633]
  Multi-modal large language models (MLLMs) can be leveraged to achieve user-driven clustering.<n>Our method first discovers that MLLMs' hidden states of text tokens are strongly related to the corresponding features.<n>We also employ a lightweight clustering head augmented with pseudo-label learning, significantly enhancing clustering accuracy.
  arXiv  Detail & Related papers  (2025-11-30T04:36:51Z)
• Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System [2.3322477552758234]
  We present an interpretable aggregation framework combining LLM scoring with the Analytic Hierarchy Process.<n>Our method generates criterion specific scores via LLM as judge, measures discriminative power using Jensennative Shannon distance, and derives statistically grounded weights through AHP pairwise comparison matrices.<n> Experiments on Amazon review quality assessment and depression related text scoring demonstrate that our approach achieves high explainability and operational efficiency while maintaining comparable predictive power, making it suitable for real time latency sensitive web services.
  arXiv  Detail & Related papers  (2025-11-12T06:17:02Z)
• In-Context Clustering with Large Language Models [50.25868718329313]
  ICC captures complex relationships among inputs through an attention mechanism.<n>We show that pretrained LLMs exhibit impressive zero-shot clustering capabilities on text-encoded numeric data.<n>Our work extends in-context learning to an unsupervised setting, showcasing the effectiveness and flexibility of LLMs for clustering.
  arXiv  Detail & Related papers  (2025-10-09T17:07:55Z)
• Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilities [4.8986761253231945]
  We propose a holistic and generalizable framework based on emphcascaded question disclosure<n>This approach collects model responses in a stagewise manner with each stage revealing partial information about the question.<n>We empirically verify this behavior on diverse reasoning and knowledge-heavy QA datasets.
  arXiv  Detail & Related papers  (2025-07-31T17:58:25Z)
• AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs [53.6200736559742]
  AGENT-CQ consists of two stages: a generation stage and an evaluation stage. CrowdLLM simulates human crowdsourcing judgments to assess generated questions and answers. Experiments on the ClariQ dataset demonstrate CrowdLLM's effectiveness in evaluating question and answer quality.
  arXiv  Detail & Related papers  (2024-10-25T17:06:27Z)
• Large Language Models Enable Few-Shot Clustering [88.06276828752553]
  We show that large language models can amplify an expert's guidance to enable query-efficient, few-shot semi-supervised text clustering. We find incorporating LLMs in the first two stages can routinely provide significant improvements in cluster quality.
  arXiv  Detail & Related papers  (2023-07-02T09:17:11Z)
• Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods [61.49061000562676]
  We introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN trained to predict labels obtained by clustering the representations with K-means. We find that CL better correlates with in-distribution model performance than other competing recent evaluation schemes.
  arXiv  Detail & Related papers  (2022-06-02T19:05:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.