AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty

• URL: http://arxiv.org/abs/2608.16690v1
• Date: Mon, 17 Aug 2026 15:10:33 GMT
• Title: AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
• Abstract summary: Multimodal large language models (MLLMs) are widely used for automated annotation, yet their per-class accuracy varies widely.<n>This paper systematically evaluates its full-frame formulation, termed AnchorScore, that assesses the classes MLLMs are least likely to annotate reliably.
• Score: 7.105430294335961
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) are widely used for automated annotation, yet their per-class accuracy varies widely (e.g., 12%-98% across the 13 classes of three classroom sub-datasets) and is expensive to measure: evaluating one 27B MLLM on 5,416 validation images takes roughly 14 hours, whereas a frozen-CLIP pass over the same images completes in about 3 minutes. A low-cost signal for ranking classes by expected MLLM annotation difficulty a priori remains underexplored. Building on the AnchorProxy construct (per-class zero-shot CLIP accuracy) introduced in the companion study, this paper systematically evaluates its full-frame formulation, termed AnchorScore here, as an a priori diagnostic that flags the classes MLLMs are least likely to annotate reliably. On classroom behavior data (SCB5, 13 classes, 6 MLLMs), AnchorScore correlates with per-class MLLM accuracy (Spearman rho = 0.769, p = 0.002, n = 13). None of the alternative difficulty predictors (DINOv2, ResNet-50, SigLIP, or MLLM self-verbalized uncertainty) showed a significant class-level correlation at n = 13. A cross-model consensus control suggests AnchorScore primarily captures a shared class-difficulty factor rather than a CLIP-specific signal. An independent replication on Stanford40 Actions yields a nearly identical effect (rho = 0.817, p < 0.001); the association is strongest on activity-recognition data and attenuates on medical and satellite imagery. Three practical applications follow: a deployable hybrid CLIP/MLLM routing strategy (predicted-class routing: up to +23 pp over CLIP-only at roughly 44% MLLM cost savings), prompt disambiguation on hard classes (exploratory), and review-priority prediction for human verification. AnchorScore does not estimate exact MLLM accuracy; it provides a low-cost ranking signal that directs expensive MLLM evaluation to the classes where it is most informative.

Related papers

• Assessing Language Models for Salient Class Identification [15.706102968915744]
  We study whether language models (LMs) can identify salient classes directly from commits without feature engineering, graph construction, or training.<n>We evaluate two large language models (LLMs), GPT-5.4 and DeepSeek-V3.2, one small language model (SLM), Qwen3.5-9B, and three prompting strategies: zero-shot, few-shot, and chain-of-thought.
  arXiv  Detail & Related papers  (2026-06-19T17:33:56Z)
• Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score [14.579552536669217]
  Unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation Code.<n>We present the first study of reliability in multimodal unlearning.<n>We introduce the Unified Quality Score (UQS), a metric with weights derived from each metric's Spearman correlation with the metric distance.
  arXiv  Detail & Related papers  (2026-05-04T04:13:00Z)
• A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction [0.0]
  This study evaluates the reliability of information extraction approaches from KRS documents using three strategies.<n>Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation.
  arXiv  Detail & Related papers  (2026-03-09T02:09:02Z)
• Evaluating Large Language Models for Gait Classification Using Text-Encoded Kinematic Waveforms [1.1635287453977645]
  Large Language Models (LLMs) may offer explanatory capabilities and confidence-aware outputs when applied to structured kinematic data.<n>This study evaluated whether general-purpose LLMs can classify continuous gait kinematics when represented as textual numeric sequences.
  arXiv  Detail & Related papers  (2026-03-04T10:23:35Z)
• Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting [59.37613121962146]
  We propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting.<n> WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs.
  arXiv  Detail & Related papers  (2026-02-13T09:58:35Z)
• MACA: A Framework for Distilling Trustworthy LLMs into Efficient Retrievers [7.123910165277138]
  We present Metadata-Aware Cross-Model Alignment (MACA), which distills a calibrated metadata aware LLM re-ranker into a compact student retriever.<n>A metadata-aware prompt verifies the teacher's trustworthiness by checking consistency under permutations and robustness to paraphrases, then supplies listwise scores, hard negatives, and calibrated relevance margins.
  arXiv  Detail & Related papers  (2026-01-01T23:31:02Z)
• Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission [87.68447072141402]
  Hybrid Language Models (HLMs) combine the low-latency efficiency of Small Language Models (SLMs) on edge devices with the high accuracy of Large Language Models (LLMs) on centralized servers.<n>We propose FedHLM, a communication-efficient HLM framework that integrates uncertainty-aware inference with Federated Learning (FL)
  arXiv  Detail & Related papers  (2025-06-30T02:56:11Z)
• PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations [13.99995125334968]
  We introduce a technique that evaluates an LLM's confidence for classifying a textual input by leveraging Metamorphic Relations (MRs)<n>MRs generate semantically equivalent yet textually divergent versions of the input.<n>We compute a perceived confidence score (PCS) based on the frequency of the predicted labels.
  arXiv  Detail & Related papers  (2025-02-11T02:25:44Z)
• Efficiently Disentangling CLIP for Multi-Object Perception [62.523137132812764]
  Vision-language models like CLIP excel at recognizing the single, prominent object in a scene, but struggle in complex scenes containing multiple objects.<n>We propose DCLIP, an efficient framework that learns an optimal level of mutual information while adding only minimal learnable parameters to a frozen VLM.
  arXiv  Detail & Related papers  (2025-02-05T08:20:31Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• Estimating Large Language Model Capabilities without Labeled Test Data [51.428562302037534]
  Large Language Models (LLMs) have the impressive ability to perform in-context learning (ICL) from only a few examples. We propose the task of ICL accuracy estimation, in which we predict the accuracy of an LLM when doing in-context learning on a new task.
  arXiv  Detail & Related papers  (2023-05-24T06:55:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.