Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation

• URL: http://arxiv.org/abs/2605.08522v2
• Date: Thu, 14 May 2026 09:18:49 GMT
• Title: Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation
• Abstract summary: This paper proposes a generalized Multi-Trait Multi-Method (MTMM) framework for Large Language Models (LLMs) evaluation.<n>We formalize and unify nine evaluation metrics, including Paraphrase Instability, Drift Score, Overton Width, and Pluralism Score, interpreting them as geometric measurements within a shared latent coordinate space.<n>By systematically separating task-irrelevant perturbations from true capability spans, the framework provides a theoretically grounded and domain-agnostic taxonomy for robust and empirically stable benchmark design.
• Score: 2.464003792743989
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The evaluation of Large Language Models (LLMs) faces a critical challenge in construct validity, where fragmented benchmarks and ad hoc metrics frequently conflate method variance, such as prompt sensitivity, with true latent capabilities. Concurrently, emerging research suggests that LLM capabilities and outputs can be modeled as continuous geometric manifolds. In this Systematization of Knowledge (SoK), we bridge these paradigms by proposing a generalized Multi-Trait Multi-Method (MTMM) framework for LLM evaluation. We formalize and unify nine evaluation metrics, including Paraphrase Instability, Drift Score, Overton Width, and Pluralism Score, interpreting them not as isolated scalar values but as geometric measurements within a shared latent coordinate space. This spatial unification factorizes model behavior into three orthogonal latent dimensions: (1) Instability and Sensitivity, (2) Position and Alignment, and (3) Coverage and Expressiveness. By systematically separating task-irrelevant perturbations from true capability spans, the framework provides a theoretically grounded and domain-agnostic taxonomy for robust and empirically stable benchmark design.

Related papers

• A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments [2.2404539130909376]
  This study proposes a dual-dimensional framework for Automated Item Similarity Analysis powered by Large Language Models (LLMs)<n>LLMs operationalize similarity through Structured Decomposition and Semantic Relatedness.<n>Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence.
  arXiv  Detail & Related papers  (2026-08-25T17:07:16Z)
• Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases [0.0]
  We introduce an analytically exact framework for the controlled behavioral evaluation of Large Language Models (LLMs)<n>We replace unstructured prompting with fully crossed factorial experiments to systematically isolate causal main and interaction effects.<n>We derive a multivariate ordinal consensus metric and a distributional ANOVA to process these PMFs analytically.<n>We validate our framework with a case study on consumer ethnocentrism across five LLMs, demonstrating how our approach isolates systemic country-of-origin biases that aggregate benchmarks otherwise obscure.
  arXiv  Detail & Related papers  (2026-08-11T05:20:06Z)
• The Art of Interrogation: Consistency Amplifies Factuality in Spatial Reasoning [65.67266333751569]
  Current Large Reasoning Models (LRMs) exhibit remarkable general capabilities but significantly underperform in spatial reasoning tasks.<n>We propose a self-supervised reinforcement learning framework that targets the internal reasoning process without requiring ground-truth annotations.<n>We show that this label-free consistency training approaches the accuracy of models trained with ground-truth supervision and achieves similar generalization across diverse tasks and data domains.
  arXiv  Detail & Related papers  (2026-06-10T10:50:06Z)
• ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation [8.82034003345674]
  Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, but often deviate in implicit behavioral decisions.<n>Existing benchmarks struggle to measure this asymmetry due to limited construct validity, multi-dimensional entanglement, and distributional biases.<n>We propose ActTraitBench, a human-grounded evaluation framework for measuring personality consistency in LLMs.
  arXiv  Detail & Related papers  (2026-05-28T11:40:35Z)
• An Interpretable and Scalable Framework for Evaluating Large Language Models [13.322241501687715]
  Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy.<n>Items Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics.<n>We propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle.
  arXiv  Detail & Related papers  (2026-05-07T23:52:12Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• ES-Merging: Biological MLLM Merging via Embedding Space Signals [52.84455878597969]
  We propose a representation-aware merging framework that estimates merging coefficients from embedding space signals.<n>Our method outperforms existing merging methods and even surpasses task-specific fine-tuned models.
  arXiv  Detail & Related papers  (2026-03-15T14:38:32Z)
• Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Models [28.281361951823765]
  We introduce Linear-LLM-SCM, a plug-and-play benchmarking framework for evaluating large language models (LLMs)<n>We show challenges in such benchmarking tasks, namely, strongity in the results in some of the models and susceptibility to DAG misspecification via spurious edges in the continuous domains.<n>We also open-source the benchmarking framework so that researchers can utilize their DAGs and any off-the-shelf LLMs plug-and-play for evaluation in their domains effortlessly.
  arXiv  Detail & Related papers  (2026-02-10T20:49:01Z)
• MS-ISSM: Objective Quality Assessment of Point Clouds Using Multi-scale Implicit Structural Similarity [65.85858856481131]
  unstructured and irregular nature of point clouds poses a significant challenge for objective quality assessment (PCQA)<n>We propose the Multi-scale Implicit Structural Similarity Measurement (MS-ISSM)
  arXiv  Detail & Related papers  (2026-01-03T14:58:52Z)
• LTD-Bench: Evaluating Large Language Models by Letting Them Draw [57.237152905238084]
  LTD-Bench is a breakthrough benchmark for large language models (LLMs)<n>It transforms LLM evaluation from abstract scores to directly observable visual outputs by requiring models to generate drawings through dot matrices or executable code.<n> LTD-Bench's visual outputs enable powerful diagnostic analysis, offering a potential approach to investigate model similarity.
  arXiv  Detail & Related papers  (2025-11-04T08:11:23Z)
• STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples [3.41981716024098]
  evaluating large language models (LLMs) has become increasingly challenging as model capabilities advance rapidly.<n>We propose the textbfStructured textbfTransition textbfEvaluation textbfMethod (STEM) as a lightweight and interpretable evaluation framework.
  arXiv  Detail & Related papers  (2025-08-16T16:36:43Z)
• Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs [56.76586846269894]
  Multimodal Large Language Models (MLLMs) have achieved success across various domains.<n>Despite its importance, the study of knowledge sharing among domain-specific MLLMs remains largely underexplored.<n>We propose a unified parameter integration framework that enables modular composition of expert capabilities.
  arXiv  Detail & Related papers  (2025-06-30T15:07:41Z)
• Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling [17.092510377905814]
  evaluating multimodal large language models (MLLMs) remains a fundamental challenge due to a lack of structured, interpretable, and theoretically grounded benchmark designs.<n>We propose a novel framework for aligning MLLM benchmark based on Structural Equation Modeling (SEM) to analyze and quantify the internal validity, dimensional separability, and contribution of benchmark components.<n> Experimental results demonstrate that the proposed benchmark exhibits stronger interpretability, reduced indicator redundancy, and clearer cognitive consistency compared to existing approaches.
  arXiv  Detail & Related papers  (2025-06-13T08:04:56Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.