Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System

• URL: http://arxiv.org/abs/2601.05267v1
• Date: Wed, 12 Nov 2025 06:17:02 GMT
• Title: Transforming User Defined Criteria into Explainable Indicators with an Integrated LLM AHP System
• Abstract summary: We present an interpretable aggregation framework combining LLM scoring with the Analytic Hierarchy Process.<n>Our method generates criterion specific scores via LLM as judge, measures discriminative power using Jensennative Shannon distance, and derives statistically grounded weights through AHP pairwise comparison matrices.<n> Experiments on Amazon review quality assessment and depression related text scoring demonstrate that our approach achieves high explainability and operational efficiency while maintaining comparable predictive power, making it suitable for real time latency sensitive web services.
• Score: 2.3322477552758234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating complex texts across domains requires converting user defined criteria into quantitative, explainable indicators, which is a persistent challenge in search and recommendation systems. Single prompt LLM evaluations suffer from complexity and latency issues, while criterion specific decomposition approaches rely on naive averaging or opaque black-box aggregation methods. We present an interpretable aggregation framework combining LLM scoring with the Analytic Hierarchy Process. Our method generates criterion specific scores via LLM as judge, measures discriminative power using Jensen Shannon distance, and derives statistically grounded weights through AHP pairwise comparison matrices. Experiments on Amazon review quality assessment and depression related text scoring demonstrate that our approach achieves high explainability and operational efficiency while maintaining comparable predictive power, making it suitable for real time latency sensitive web services.

Related papers

• (Towards) Scalable Reliable Automated Evaluation with Large Language Models [0.0]
  This work introduces a novel evaluation framework designed to approximate expert-level assessments of Large Language Models (LLMs) outputs.<n>An Elo rating system is used to generate stable and interpretable rankings.<n>Preliminary results show that automatically derived rankings correlate well with expert judgments.
  arXiv  Detail & Related papers  (2026-07-30T14:31:07Z)
• Capturing LLM Capabilities via Evidence-Calibrated Query Clustering [12.71199389824453]
  We propose ECC, an algorithm that calibrates prior semantic embeddings to bridge the gap between surface-level semantics and latent capability requirements.<n> ECC characterizes each cluster through a capability profile parameterized by a Bradley-Terry model and uses trainable mixture weights to accommodate queries with mixed capability demands.
  arXiv  Detail & Related papers  (2026-05-16T18:30:37Z)
• An Interpretable and Scalable Framework for Evaluating Large Language Models [13.322241501687715]
  Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy.<n>Items Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics.<n>We propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle.
  arXiv  Detail & Related papers  (2026-05-07T23:52:12Z)
• Programmatic Context Augmentation for LLM-based Symbolic Regression [65.01826333382738]
  Large language model (LLM)-based evolutionary search methods have been introduced into Symbolic regression.<n>We propose a novel LLM-based evolutionary search framework that incorporates programmatic context augmentation.<n>By enabling code-based interactions with the dataset, our method can actively perform data analysis and extract informative signals.
  arXiv  Detail & Related papers  (2026-05-04T19:34:03Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• Nonparametric LLM Evaluation from Preference Data [86.96268870461472]
  We propose a nonparametric statistical framework, DMLEval, for comparing and ranking large language models (LLMs) from preference data.<n>Our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs.
  arXiv  Detail & Related papers  (2026-01-29T15:00:07Z)
• LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest [3.306725465028306]
  We present our approach at Pinterest Search to automate relevance evaluation for online experiments using fine-tuned LLMs.<n>We rigorously validate the alignment between LLM-generated judgments and human annotations.<n>This approach leads to higher-quality relevance metrics and significantly reduces the Minimum Detectable Effect (MDE) in online experiment measurements.
  arXiv  Detail & Related papers  (2025-09-03T23:07:49Z)
• AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume [2.2153783542347805]
  This paper investigates challenges in automatic text summarization evaluation.<n>Based on experiments conducted across six representative metrics, we highlight significant discrepancies between reported performances in the literature and those observed in our experimental setting.<n>We introduce a unified, open-source framework, applied to the SummEval dataset and designed to support fair and transparent comparison of evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-29T08:05:00Z)
• LLM-Symbolic Integration for Robust Temporal Tabular Reasoning [69.27153114778748]
  We introduce TempTabQA-C, a synthetic dataset designed for systematic and controlled evaluations.<n>This structured approach allows Large Language Models (LLMs) to generate and executesql queries, enhancing generalization and mitigating biases.
  arXiv  Detail & Related papers  (2025-06-06T05:14:04Z)
• Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMs [1.6332728502735252]
  Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly deployed in industry applications.<n>Their reliability remains hampered by challenges in detecting hallucinations.<n>This paper addresses the bottleneck of data annotation by investigating the feasibility of reducing training data requirements.
  arXiv  Detail & Related papers  (2025-05-29T09:50:56Z)
• An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning [52.29223403698673]
  This paper examines the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP)<n>We apply CLM to generate sets of ASP programs from an LLM, providing statistical guarantees on the correctness of the outputs.<n> Experimental results show that CLM significantly outperforms baseline models that use standard sampling methods.
  arXiv  Detail & Related papers  (2025-03-07T14:10:10Z)
• Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis [10.133537818749291]
  Large language models (LLMs) have demonstrated significant utilities in real-world applications.<n> Benchmark evaluations are crucial for assessing the capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-13T03:43:33Z)
• SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts [0.6291443816903801]
  This paper introduces a novel framework designed to autonomously evaluate the robustness of large language models (LLMs)<n>Our method generates descriptive sentences from domain-constrained knowledge graph triplets to formulate adversarial prompts.<n>This self-evaluation mechanism allows the LLM to evaluate its robustness without the need for external benchmarks.
  arXiv  Detail & Related papers  (2024-12-01T10:58:53Z)
• LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning [56.273799410256075]
  The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path. The framework has been tested on general and advanced benchmarks, showing superior performance in terms of search efficiency and problem-solving capability.
  arXiv  Detail & Related papers  (2024-10-03T18:12:29Z)
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures [57.886592207948844]
  We propose MixEval, a new paradigm for establishing efficient, gold-standard evaluation by strategically mixing off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks.
  arXiv  Detail & Related papers  (2024-06-03T05:47:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.