Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean

• URL: http://arxiv.org/abs/2604.08595v1
• Date: Sat, 04 Apr 2026 09:04:38 GMT
• Title: Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean
• Abstract summary: This paper presents a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor.<n>Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing evaluation methods for LLM-based AI systems, such as LLM-as-a-Judge, verdict systems, and NLI, do not always align well with human assessment because they cannot adapt their strictness to the application domain. This paper presents Temperature-Controlled Verdict Aggregation (TCVA), a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor. Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI. Experimental evaluation on three benchmark datasets with human Likert-scale annotations (SummEval and USR) shows that TCVA achieves correlation with human judgments comparable to RAGAS on faithfulness (Spearman = 0.667 vs. 0.676) while consistently outperforming DeepEval. The method requires no additional LLM calls when adjusting the temperature parameter.

Related papers

• Reliable Self-Harm Risk Screening via Adaptive Multi-Agent LLM Systems [6.14481021961242]
  We present a statistical framework for multi-agent pipelines structured as directed acyclic graphs (DAGs)<n>We model each agent as a categorical decision and introduce tighter agent-level performance confidence bounds.<n>We evaluate our system on two labeled datasets in behavioral health.
  arXiv  Detail & Related papers  (2026-04-24T01:52:54Z)
• The Necessity of Setting Temperature in LLM-as-a-Judge [10.631912729138152]
  We investigate the relationship between temperature and judge performance through a series of controlled experiments.<n>We also adopt a causal inference framework within our empirical statistical analysis to rigorously examine the direct causal effect of temperature on judge behavior.
  arXiv  Detail & Related papers  (2026-03-30T11:31:29Z)
• Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL [30.357975264905978]
  We propose a hierarchical reinforcement learning framework that learns to control sampling temperature during generation.<n>At each decoding step, the model selects a temperature based on its hidden state and samples the next token from the resulting distribution.<n>Temperature and token policies are jointly optimized from downstream rewards using a coordinate ascent scheme.
  arXiv  Detail & Related papers  (2026-02-13T15:42:59Z)
• Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning [47.83947232413507]
  Temperature controls the trade-off between exploration and exploitation in large language models (LLMs)<n>High temperatures encourage diverse but noisy outputs, while low temperatures produce focused outputs but may cause premature convergence.<n>We propose Temperature Adaptive Meta Policy Optimization (TAMPO), a new framework that recasts temperature control as a learnable meta-policy.
  arXiv  Detail & Related papers  (2026-02-12T09:59:58Z)
• TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness [3.796841688047966]
  This work presents a systematic investigation of how text perturbations interact with temperature settings across multiple LLM runs.<n>We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings.<n>Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.
  arXiv  Detail & Related papers  (2025-12-01T01:46:36Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)
• Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator [19.673388630963807]
  This work systematically defines and validate the phenomenon of inflated performance in models evaluated on their self-generated benchmarks.<n>We propose Silencer, a framework that leverages the heterogeneity between multiple generators at both the sample and benchmark levels to neutralize bias and generate high-quality, self-bias-silenced benchmark.<n> Experimental results across various settings demonstrate that Silencer can suppress self-bias to near zero, significantly improve evaluation effectiveness of the generated benchmark.
  arXiv  Detail & Related papers  (2025-05-27T05:28:45Z)
• HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation [25.193026443079987]
  HypoEval is a Hypothesis-guided Evaluation framework for large language models (LLMs)<n>With only 30 human evaluations, HypoEval achieves state-of-the-art performance in alignment with both human rankings (Spearman correlation) and human scores (Pearson correlation)<n>We conduct systematic studies to assess the robustness of HypoEval, highlighting its effectiveness as a reliable and interpretable automated evaluation framework.
  arXiv  Detail & Related papers  (2025-04-09T18:00:01Z)
• Calibrating Language Models with Adaptive Temperature Scaling [58.056023173579625]
  We introduce Adaptive Temperature Scaling (ATS), a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction. ATS improves calibration by over 10-50% across three downstream natural language evaluation benchmarks compared to prior calibration methods.
  arXiv  Detail & Related papers  (2024-09-29T22:54:31Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization [51.41175648612714]
  We propose a new robust contrastive loss inspired by distributionally robust optimization (DRO) We show that our algorithm automatically learns a suitable $tau$ for each sample. Our method outperforms prior strong baselines on unimodal and bimodal datasets.
  arXiv  Detail & Related papers  (2023-05-19T19:25:56Z)
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking [59.691919635037216]
  Policy evaluation is a key process in Reinforcement Learning (RL) We devise an optimization method, called Kalman Optimization for Value Approximation (KOVA) KOVA minimizes a regularized objective function that concerns both parameter and noisy return uncertainties.
  arXiv  Detail & Related papers  (2020-02-17T13:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.