Multiple-criteria Heuristic Rating Estimation

• URL: http://arxiv.org/abs/2205.10428v1
• Date: Fri, 20 May 2022 20:12:04 GMT
• Title: Multiple-criteria Heuristic Rating Estimation
• Authors: Anna K\k{e}dzior and Konrad Ku{\l}akowski
• Abstract summary: Heuristic Rating Estimation (HRE) method proposed in 2014 tried to bring answer to this question. We analyze how HRE can be used as part of the Analytic Hierarchy Process hierarchical framework.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the most widespread multi-criteria decision-making methods is the Analytic Hierarchy Process (AHP). AHP successfully combines the pairwise comparisons method and the hierarchical approach. It allows the decision-maker to set priorities for all ranked alternatives. But what if, for some of them, their ranking value is known (e.g., it can be determined differently)? The Heuristic Rating Estimation (HRE) method proposed in 2014 tried to bring the answer to this question. However, the considerations were limited to a model that did not consider many criteria. In this work, we go a step further and analyze how HRE can be used as part of the AHP hierarchical framework. The theoretical considerations are accompanied by illustrative examples showing HRE as a multiple-criteria decision-making method.

Related papers

• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection. Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains. We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat [7.8905223445925055]
  Pairwise ranking has emerged as a new method for evaluating human preferences for large language models (LLM) We explore the effectiveness of ranking systems for head-to-head comparisons of LLMs. Our analysis uncovers key insights into the factors that affect ranking accuracy and efficiency.
  arXiv  Detail & Related papers  (2024-11-19T20:16:26Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification [120.37051160567277]
  This paper proposes a novel measure named Top-K Pairwise Ranking (TKPR) A series of analyses show that TKPR is compatible with existing ranking-based measures. On the other hand, we establish a sharp generalization bound for the proposed framework based on a novel technique named data-dependent contraction.
  arXiv  Detail & Related papers  (2024-07-09T09:36:37Z)
• Multi-Criteria Comparison as a Method of Advancing Knowledge-Guided Machine Learning [1.6574413179773761]
  This paper describes a generalizable model evaluation method that can be adapted to evaluate AI/ML models. The method evaluates a group of candidate models of varying type and structure across multiple scientific, theoretic, and practical criteria.
  arXiv  Detail & Related papers  (2024-03-18T14:50:48Z)
• HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition [92.17397504834825]
  HD-Eval is a framework that iteratively aligns large language models evaluators with human preference. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators.
  arXiv  Detail & Related papers  (2024-02-24T08:01:32Z)
• Risk Consistent Multi-Class Learning from Label Proportions [64.0125322353281]
  This study addresses a multiclass learning from label proportions (MCLLP) setting in which training instances are provided in bags. Most existing MCLLP methods impose bag-wise constraints on the prediction of instances or assign them pseudo-labels. A risk-consistent method is proposed for instance classification using the empirical risk minimization framework.
  arXiv  Detail & Related papers  (2022-03-24T03:49:04Z)
• On the Evaluation of Answer-Agnostic Paragraph-level Multi-Question Generation [57.630606799713526]
  We study the task of predicting a set of salient questions from a given paragraph without any prior knowledge of the precise answer. First, we propose a new method to evaluate a set of predicted questions against the set of references by using the Hungarian algorithm to assign predicted questions to references before scoring the assigned pairs. Second, we compare different strategies to utilize a pre-trained seq2seq model to generate and select a set of questions related to a given paragraph.
  arXiv  Detail & Related papers  (2022-03-09T00:55:54Z)
• R\&D evaluation methodology based on group-AHP with uncertainty [0.17689918341582753]
  We present an approach to evaluate Research & Development (R&D) performance based on the Analytic Hierarchy Process (AHP) method. We single out a set of indicators needed for R&D performance evaluation. The numerical values associated with all the indicators are then used to assign a score to a given R&D project.
  arXiv  Detail & Related papers  (2021-08-05T13:04:33Z)
• A study of the Multicriteria decision analysis based on the time-series features and a TOPSIS method proposal for a tensorial approach [1.3750624267664155]
  We propose a new approach to rank the alternatives based on the criteria time-series features (tendency, variance, etc.) In this novel approach, the data is structured in three dimensions, which require a more complex data structure, as the textittensors. Computational results reveal that it is possible to rank the alternatives from a new perspective by considering meaningful decision-making information.
  arXiv  Detail & Related papers  (2020-10-21T14:37:02Z)
• Selective Classification via One-Sided Prediction [54.05407231648068]
  One-sided prediction (OSP) based relaxation yields an SC scheme that attains near-optimal coverage in the practically relevant high target accuracy regime. We theoretically derive bounds generalization for SC and OSP, and empirically we show that our scheme strongly outperforms state of the art methods in coverage at small error levels.
  arXiv  Detail & Related papers  (2020-10-15T16:14:27Z)
• Application of independent component analysis and TOPSIS to deal with dependent criteria in multicriteria decision problems [8.637110868126546]
  We propose a novel approach whose aim is to estimate, from the observed data, a set of independent latent criteria. A central element of our approach is to formulate the decision problem as a blind source separation problem. We consider TOPSIS-based approaches to obtain the ranking of alternatives from the latent criteria.
  arXiv  Detail & Related papers  (2020-02-06T13:51:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.