Quality Indicators for Preference-based Evolutionary Multi-objective Optimization Using a Reference Point: A Review and Analysis

• URL: http://arxiv.org/abs/2301.12148v3
• Date: Tue, 26 Sep 2023 04:55:31 GMT
• Title: Quality Indicators for Preference-based Evolutionary Multi-objective Optimization Using a Reference Point: A Review and Analysis
• Authors: Ryoji Tanabe and Ke Li
• Abstract summary: Some quality indicators have been proposed for benchmarking preference-based evolutionary multi-objective optimization algorithms. This paper reviews existing regions of interest and quality indicators for preference-based evolutionary multi-objective optimization using the reference point. We show that the ranking of preference-based evolutionary multi-objective optimization algorithms depends on the choice of quality indicators.
• Score: 5.074835777266041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Some quality indicators have been proposed for benchmarking preference-based evolutionary multi-objective optimization algorithms using a reference point. Although a systematic review and analysis of the quality indicators are helpful for both benchmarking and practical decision-making, neither has been conducted. In this context, first, this paper reviews existing regions of interest and quality indicators for preference-based evolutionary multi-objective optimization using the reference point. We point out that each quality indicator was designed for a different region of interest. Then, this paper investigates the properties of the quality indicators. We demonstrate that an achievement scalarizing function value is not always consistent with the distance from a solution to the reference point in the objective space. We observe that the regions of interest can be significantly different depending on the position of the reference point and the shape of the Pareto front. We identify undesirable properties of some quality indicators. We also show that the ranking of preference-based evolutionary multi-objective optimization algorithms depends on the choice of quality indicators.

Related papers

• Illuminating the Diversity-Fitness Trade-Off in Black-Box Optimization [9.838618121102053]
  In real-world applications, users often favor structurally diverse design choices over one high-quality solution. This paper presents a fresh perspective on this challenge by considering the problem of identifying a fixed number of solutions with a pairwise distance above a specified threshold.
  arXiv  Detail & Related papers  (2024-08-29T09:55:55Z)
• Improving Point-based Crowd Counting and Localization Based on Auxiliary Point Guidance [59.71186244597394]
  We introduce an effective approach to stabilize the proposal-target matching in point-based methods. We propose Auxiliary Point Guidance (APG) to provide clear and effective guidance for proposal selection and optimization. We also develop Implicit Feature Interpolation (IFI) to enable adaptive feature extraction in diverse crowd scenarios.
  arXiv  Detail & Related papers  (2024-05-17T07:23:27Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• Towards Assessing Spread in Sets of Software Architecture Designs [2.2120851074630177]
  We propose a quality indicator for the spread that assesses the diversity of alternatives by taking into account architectural features. We demonstrate how our architectural quality indicator can be applied to a dataset from the literature.
  arXiv  Detail & Related papers  (2024-02-29T13:52:39Z)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning [65.51668094117802]
  We propose a human-centered interactive HPO approach tailored towards multi-objective machine learning (ML) Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator.
  arXiv  Detail & Related papers  (2023-09-07T09:22:05Z)
• Gap-closing Matters: Perceptual Quality Evaluation and Optimization of Low-Light Image Enhancement [55.8106019031768]
  There is a growing consensus in the research community that the optimization of low-light image enhancement approaches should be guided by the visual quality perceived by end users. We propose a gap-closing framework for assessing subjective and objective quality systematically. We validate the effectiveness of our proposed framework through both the accuracy of quality prediction and the perceptual quality of image enhancement.
  arXiv  Detail & Related papers  (2023-02-22T15:57:03Z)
• A Survey of Parameters Associated with the Quality of Benchmarks in NLP [24.6240575061124]
  Recent studies have shown that models triumph over several popular benchmarks just by overfitting on spurious biases, without truly learning the desired task. A potential solution to these issues -- a metric quantifying quality -- remains underexplored. We take the first step towards a metric by identifying certain language properties that can represent various possible interactions leading to biases in a benchmark.
  arXiv  Detail & Related papers  (2022-10-14T06:44:14Z)
• Blind Quality Assessment of 3D Dense Point Clouds with Structure Guided Resampling [71.68672977990403]
  We propose an objective point cloud quality index with Structure Guided Resampling (SGR) to automatically evaluate the perceptually visual quality of 3D dense point clouds. The proposed SGR is a general-purpose blind quality assessment method without the assistance of any reference information.
  arXiv  Detail & Related papers  (2022-08-31T02:42:55Z)
• Maximizing Welfare with Incentive-Aware Evaluation Mechanisms [18.304048425012503]
  We propose an evaluation problem where the inputs are controlled by strategic individuals who can modify their features at a cost. A learner can only partially observe the features, and aims to classify individuals with respect to a quality score.
  arXiv  Detail & Related papers  (2020-11-03T19:00:43Z)
• An Analysis of Quality Indicators Using Approximated Optimal Distributions in a Three-dimensional Objective Space [7.81768535871051]
  Quality indicators play a crucial role in benchmarking evolutionary multi-objective optimization algorithms. It is difficult to obtain the optimal distribution for each quality indicator, especially when its theoretical property is unknown. We analyze the nine quality indicators using their approximated optimal distributions on eight types of three-objective problems.
  arXiv  Detail & Related papers  (2020-09-27T08:30:43Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.