Measuring Diversity: Axioms and Challenges

• URL: http://arxiv.org/abs/2410.14556v1
• Date: Fri, 18 Oct 2024 15:59:54 GMT
• Title: Measuring Diversity: Axioms and Challenges
• Authors: Mikhail Mironov, Liudmila Prokhorenkova,
• Abstract summary: We make a systematic review of existing diversity measures and explore their undesirable behavior in some cases. We formulate three desirable properties (axioms) of a reliable diversity measure: monotonicity, uniqueness, and continuity. We construct two examples of measures that have all the desirable properties, thus proving that the list of axioms is not self-contradicting.
• Score: 18.497863598167257
• License:
• Abstract: The concept of diversity is widely used in various applications: from image or molecule generation to recommender systems. Thus, being able to properly measure diversity is important. This paper addresses the problem of quantifying diversity for a set of objects. First, we make a systematic review of existing diversity measures and explore their undesirable behavior in some cases. Based on this review, we formulate three desirable properties (axioms) of a reliable diversity measure: monotonicity, uniqueness, and continuity. We show that none of the existing measures has all three properties and thus these measures are not suitable for quantifying diversity. Then, we construct two examples of measures that have all the desirable properties, thus proving that the list of axioms is not self-contradicting. Unfortunately, the constructed examples are too computationally complex for practical use, thus we pose an open problem of constructing a diversity measure that has all the listed properties and can be computed in practice.

Related papers

• GRADE: Quantifying Sample Diversity in Text-to-Image Models [66.12068246962762]
  We propose GRADE: Granular Attribute Diversity Evaluation, an automatic method for quantifying sample diversity. We measure the overall diversity of 12 T2I models using 400 concept-attribute pairs, revealing that all models display limited variation. Our work proposes a modern, semantically-driven approach to measure sample diversity and highlights the stunning homogeneity in outputs by T2I models.
  arXiv  Detail & Related papers  (2024-10-29T23:10:28Z)
• Metric Space Magnitude for Evaluating the Diversity of Latent Representations [13.272500655475486]
  We develop a family of magnitude-based measures of the intrinsic diversity of latent representations. Our measures are provably stable under perturbations of the data, can be efficiently calculated, and enable a rigorous multi-scale characterisation and comparison of latent representations. We show their utility and superior performance across different domains and tasks, including (i) the automated estimation of diversity, (ii) the detection of mode collapse, and (iii) the evaluation of generative models for text, image, and graph data.
  arXiv  Detail & Related papers  (2023-11-27T18:19:07Z)
• Diversify Question Generation with Retrieval-Augmented Style Transfer [68.00794669873196]
  We propose RAST, a framework for Retrieval-Augmented Style Transfer. The objective is to utilize the style of diverse templates for question generation. We develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward.
  arXiv  Detail & Related papers  (2023-10-23T02:27:31Z)
• Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs [78.31625291513589]
  We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency and compositional consistency. We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:25:59Z)
• Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
  Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning. We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
  arXiv  Detail & Related papers  (2023-05-19T08:22:23Z)
• Achieving Diversity in Counterfactual Explanations: a Review and Discussion [3.6066164404432883]
  In the field of Explainable Artificial Intelligence (XAI), counterfactual examples explain to a user the predictions of a trained decision model. This paper proposes a review of the numerous, sometimes conflicting, definitions that have been proposed for this notion of diversity.
  arXiv  Detail & Related papers  (2023-05-10T02:09:19Z)
• System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning [8.280943341629161]
  We introduce System Neural Diversity (SND): a measure of behavioral heterogeneity in multi-agent systems. We show how SND allows us to measure latent resilience skills acquired by the agents, while other proxies, such as task performance (reward), fail. We demonstrate how this paradigm can be used to bootstrap the exploration phase, finding optimal policies faster.
  arXiv  Detail & Related papers  (2023-05-03T13:58:13Z)
• A Unified Theory of Diversity in Ensemble Learning [4.773356856466191]
  We present a theory of ensemble diversity, explaining the nature of diversity for a wide range of supervised learning scenarios. This challenge has been referred to as the holy grail of ensemble learning, an open research issue for over 30 years.
  arXiv  Detail & Related papers  (2023-01-10T13:51:07Z)
• Robust Allocations with Diversity Constraints [65.3799850959513]
  We show that the Nash Welfare rule that maximizes product of agent values is uniquely positioned to be robust when diversity constraints are introduced. We also show that the guarantees achieved by Nash Welfare are nearly optimal within a widely studied class of allocation rules.
  arXiv  Detail & Related papers  (2021-09-30T11:09:31Z)
• What can phylogenetic metrics tell us about useful diversity in evolutionary algorithms? [62.997667081978825]
  Phylogenetic diversity metrics are a class of metrics popularly used in biology. We find that, in most cases, phylogenetic metrics behave meaningfully differently from other diversity metrics.
  arXiv  Detail & Related papers  (2021-08-28T06:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.