Interpretable Clustering with the Distinguishability Criterion

• URL: http://arxiv.org/abs/2404.15967v2
• Date: Thu, 25 Apr 2024 17:13:45 GMT
• Title: Interpretable Clustering with the Distinguishability Criterion
• Authors: Ali Turfah, Xiaoquan Wen,
• Abstract summary: We present a global criterion called the Distinguishability criterion to quantify the separability of identified clusters and validate inferred cluster configurations. We propose a combined loss function-based computational framework that integrates the Distinguishability criterion with many commonly used clustering procedures. We present these new algorithms as well as the results from comprehensive data analysis based on simulation studies and real data applications.
• Score: 0.4419843514606336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cluster analysis is a popular unsupervised learning tool used in many disciplines to identify heterogeneous sub-populations within a sample. However, validating cluster analysis results and determining the number of clusters in a data set remains an outstanding problem. In this work, we present a global criterion called the Distinguishability criterion to quantify the separability of identified clusters and validate inferred cluster configurations. Our computational implementation of the Distinguishability criterion corresponds to the Bayes risk of a randomized classifier under the 0-1 loss. We propose a combined loss function-based computational framework that integrates the Distinguishability criterion with many commonly used clustering procedures, such as hierarchical clustering, k-means, and finite mixture models. We present these new algorithms as well as the results from comprehensive data analysis based on simulation studies and real data applications.

Related papers

• From A-to-Z Review of Clustering Validation Indices [4.08908337437878]
  We review and evaluate the performance of internal and external clustering validation indices on the most common clustering algorithms. We suggest a classification framework for examining the functionality of both internal and external clustering validation measures.
  arXiv  Detail & Related papers  (2024-07-18T13:52:02Z)
• A structured regression approach for evaluating model performance across intersectional subgroups [53.91682617836498]
  Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups. We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
  arXiv  Detail & Related papers  (2024-01-26T14:21:45Z)
• Robust and Automatic Data Clustering: Dirichlet Process meets Median-of-Means [18.3248037914529]
  We present an efficient and automatic clustering technique by integrating the principles of model-based and centroid-based methodologies. Statistical guarantees on the upper bound of clustering error suggest the advantages of our proposed method over existing state-of-the-art clustering algorithms.
  arXiv  Detail & Related papers  (2023-11-26T19:01:15Z)
• A testing-based approach to assess the clusterability of categorical data [6.7937877930001775]
  TestCat is a testing-based approach to assess the clusterability of categorical data in terms of an analytical $p$-value. We apply our method to a set of benchmark categorical data sets, showing that TestCat outperforms those solutions for numeric data.
  arXiv  Detail & Related papers  (2023-07-14T13:50:00Z)
• Instance-Optimal Cluster Recovery in the Labeled Stochastic Block Model [79.46465138631592]
  We devise an efficient algorithm that recovers clusters using the observed labels. We present Instance-Adaptive Clustering (IAC), the first algorithm whose performance matches these lower bounds both in expectation and with high probability.
  arXiv  Detail & Related papers  (2023-06-18T08:46:06Z)
• Significance-Based Categorical Data Clustering [7.421725101465365]
  We use the likelihood ratio test to derive a test statistic that can serve as a significance-based objective function in categorical data clustering. A new clustering algorithm is proposed in which the significance-based objective function is optimized via a Monte Carlo search procedure.
  arXiv  Detail & Related papers  (2022-11-08T02:06:31Z)
• Detection and Evaluation of Clusters within Sequential Data [58.720142291102135]
  Clustering algorithms for Block Markov Chains possess theoretical optimality guarantees. In particular, our sequential data is derived from human DNA, written text, animal movement data and financial markets. It is found that the Block Markov Chain model assumption can indeed produce meaningful insights in exploratory data analyses.
  arXiv  Detail & Related papers  (2022-10-04T15:22:39Z)
• A One-shot Framework for Distributed Clustered Learning in Heterogeneous Environments [54.172993875654015]
  The paper proposes a family of communication efficient methods for distributed learning in heterogeneous environments. One-shot approach, based on local computations at the users and a clustering based aggregation step at the server is shown to provide strong learning guarantees. For strongly convex problems it is shown that, as long as the number of data points per user is above a threshold, the proposed approach achieves order-optimal mean-squared error rates in terms of the sample size.
  arXiv  Detail & Related papers  (2022-09-22T09:04:10Z)
• Differentially-Private Clustering of Easy Instances [67.04951703461657]
  In differentially private clustering, the goal is to identify $k$ cluster centers without disclosing information on individual data points. We provide implementable differentially private clustering algorithms that provide utility when the data is "easy" We propose a framework that allows us to apply non-private clustering algorithms to the easy instances and privately combine the results.
  arXiv  Detail & Related papers  (2021-12-29T08:13:56Z)
• Scalable Hierarchical Agglomerative Clustering [65.66407726145619]
  Existing scalable hierarchical clustering methods sacrifice quality for speed. We present a scalable, agglomerative method for hierarchical clustering that does not sacrifice quality and scales to billions of data points.
  arXiv  Detail & Related papers  (2020-10-22T15:58:35Z)
• reval: a Python package to determine best clustering solutions with stability-based relative clustering validation [1.8129328638036126]
  reval is a Python package that leverages stability-based relative clustering validation methods to determine best clustering solutions. This work aims at developing a stability-based method that selects the best clustering solution as the one that replicates, via supervised learning, on unseen subsets of data.
  arXiv  Detail & Related papers  (2020-08-27T10:36:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.