CEREAL: Few-Sample Clustering Evaluation
- URL: http://arxiv.org/abs/2210.00064v1
- Date: Fri, 30 Sep 2022 19:52:41 GMT
- Title: CEREAL: Few-Sample Clustering Evaluation
- Authors: Nihal V. Nayak, Ethan R. Elenberg, Clemens Rosenbaum
- Abstract summary: We focus on the underexplored problem of estimating clustering quality with limited labels.
We introduce CEREAL, a comprehensive framework for few-sample clustering evaluation.
Our results show that CEREAL reduces the area under the absolute error curve by up to 57% compared to the best sampling baseline.
- Score: 4.569028973407756
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Evaluating clustering quality with reliable evaluation metrics like
normalized mutual information (NMI) requires labeled data that can be expensive
to annotate. We focus on the underexplored problem of estimating clustering
quality with limited labels. We adapt existing approaches from the few-sample
model evaluation literature to actively sub-sample, with a learned surrogate
model, the most informative data points for annotation to estimate the
evaluation metric. However, we find that their estimation can be biased and
only relies on the labeled data. To that end, we introduce CEREAL, a
comprehensive framework for few-sample clustering evaluation that extends
active sampling approaches in three key ways. First, we propose novel NMI-based
acquisition functions that account for the distinctive properties of clustering
and uncertainties from a learned surrogate model. Next, we use ideas from
semi-supervised learning and train the surrogate model with both the labeled
and unlabeled data. Finally, we pseudo-label the unlabeled data with the
surrogate model. We run experiments to estimate NMI in an active sampling
pipeline on three datasets across vision and language. Our results show that
CEREAL reduces the area under the absolute error curve by up to 57% compared to
the best sampling baseline. We perform an extensive ablation study to show that
our framework is agnostic to the choice of clustering algorithm and evaluation
metric. We also extend CEREAL from clusterwise annotations to pairwise
annotations. Overall, CEREAL can efficiently evaluate clustering with limited
human annotations.
Related papers
- Self Supervised Correlation-based Permutations for Multi-View Clustering [7.972599673048582]
We propose an end-to-end deep learning-based MVC framework for general data.
Our approach involves learning meaningful fused data representations with a novel permutation-based canonical correlation objective.
We demonstrate the effectiveness of our model using ten MVC benchmark datasets.
arXiv Detail & Related papers (2024-02-26T08:08:30Z) - A structured regression approach for evaluating model performance across intersectional subgroups [53.91682617836498]
Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups.
We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
arXiv Detail & Related papers (2024-01-26T14:21:45Z) - Generalized Category Discovery with Clustering Assignment Consistency [56.92546133591019]
Generalized category discovery (GCD) is a recently proposed open-world task.
We propose a co-training-based framework that encourages clustering consistency.
Our method achieves state-of-the-art performance on three generic benchmarks and three fine-grained visual recognition datasets.
arXiv Detail & Related papers (2023-10-30T00:32:47Z) - Bring Your Own Data! Self-Supervised Evaluation for Large Language
Models [52.15056231665816]
We propose a framework for self-supervised evaluation of Large Language Models (LLMs)
We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence.
We find strong correlations between self-supervised and human-supervised evaluations.
arXiv Detail & Related papers (2023-06-23T17:59:09Z) - C3: Cross-instance guided Contrastive Clustering [8.953252452851862]
Clustering is the task of gathering similar data samples into clusters without using any predefined labels.
We propose a novel contrastive clustering method, Cross-instance guided Contrastive Clustering (C3)
Our proposed method can outperform state-of-the-art algorithms on benchmark computer vision datasets.
arXiv Detail & Related papers (2022-11-14T06:28:07Z) - Unified Multi-View Orthonormal Non-Negative Graph Based Clustering
Framework [74.25493157757943]
We formulate a novel clustering model, which exploits the non-negative feature property and incorporates the multi-view information into a unified joint learning framework.
We also explore, for the first time, the multi-model non-negative graph-based approach to clustering data based on deep features.
arXiv Detail & Related papers (2022-11-03T08:18:27Z) - Rethinking Clustering-Based Pseudo-Labeling for Unsupervised
Meta-Learning [146.11600461034746]
Method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling.
This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data.
We prove that the core reason for this is lack of a clustering-friendly property in the embedding space.
arXiv Detail & Related papers (2022-09-27T19:04:36Z) - Multi-Objective Few-shot Learning for Fair Classification [23.05869193599414]
We propose a framework for mitigating the disparities of the predicted classes with respect to secondary attributes within the data.
Our proposed method involves learning a multi-objective function that in addition to learning the primary objective of predicting the primary class labels from the data, also employs a clustering-based to minimize the disparities of the class label distribution with respect to the cluster memberships.
arXiv Detail & Related papers (2021-10-05T11:28:58Z) - Semi-Automatic Data Annotation guided by Feature Space Projection [117.9296191012968]
We present a semi-automatic data annotation approach based on suitable feature space projection and semi-supervised label estimation.
We validate our method on the popular MNIST dataset and on images of human intestinal parasites with and without fecal impurities.
Our results demonstrate the added-value of visual analytics tools that combine complementary abilities of humans and machines for more effective machine learning.
arXiv Detail & Related papers (2020-07-27T17:03:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.