Balancing Complexity and Informativeness in LLM-Based Clustering: Finding the Goldilocks Zone

• URL: http://arxiv.org/abs/2504.04314v1
• Date: Sun, 06 Apr 2025 01:16:22 GMT
• Title: Balancing Complexity and Informativeness in LLM-Based Clustering: Finding the Goldilocks Zone
• Authors: Justin Miller, Tristram Alexander,
• Abstract summary: This paper investigates the optimal number of clusters by quantifying the trade-off between informativeness and cognitive simplicity.<n>We use large language models (LLMs) to generate cluster names and evaluate their effectiveness.<n>We identify an optimal range of 16-22 clusters, paralleling linguistic efficiency in lexical categorization.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The challenge of clustering short text data lies in balancing informativeness with interpretability. Traditional evaluation metrics often overlook this trade-off. Inspired by linguistic principles of communicative efficiency, this paper investigates the optimal number of clusters by quantifying the trade-off between informativeness and cognitive simplicity. We use large language models (LLMs) to generate cluster names and evaluate their effectiveness through semantic density, information theory, and clustering accuracy. Our results show that Gaussian Mixture Model (GMM) clustering on embeddings generated by a LLM, increases semantic density compared to random assignment, effectively grouping similar bios. However, as clusters increase, interpretability declines, as measured by a generative LLM's ability to correctly assign bios based on cluster names. A logistic regression analysis confirms that classification accuracy depends on the semantic similarity between bios and their assigned cluster names, as well as their distinction from alternatives. These findings reveal a "Goldilocks zone" where clusters remain distinct yet interpretable. We identify an optimal range of 16-22 clusters, paralleling linguistic efficiency in lexical categorization. These insights inform both theoretical models and practical applications, guiding future research toward optimising cluster interpretability and usefulness.

Related papers

• k-LLMmeans: Summaries as Centroids for Interpretable and Scalable LLM-Based Text Clustering [0.0]
  We introduce k-LLMmeans, a novel modification of the k-means clustering algorithm that utilizes LLMs to generate textual summaries as cluster centroids.<n>This modification preserves the properties of k-means while offering greater interpretability.<n>We present a case study showcasing the interpretability of evolving cluster centroids in sequential text streams.
  arXiv  Detail & Related papers  (2025-02-12T19:50:22Z)
• Interaction-Aware Gaussian Weighting for Clustered Federated Learning [58.92159838586751]
  Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy.<n>We propose a novel clustered FL method, FedGWC (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution.<n>Our experiments on benchmark datasets show that FedGWC outperforms existing FL algorithms in cluster quality and classification accuracy.
  arXiv  Detail & Related papers  (2025-02-05T16:33:36Z)
• Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues [18.744211667479995]
  This paper investigates the effectiveness of fine-tuned.<n>LLMs in semantic coherence evaluation and intent cluster naming.<n>It also proposes an.<n>LLM-ITL clustering algorithm that facilitates the iterative discovery of.<n>coherent intent clusters.
  arXiv  Detail & Related papers  (2024-12-12T08:19:01Z)
• Categorical Data Clustering via Value Order Estimated Distance Metric Learning [31.851890008893847]
  This paper introduces a new finding that the order relation among attribute values is the decisive factor in clustering accuracy.<n>We propose a new learning paradigm that allows joint learning of clusters and the orders.<n>The algorithm achieves superior clustering accuracy with a convergence guarantee.
  arXiv  Detail & Related papers  (2024-11-19T08:23:25Z)
• Deep Clustering via Distribution Learning [7.437581715698929]
  We provide a theoretical analysis to guide the optimization of clustering via distribution learning. We introduce a clustering-oriented distribution learning method called Monte-Carlo Marginalization for Clustering. The proposed Deep Clustering via Distribution Learning (DCDL) achieves promising results compared to state-of-the-art methods on popular datasets.
  arXiv  Detail & Related papers  (2024-08-06T19:01:47Z)
• Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens [57.37893387775829]
  We introduce a fast and balanced clustering method, named textbfSemantic textbfEquitable textbfClustering (SEC) SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner. We propose a versatile vision backbone, SECViT, to serve as a vision language connector.
  arXiv  Detail & Related papers  (2024-05-22T04:49:00Z)
• CLAMS: A Cluster Ambiguity Measure for Estimating Perceptual Variability in Visual Clustering [23.625877882403227]
  We study perceptual variability in conducting visual clustering, which we call Cluster Ambiguity. We introduce CLAMS, a data-driven visual quality measure for automatically predicting cluster ambiguity in monochrome scatterplots.
  arXiv  Detail & Related papers  (2023-08-01T04:46:35Z)
• Large Language Models Enable Few-Shot Clustering [88.06276828752553]
  We show that large language models can amplify an expert's guidance to enable query-efficient, few-shot semi-supervised text clustering. We find incorporating LLMs in the first two stages can routinely provide significant improvements in cluster quality.
  arXiv  Detail & Related papers  (2023-07-02T09:17:11Z)
• Interpretable Deep Clustering for Tabular Data [7.972599673048582]
  Clustering is a fundamental learning task widely used in data analysis. We propose a new deep-learning framework that predicts interpretable cluster assignments at the instance and cluster levels. We show that the proposed method can reliably predict cluster assignments in biological, text, image, and physics datasets.
  arXiv  Detail & Related papers  (2023-06-07T21:08:09Z)
• Self-supervised Contrastive Attributed Graph Clustering [110.52694943592974]
  We propose a novel attributed graph clustering network, namely Self-supervised Contrastive Attributed Graph Clustering (SCAGC) In SCAGC, by leveraging inaccurate clustering labels, a self-supervised contrastive loss, are designed for node representation learning. For the OOS nodes, SCAGC can directly calculate their clustering labels.
  arXiv  Detail & Related papers  (2021-10-15T03:25:28Z)
• You Never Cluster Alone [150.94921340034688]
  We extend the mainstream contrastive learning paradigm to a cluster-level scheme, where all the data subjected to the same cluster contribute to a unified representation. We define a set of categorical variables as clustering assignment confidence, which links the instance-level learning track with the cluster-level one. By reparametrizing the assignment variables, TCC is trained end-to-end, requiring no alternating steps.
  arXiv  Detail & Related papers  (2021-06-03T14:59:59Z)
• Deep Fair Discriminative Clustering [24.237000220172906]
  We study a general notion of group-level fairness for binary and multi-state protected status variables (PSVs) We propose a refinement learning algorithm to combine the clustering goal with the fairness objective to learn fair clusters adaptively. Our framework shows promising results for novel clustering tasks including flexible fairness constraints, multi-state PSVs and predictive clustering.
  arXiv  Detail & Related papers  (2021-05-28T23:50:48Z)
• Scalable Hierarchical Agglomerative Clustering [65.66407726145619]
  Existing scalable hierarchical clustering methods sacrifice quality for speed. We present a scalable, agglomerative method for hierarchical clustering that does not sacrifice quality and scales to billions of data points.
  arXiv  Detail & Related papers  (2020-10-22T15:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.