Human-interpretable clustering of short-text using large language models

• URL: http://arxiv.org/abs/2405.07278v1
• Date: Sun, 12 May 2024 12:55:40 GMT
• Title: Human-interpretable clustering of short-text using large language models
• Authors: Justin K. Miller, Tristram J. Alexander,
• Abstract summary: We show that large language models can be used to cluster human-generated content. This success is validated by both human reviewers and ChatGPT.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models have seen extraordinary growth in popularity due to their human-like content generation capabilities. We show that these models can also be used to successfully cluster human-generated content, with success defined through the measures of distinctiveness and interpretability. This success is validated by both human reviewers and ChatGPT, providing an automated means to close the 'validation gap' that has challenged short-text clustering. Comparing the machine and human approaches we identify the biases inherent in each, and question the reliance on human-coding as the 'gold standard'. We apply our methodology to Twitter bios and find characteristic ways humans describe themselves, agreeing well with prior specialist work, but with interesting differences characteristic of the medium used to express identity.

Related papers

• The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks [17.5336703613751]
  This study benchmarks leading large language models and vision language models against human performance on the Wechsler Adult Intelligence Scale (WAIS-IV) Most models demonstrated exceptional capabilities in the storage, retrieval, and manipulation of tokens such as arbitrary sequences of letters and numbers. Despite these broad strengths, we observed consistently poor performance on the Perceptual Reasoning Index (PRI) from multimodal models.
  arXiv  Detail & Related papers  (2024-10-09T19:22:26Z)
• Virtual Personas for Language Models via an Anthology of Backstories [5.2112564466740245]
  "Anthology" is a method for conditioning large language models to particular virtual personas by harnessing open-ended life narratives. We show that our methodology enhances the consistency and reliability of experimental outcomes while ensuring better representation of diverse sub-populations.
  arXiv  Detail & Related papers  (2024-07-09T06:11:18Z)
• ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness [3.126776200660494]
  We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models. Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks.
  arXiv  Detail & Related papers  (2023-05-22T11:46:32Z)
• AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays [66.36541161082856]
  ChatGPT and similar generative AI models have attracted hundreds of millions of users. This study compares human-written versus ChatGPT-generated argumentative student essays.
  arXiv  Detail & Related papers  (2023-04-24T12:58:28Z)
• Auditing Gender Presentation Differences in Text-to-Image Models [54.16959473093973]
  We study how gender is presented differently in text-to-image models. By probing gender indicators in the input text, we quantify the frequency differences of presentation-centric attributes. We propose an automatic method to estimate such differences.
  arXiv  Detail & Related papers  (2023-02-07T18:52:22Z)
• Chain of Hindsight Aligns Language Models with Feedback [62.68665658130472]
  We propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.
  arXiv  Detail & Related papers  (2023-02-06T10:28:16Z)
• COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation [56.520470678876656]
  bias inherent in user written text can associate different levels of linguistic quality with users' protected attributes. We introduce a general framework to achieve measure-specific counterfactual fairness in explanation generation.
  arXiv  Detail & Related papers  (2022-10-14T02:29:10Z)
• Estimating the Personality of White-Box Language Models [0.589889361990138]
  Large-scale language models, which are trained on large corpora of text, are being used in a wide range of applications everywhere. Existing research shows that these models can and do capture human biases. Many of these biases, especially those that could potentially cause harm, are being well-investigated. However, studies that infer and change human personality traits inherited by these models have been scarce or non-existent.
  arXiv  Detail & Related papers  (2022-04-25T23:53:53Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems [17.749995931459136]
  We propose that a metric based on linguistic features may be able to maintain good correlation with human judgment and be interpretable. To support this proposition, we measure and analyze various linguistic features on dialogues produced by multiple dialogue models. We find that the features' behaviour is consistent with the known properties of the models tested, and is similar across domains.
  arXiv  Detail & Related papers  (2021-04-13T16:28:00Z)
• Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning [91.58529629419135]
  We consider how to characterise visual groupings discovered automatically by deep neural networks. We introduce two concepts, visual learnability and describability, that can be used to quantify the interpretability of arbitrary image groupings.
  arXiv  Detail & Related papers  (2020-10-27T18:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.