Incubating Text Classifiers Following User Instruction with Nothing but LLM

• URL: http://arxiv.org/abs/2404.10877v2
• Date: Mon, 20 May 2024 07:42:53 GMT
• Title: Incubating Text Classifiers Following User Instruction with Nothing but LLM
• Authors: Letian Peng, Jingbo Shang,
• Abstract summary: We propose a framework to generate text classification data given arbitrary class definitions (i.e., user instruction) Our proposed Incubator is the first framework that can handle complicated and even mutually dependent classes.
• Score: 37.92922713921964
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we aim to generate text classification data given arbitrary class definitions (i.e., user instruction), so one can train a small text classifier without any human annotation or raw corpus. Compared with pioneer attempts, our proposed Incubator is the first framework that can handle complicated and even mutually dependent classes (e.g., "TED Talk given by Educator" and "Other"). Specifically, Incubator is an LLM firstly tuned on the instruction-to-data mappings that we obtained from classification datasets and descriptions on HuggingFace together with in-context augmentation by GPT-4. We then refine Incubator by learning on the cluster centers of semantic textual embeddings to emphasize the uniformity and semantic diversity in generations. We compare Incubator on various classification tasks with strong baselines such as direct LLM-based inference and training data generation by prompt engineering. Experiments show Incubator is able to (1) perform well on traditional benchmarks, (2) take label dependency and user preference into consideration, and (3) enable logical text mining by incubating multiple classifiers.

Related papers

• Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels [0.0]
  We assess the potential for researchers to augment or replace human-generated training data with surrogate training labels from large language models (LLMs) We employ a novel corpus of English-language text classification data sets from recent CSS articles in high-impact journals. For each task, we compare supervised classifiers fine-tuned using GPT-4 labels against classifiers fine-tuned with human annotations and against labels from GPT-4 and Mistral-7B with few-shot in-context learning. Our findings indicate that supervised classification models fine-tuned on LLM-generated labels perform comparably to models fine-tuned with labels from human
  arXiv  Detail & Related papers  (2024-06-25T15:20:25Z)
• Co-training for Low Resource Scientific Natural Language Inference [65.37685198688538]
  We propose a novel co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels. By assigning importance weights instead of filtering out examples based on an arbitrary threshold on the predicted confidence, we maximize the usage of automatically labeled data. The proposed method obtains an improvement of 1.5% in Macro F1 over the distant supervision baseline, and substantial improvements over several other strong SSL baselines.
  arXiv  Detail & Related papers  (2024-06-20T18:35:47Z)
• TnT-LLM: Text Mining at Scale with Large Language Models [24.731544646232962]
  Large Language Models (LLMs) automate the process of end-to-end label generation and assignment with minimal human effort. We show that TnT-LLM generates more accurate and relevant label when compared against state-of-the-art baselines. We also share our practical experiences and insights on the challenges and opportunities of using LLMs for large-scale text mining in real-world applications.
  arXiv  Detail & Related papers  (2024-03-18T18:45:28Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• Distinguishability Calibration to In-Context Learning [31.375797763897104]
  We propose a method to map a PLM-encoded embedding into a new metric space to guarantee the distinguishability of the resulting embeddings. We also take the advantage of hyperbolic embeddings to capture the hierarchical relations among fine-grained class-associated token embedding.
  arXiv  Detail & Related papers  (2023-02-13T09:15:00Z)
• Beyond prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations [24.3378487252621]
  We show that zero-shot text classification can be improved simply by clustering texts in the embedding spaces of pre-trained language models. Our approach achieves an average of 20% absolute improvement over prompt-based zero-shot learning.
  arXiv  Detail & Related papers  (2022-10-29T16:01:51Z)
• LeQua@CLEF2022: Learning to Quantify [76.22817970624875]
  LeQua 2022 is a new lab for the evaluation of methods for learning to quantify'' in textual datasets. The goal of this lab is to provide a setting for the comparative evaluation of methods for learning to quantify, both in the binary setting and in the single-label multiclass setting.
  arXiv  Detail & Related papers  (2021-11-22T14:54:20Z)
• Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems [20.55054374525828]
  This paper explores three novel approaches to improve the performance of speaker verification systems based on deep neural networks (DNN) Firstly, we propose the use of a learnable vector called Class token to replace the average global pooling mechanism to extract the embeddings. Second, we have added a distilled representation token for training a teacher-student pair of networks using the Knowledge Distillation (KD) philosophy.
  arXiv  Detail & Related papers  (2021-11-06T09:47:05Z)
• Binary Classification from Multiple Unlabeled Datasets via Surrogate Set Classification [94.55805516167369]
  We propose a new approach for binary classification from m U-sets for $mge2$. Our key idea is to consider an auxiliary classification task called surrogate set classification (SSC)
  arXiv  Detail & Related papers  (2021-02-01T07:36:38Z)
• Learning and Evaluating Representations for Deep One-class Classification [59.095144932794646]
  We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T23:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.