Large Language Models Do Multi-Label Classification Differently

• URL: http://arxiv.org/abs/2505.17510v1
• Date: Fri, 23 May 2025 06:04:36 GMT
• Title: Large Language Models Do Multi-Label Classification Differently
• Authors: Marcus Ma, Georgios Chochlakis, Niyantha Maruthu Pandiyan, Jesse Thomason, Shrikanth Narayanan,
• Abstract summary: Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied.<n>We investigate how autoregressive LLMs perform multi-label classification, with a focus on subjective tasks, by analyzing the output distributions of the models in each generation step.<n>We find that their predictive behavior reflects the multiple steps in the underlying language modeling required to generate all relevant labels as they tend to suppress all but one label at each step.
• Score: 27.818253292071386
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. We investigate how autoregressive LLMs perform multi-label classification, with a focus on subjective tasks, by analyzing the output distributions of the models in each generation step. We find that their predictive behavior reflects the multiple steps in the underlying language modeling required to generate all relevant labels as they tend to suppress all but one label at each step. We further observe that as model scale increases, their token distributions exhibit lower entropy, yet the internal ranking of the labels improves. Finetuning methods such as supervised finetuning and reinforcement learning amplify this phenomenon. To further study this issue, we introduce the task of distribution alignment for multi-label settings: aligning LLM-derived label distributions with empirical distributions estimated from annotator responses in subjective tasks. We propose both zero-shot and supervised methods which improve both alignment and predictive performance over existing approaches.

Related papers

• Learning Semantic-Aware Threshold for Multi-Label Image Recognition with Partial Labels [12.477433449244543]
  Multi-label image recognition with partial labels (MLR-PL) is designed to train models using a mix of known and unknown labels.<n>Traditional methods rely on semantic or feature correlations to create pseudo-labels for unidentified labels.<n>In our study, we introduce the Semantic-Aware Threshold Learning (SATL) algorithm.
  arXiv  Detail & Related papers  (2025-07-31T05:54:10Z)
• Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning [12.052388861361937]
  Recent studies have overemphasized co-occurrence relationships among labels, leading to suboptimal models.<n>We propose the Multi-Label Visual Prompt Tuning framework to balance correlative and discriminative relationships among labels.<n>Our proposed approach achieves competitive results and outperforms SOTA methods on multiple pre-trained models.
  arXiv  Detail & Related papers  (2025-04-14T08:52:50Z)
• Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification [49.09505771145326]
  We propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels. Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-04-26T06:00:27Z)
• Class-Distribution-Aware Pseudo Labeling for Semi-Supervised Multi-Label Learning [97.88458953075205]
  Pseudo-labeling has emerged as a popular and effective approach for utilizing unlabeled data. This paper proposes a novel solution called Class-Aware Pseudo-Labeling (CAP) that performs pseudo-labeling in a class-aware manner.
  arXiv  Detail & Related papers  (2023-05-04T12:52:18Z)
• Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision [75.1860418333995]
  Programmatic Weak Supervision (PWS) has emerged as a widespread paradigm to synthesize training labels efficiently. The core component of PWS is the label model, which infers true labels by aggregating the outputs of multiple noisy supervision sources as labeling functions. Existing statistical label models typically rely only on the outputs of LF, ignoring the instance features when modeling the underlying generative process.
  arXiv  Detail & Related papers  (2022-10-06T07:28:53Z)
• A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation [42.0958430465578]
  We study the partial multi-label (PML) image classification problem. Existing PML methods typically design a disambiguation strategy to filter out noisy labels. We propose a deep model for PML to enhance the representation and discrimination ability.
  arXiv  Detail & Related papers  (2022-07-06T02:49:02Z)
• Self-Adaptive Label Augmentation for Semi-supervised Few-shot Classification [121.63992191386502]
  Few-shot classification aims to learn a model that can generalize well to new tasks when only a few labeled samples are available. We propose a semi-supervised few-shot classification method that assigns an appropriate label to each unlabeled sample by a manually defined metric. A major novelty of SALA is the task-adaptive metric, which can learn the metric adaptively for different tasks in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-06-16T13:14:03Z)
• Evolving Multi-Label Fuzzy Classifier [5.53329677986653]
  Multi-label classification has attracted much attention in the machine learning community to address the problem of assigning single samples to more than one class at the same time. We propose an evolving multi-label fuzzy classifier (EFC-ML) which is able to self-adapt and self-evolve its structure with new incoming multi-label samples in an incremental, single-pass manner.
  arXiv  Detail & Related papers  (2022-03-29T08:01:03Z)
• PLM: Partial Label Masking for Imbalanced Multi-label Classification [59.68444804243782]
  Neural networks trained on real-world datasets with long-tailed label distributions are biased towards frequent classes and perform poorly on infrequent classes. We propose a method, Partial Label Masking (PLM), which utilizes this ratio during training. Our method achieves strong performance when compared to existing methods on both multi-label (MultiMNIST and MSCOCO) and single-label (imbalanced CIFAR-10 and CIFAR-100) image classification datasets.
  arXiv  Detail & Related papers  (2021-05-22T18:07:56Z)
• UniT: Unified Knowledge Transfer for Any-shot Object Detection and Segmentation [52.487469544343305]
  Methods for object detection and segmentation rely on large scale instance-level annotations for training. We propose an intuitive and unified semi-supervised model that is applicable to a range of supervision.
  arXiv  Detail & Related papers  (2020-06-12T22:45:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.