LabelFusion: Learning to Fuse LLMs and Transformer Classifiers for Robust Text Classification

• URL: http://arxiv.org/abs/2512.10793v1
• Date: Thu, 11 Dec 2025 16:39:07 GMT
• Title: LabelFusion: Learning to Fuse LLMs and Transformer Classifiers for Robust Text Classification
• Authors: Michael Schlee, Christoph Weisser, Timo Kivimäki, Melchizedek Mashiku, Benjamin Saefken,
• Abstract summary: LabelFusion is a fusion ensemble for text classification.<n>It learns to combine a transformer-based classifier with one or more Large Language Models.<n>It achieves 92.4% accuracy on AG News and 92.3% on 10-class Reuters 21578 topic classification.
• Score: 0.7611870296994722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LabelFusion is a fusion ensemble for text classification that learns to combine a traditional transformer-based classifier (e.g., RoBERTa) with one or more Large Language Models (LLMs such as OpenAI GPT, Google Gemini, or DeepSeek) to deliver accurate and cost-aware predictions across multi-class and multi-label tasks. The package provides a simple high-level interface (AutoFusionClassifier) that trains the full pipeline end-to-end with minimal configuration, and a flexible API for advanced users. Under the hood, LabelFusion integrates vector signals from both sources by concatenating the ML backbone's embeddings with the LLM-derived per-class scores -- obtained through structured prompt-engineering strategies -- and feeds this joint representation into a compact multi-layer perceptron (FusionMLP) that produces the final prediction. This learned fusion approach captures complementary strengths of LLM reasoning and traditional transformer-based classifiers, yielding robust performance across domains -- achieving 92.4% accuracy on AG News and 92.3% on 10-class Reuters 21578 topic classification -- while enabling practical trade-offs between accuracy, latency, and cost.

Related papers

• Contextual Gating within the Transformer Stack: Synergistic Feature Modulation for Enhanced Lyrical Classification and Calibration [0.0]
  This study introduces a significant architectural advancement in feature fusion for lyrical content classification.<n>I propose the SFL Transformer, a novel deep learning model that utilizes a Contextual Gating mechanism.<n>The model is applied to a challenging binary classification task derived from UMAP-reduced lyrical embeddings.
  arXiv  Detail & Related papers  (2025-11-27T08:23:45Z)
• Lightweight Safety Classification Using Pruned Language Models [0.0]
  We introduce a novel technique for content safety and prompt injection classification for Large Language Models.<n>Our approach delivers superior performance surpassing GPT-4o and special-purpose models fine-tuned for each task.<n>Our results indicate that a single general-purpose LLM can be used to classify content safety, detect prompt injections, and simultaneously generate output tokens.
  arXiv  Detail & Related papers  (2024-12-18T02:13:13Z)
• How to Make LLMs Strong Node Classifiers? [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks [2.497666465251894]
  TransformerRanker is a lightweight library that ranks pre-trained language models for classification tasks. Our library implements current approaches for transferability estimation. We make TransformerRanker available as a pip-installable open-source library.
  arXiv  Detail & Related papers  (2024-09-09T18:47:00Z)
• Cool-Fusion: Fuse Large Language Models without Training [73.17551121242602]
  Cool-Fusion fuses the knowledge of source LLMs, which does not require training.<n>Experiments have been conducted across a variety of benchmark datasets.<n>On GSM8K, Cool-Fusion increases accuracy from three strong source LLMs by a significant margin of 17.4%.
  arXiv  Detail & Related papers  (2024-07-29T09:02:19Z)
• Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition [48.84506301960988]
  Cued Speech (CS) is a pure visual coding method used by hearing-impaired people. automatic CS recognition (ACSR) seeks to transcribe visual cues of speech into text.
  arXiv  Detail & Related papers  (2024-01-31T05:20:29Z)
• Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation [59.91357714415056]
  We propose two Transformer variants: Context-Sharing Transformer (CST) and Semantic Gathering-Scattering Transformer (S GST) CST learns the global-shared contextual information within image frames with a lightweight computation; S GST models the semantic correlation separately for the foreground and background. Compared with the baseline that uses vanilla Transformers for multi-stage fusion, ours significantly increase the speed by 13 times and achieves new state-of-the-art ZVOS performance.
  arXiv  Detail & Related papers  (2023-08-13T06:12:00Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• A multi-model-based deep learning framework for short text multiclass classification with the imbalanced and extremely small data set [0.6875312133832077]
  This paper proposes a multimodel-based deep learning framework for short-text multiclass classification with an imbalanced and extremely small data set. It retains the state-of-the-art baseline performance in terms of precision, recall, accuracy, and F1 score.
  arXiv  Detail & Related papers  (2022-06-24T00:51:02Z)
• Exploiting Local and Global Features in Transformer-based Extreme Multi-label Text Classification [28.28186933768281]
  We propose an approach that combines both the local and global features produced by Transformer models to improve the prediction power of the classifier. Our experiments show that the proposed model either outperforms or is comparable to the state-of-the-art methods on benchmark datasets.
  arXiv  Detail & Related papers  (2022-04-02T19:55:23Z)
• Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification [78.83284164605473]
  emphFunnelling (Fun) is a recently proposed method for cross-lingual text classification. We describe emphGeneralized Funnelling (gFun) as a generalization of Fun. We show that gFun substantially improves over Fun and over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-09-17T23:33:04Z)
• Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks [75.69896269357005]
  Mixup is the latest data augmentation technique that linearly interpolates input examples and the corresponding labels. In this paper, we explore how to apply mixup to natural language processing tasks. We incorporate mixup to transformer-based pre-trained architecture, named "mixup-transformer", for a wide range of NLP tasks.
  arXiv  Detail & Related papers  (2020-10-05T23:37:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.