SALSA: Single-pass Autoregressive LLM Structured Classification

• URL: http://arxiv.org/abs/2510.22691v1
• Date: Sun, 26 Oct 2025 14:28:42 GMT
• Title: SALSA: Single-pass Autoregressive LLM Structured Classification
• Abstract summary: We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning.<n>During inference, the model's output is projected only onto the logits of the relevant class tokens, enabling efficient and accurate classification in a single forward pass.
• Score: 1.9371675344367494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite their impressive generalization capabilities, instruction-tuned Large Language Models often underperform on text classification benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning, thereby avoiding cold-start training. Each class label is mapped to a distinct output token, and prompts are constructed to elicit a single-token response. During inference, the model's output is projected only onto the logits of the relevant class tokens, enabling efficient and accurate classification in a single forward pass. SALSA achieves state-of-the-art results across diverse benchmarks, demonstrating its robustness and scalability for LLM-based classification applications.

Related papers

• Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection [1.9371675344367494]
  SemEval-2026 Task 13 Subtask A operationalizes detection as binary classification over code snippets.<n>We propose a SALSA-style formulation that maps each class to a dedicated output token and trains the model to emit a single-token label in a structured response.
  arXiv  Detail & Related papers  (2026-06-23T19:17:11Z)
• Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER [0.0]
  Large language models encode extensive world knowledge valuable for zero-shot named entity recognition.<n>We propose Just Pass Twice (JPT), a simple yet effective method that enables causal LLMs to perform discriminative token classification with full bidirectional context.<n>Our approach achieves state-of-the-art results on zero-shot NER benchmarks, surpassing the previous best method by +7.9 F1 on average across CrossNER and MIT benchmarks, being over 20x faster than comparable generative methods.
  arXiv  Detail & Related papers  (2026-04-06T20:39:31Z)
• Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches [0.0]
  We explore strategies to fine-tune decoder-only Large Language Models (LLMs) for downstream text classification under resource constraints.<n>Two approaches are investigated: (1) attaching a classification head to a pre-trained causal LLM and fine-tuning on the task, and (2) instruction-tuning the LLM in a prompt->response format for classification.
  arXiv  Detail & Related papers  (2025-12-14T13:02:06Z)
• Divide, Cache, Conquer: Dichotomic Prompting for Efficient Multi-Label LLM-Based Classification [0.2799896314754614]
  We introduce a method for efficient multi-label text classification with large language models (LLMs)<n>Instead of generating all labels in a single structured response, each target dimension is queried independently.<n>Our findings suggest that decomposing multi-label classification into dichotomic queries offers a scalable and effective framework.
  arXiv  Detail & Related papers  (2025-11-05T19:53:51Z)
• Hierarchical Representation Matching for CLIP-based Class-Incremental Learning [80.2317078787969]
  Class-Incremental Learning (CIL) aims to endow models with the ability to continuously adapt to evolving data streams.<n>Recent advances in pre-trained vision-language models (e.g., CLIP) provide a powerful foundation for this task.<n>We introduce HiErarchical Representation MAtchiNg (HERMAN) for CLIP-based CIL.
  arXiv  Detail & Related papers  (2025-09-26T17:59:51Z)
• GLiClass: Generalist Lightweight Model for Sequence Classification Tasks [49.2639069781367]
  We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks.<n>Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios.
  arXiv  Detail & Related papers  (2025-08-11T06:22:25Z)
• Label-Guided In-Context Learning for Named Entity Recognition [14.63059248497416]
  In-context learning (ICL) enables large language models to perform new tasks using only a few demonstrations.<n>We introduce DEER, a new method that leverages training labels through token-level statistics to improve ICL performance.
  arXiv  Detail & Related papers  (2025-05-29T17:54:32Z)
• CAARMA: Class Augmentation with Adversarial Mixup Regularization [59.938216853878636]
  Speaker verification is a typical zero-shot learning task.<n>We introduce CAARMA, a class augmentation framework.<n>Our framework demonstrates a significant improvement of 8% over all baseline models.
  arXiv  Detail & Related papers  (2025-03-20T21:41:16Z)
• Self-Regularization with Sparse Autoencoders for Controllable LLM-based Classification [29.74457390987092]
  We propose a novel framework to identify and regularize unintended features in large language models (LLMs) latent spaces.<n>We evaluate the proposed framework on three real-world tasks, including toxic chat detection, reward modeling, and disease diagnosis.
  arXiv  Detail & Related papers  (2025-02-19T22:27:59Z)
• SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification [6.227343685358882]
  We present a model-agnostic framework that sparsifies and integrates internal neurons of intermediate layers of Large Language Models for text classification. SPIN significantly improves text classification accuracy, efficiency, and interpretability.
  arXiv  Detail & Related papers  (2023-11-27T16:28:20Z)
• LLM-augmented Preference Learning from Natural Language [19.700169351688768]
  Large Language Models (LLMs) are equipped to deal with larger context lengths. LLMs can consistently outperform the SotA when the target text is large. Few-shot learning yields better performance than zero-shot learning.
  arXiv  Detail & Related papers  (2023-10-12T17:17:27Z)
• MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation [90.73815426893034]
  We propose a transformer-based framework that aims to enhance weakly supervised semantic segmentation. We introduce a Multi-Class Token transformer, which incorporates multiple class tokens to enable class-aware interactions with the patch tokens. A Contrastive-Class-Token (CCT) module is proposed to enhance the learning of discriminative class tokens.
  arXiv  Detail & Related papers  (2023-08-06T03:30:20Z)
• Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models [94.30953696090758]
  We build compositional end-to-end spoken language understanding systems. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to token-level representations. Our models outperform both cascaded and direct end-to-end models on a labeling task of named entity recognition.
  arXiv  Detail & Related papers  (2022-10-27T19:33:18Z)
• Self-Adaptive Label Augmentation for Semi-supervised Few-shot Classification [121.63992191386502]
  Few-shot classification aims to learn a model that can generalize well to new tasks when only a few labeled samples are available. We propose a semi-supervised few-shot classification method that assigns an appropriate label to each unlabeled sample by a manually defined metric. A major novelty of SALA is the task-adaptive metric, which can learn the metric adaptively for different tasks in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-06-16T13:14:03Z)
• MGIMN: Multi-Grained Interactive Matching Network for Few-shot Text Classification [9.9875634964736]
  Text classification struggles to generalize to unseen classes with very few labeled text instances per class. We propose a meta-learning based method MGIMN which performs instance-wise comparison followed by aggregation to generate class-wise matching vectors.
  arXiv  Detail & Related papers  (2022-04-11T08:58:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.