Class Distillation with Mahalanobis Contrast: An Efficient Training Paradigm for Pragmatic Language Understanding Tasks

• URL: http://arxiv.org/abs/2505.11829v1
• Date: Sat, 17 May 2025 04:34:19 GMT
• Title: Class Distillation with Mahalanobis Contrast: An Efficient Training Paradigm for Pragmatic Language Understanding Tasks
• Authors: Chenlu Wang, Weimin Lyu, Ritwik Banerjee,
• Abstract summary: We propose textbfClass textbfDistillation (ClaD), a novel training paradigm that targets the core challenge: distilling a small, well-defined target class.<n>ClaD integrates two key innovations: (i) a loss function informed by the structural properties of class distributions, based on Mahalanobis distance, and (ii) an interpretable decision algorithm optimized for class separation.
• Score: 1.1060425537315088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting deviant language such as sexism, or nuanced language such as metaphors or sarcasm, is crucial for enhancing the safety, clarity, and interpretation of online social discourse. While existing classifiers deliver strong results on these tasks, they often come with significant computational cost and high data demands. In this work, we propose \textbf{Cla}ss \textbf{D}istillation (ClaD), a novel training paradigm that targets the core challenge: distilling a small, well-defined target class from a highly diverse and heterogeneous background. ClaD integrates two key innovations: (i) a loss function informed by the structural properties of class distributions, based on Mahalanobis distance, and (ii) an interpretable decision algorithm optimized for class separation. Across three benchmark detection tasks -- sexism, metaphor, and sarcasm -- ClaD outperforms competitive baselines, and even with smaller language models and orders of magnitude fewer parameters, achieves performance comparable to several large language models (LLMs). These results demonstrate ClaD as an efficient tool for pragmatic language understanding tasks that require gleaning a small target class from a larger heterogeneous background.

Related papers

• Multi-task Learning with Active Learning for Arabic Offensive Speech Detection [1.534667887016089]
  This paper proposes a novel framework that integrates multi-task learning (MTL) with active learning to enhance offensive speech detection in Arabic social media text.<n>Our approach dynamically adjusts task weights during training to balance the contribution of each task and optimize performance.<n> Experimental results on the OSACT2022 dataset show that the proposed framework achieves a state-of-the-art macro F1-score of 85.42%.
  arXiv  Detail & Related papers  (2025-06-03T11:17:03Z)
• Extreme Speech Classification in the Era of LLMs: Exploring Open-Source and Proprietary Models [0.30693357740321775]
  ChatGPT has drawn global attention to the potential applications of Large Language Models (LLMs)<n>We leverage the Indian subset of the extreme speech dataset from Maronikolakis et al. (2022) to develop an effective classification framework using LLMs.<n>We evaluate open-source Llama models against closed-source OpenAI models, finding that while pre-trained LLMs show moderate efficacy, fine-tuning with domain-specific data significantly enhances performance.
  arXiv  Detail & Related papers  (2025-02-21T02:31:05Z)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• LARA: Linguistic-Adaptive Retrieval-Augmentation for Multi-Turn Intent Classification [6.459396785817196]
  LARA is a Linguistic-Adaptive Retrieval-Augmentation framework to enhance accuracy in multi-turn classification tasks across six languages. Our experiments demonstrate that LARA achieves state-of-the-art performance on multi-turn intent classification tasks.
  arXiv  Detail & Related papers  (2024-03-25T07:38:40Z)
• Intent-conditioned and Non-toxic Counterspeech Generation using Multi-Task Instruction Tuning with RLAIF [14.2594830589926]
  Counterspeech, defined as a response to online hate speech, is increasingly used as a non-censorial solution. Our study introduces CoARL, a novel framework enhancing counterspeech generation by modeling the pragmatic implications underlying social biases in hateful statements. CoARL's first two phases involve sequential multi-instruction tuning, teaching the model to understand intents, reactions, and harms of offensive statements, and then learning task-specific low-rank adapter weights for generating intent-conditioned counterspeech.
  arXiv  Detail & Related papers  (2024-03-15T08:03:49Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• Distilling Large Vision-Language Model with Out-of-Distribution Generalizability [43.984177729641615]
  This paper investigates the distillation of visual representations in large teacher vision-language models into lightweight student models. We propose several metrics and conduct extensive experiments to investigate their techniques. The results demonstrate significant improvements in zero-shot and few-shot student performance on open-vocabulary out-of-distribution classification.
  arXiv  Detail & Related papers  (2023-07-06T17:05:26Z)
• Sentence Representation Learning with Generative Objective rather than Contrastive Objective [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
  arXiv  Detail & Related papers  (2022-10-16T07:47:46Z)
• MaPLe: Multi-modal Prompt Learning [54.96069171726668]
  We propose Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable performance and achieves an absolute gain of 3.45% on novel classes.
  arXiv  Detail & Related papers  (2022-10-06T17:59:56Z)
• A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference [54.678516076366506]
  Natural Language Inference (NLI) is a growingly essential task in natural language understanding. Here we propose a multi-level supervised contrastive learning framework named MultiSCL for low-resource natural language inference.
  arXiv  Detail & Related papers  (2022-05-31T05:54:18Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss [37.8626106992769]
  We study the effects of minimizing an alternate training objective that fosters a model to generate alternate response and score it on semantic similarity. We explore this idea on two different sized data sets on the task of next utterance generation in goal oriented dialogues.
  arXiv  Detail & Related papers  (2021-06-20T04:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.