Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student

• URL: http://arxiv.org/abs/2510.25303v1
• Date: Wed, 29 Oct 2025 09:14:41 GMT
• Title: Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student
• Abstract summary: Multimodal sarcasm detection is challenging in low-resource settings.<n>We propose PEKD, a unified framework that enhances PEFT methods via distillation from an expert model trained on large-scale sarcasm data.
• Score: 2.8074191213147652
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal sarcasm detection is challenging, especially in low-resource settings where subtle image-text contradictions are hard to learn due to scarce annotated data, which hinders the model's performance. Parameter-efficient fine-tuning (PEFT) methods like adapters, LoRA, and prompt tuning reduce overfitting but struggle to reach optimal performance due to limited supervision from few-shot data. We propose PEKD, a unified framework that enhances PEFT methods via distillation from an expert model trained on large-scale sarcasm data, which acts as the teacher. To mitigate unreliable signals from the teacher, we introduce an entropy-aware gating mechanism that dynamically adjusts the distillation strength based on teacher confidence. Experiments on two public datasets demonstrate that our PEKD framework enables PEFT methods to outperform both prior parameter-efficient approaches and large multimodal models, achieving strong results in the few-shot scenario. The framework is modular and adaptable to a wide range of multimodal models and tasks.

Related papers

• Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations [6.992477865573147]
  Shift-Augmented Knowledge Distillation (SAKD) is a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation.<n>SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches.
  arXiv  Detail & Related papers  (2026-07-13T13:39:35Z)
• Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families [32.34854415784466]
  On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.<n>Existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series.<n>In this work, we enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers.
  arXiv  Detail & Related papers  (2026-06-08T13:12:01Z)
• Deep Reprogramming Distillation for Medical Foundation Models [74.76894329393512]
  We propose a novel framework called Deep Reprogramming Distillation (DRD) to combat the general adaptation challenge.<n>Specifically, DRD introduces the novel reprogramming module that on the one side overcomes the domain and task discrepancy between pretraining and downstream scenarios.<n>To mitigate variability under different training conditions, we design a centered kernel alignment (CKA) distillation method to promote robust knowledge transfer.
  arXiv  Detail & Related papers  (2026-05-06T03:22:39Z)
• Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations [11.869032690386524]
  We introduce a novel strategy called Counterfactual-explanation-infused Distillation CoD for few-shot task-aware knowledge distillation.<n>We mathematically show that CFEs can improve parameter estimation by providing more informative examples near the teacher's decision boundary.<n>We perform experiments across various datasets and LLMs to show that CoD outperforms standard distillation approaches in few-shot regimes.
  arXiv  Detail & Related papers  (2025-10-24T16:36:34Z)
• AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models [35.71783914954563]
  We propose a novel framework that integrates multi-modal feature fusion, multi-teacher distillation, and adaptive optimization to deliver lightweight yet effective retrieval models.<n>Experiments on three benchmark datasets demonstrate that AMMKD achieves superior performance while significantly reducing model complexity, validating its effectiveness and flexibility.
  arXiv  Detail & Related papers  (2025-08-23T04:52:20Z)
• PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning [54.73049408950049]
  We propose a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning.<n>Our approach improves unified multimodal retrieval from both structural and learning perspectives.
  arXiv  Detail & Related papers  (2025-07-10T16:47:25Z)
• Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detection [2.497926557563177]
  multimodal image-text sarcasm on social media poses challenges for opinion mining systems.<n>We propose AdS-CLIP, a lightweight framework that inserts adapters only in the upper layers to preserve low-level unimodal representations.<n> Experiments on two public benchmarks demonstrate that AdS-CLIP not only outperforms standard PEFT methods but also existing multimodal baselines with significantly fewer trainable parameters.
  arXiv  Detail & Related papers  (2025-07-06T18:51:00Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation [10.48108719012248]
  We focus on Knowledge Distillation (KD), where a compact student model is trained to mimic a larger teacher model. In contrast to much of the previous work, we scale up the parameters of the student model during training.
  arXiv  Detail & Related papers  (2024-11-10T12:40:59Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [58.41784639847413]
  Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. Results indicate that our proposed method can outperform SOTA PKD methods.
  arXiv  Detail & Related papers  (2024-08-16T22:11:01Z)
• LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models [20.5908375260123]
  Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. We present LoRETTA, a framework that significantly reduces trainable parameters through tensor-train decomposition. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to $100times$ fewer parameters on the LLaMA-2-7B models.
  arXiv  Detail & Related papers  (2024-02-18T01:20:00Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.