Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detection

• URL: http://arxiv.org/abs/2507.04508v2
• Date: Wed, 29 Oct 2025 05:51:00 GMT
• Title: Adapter-state Sharing CLIP for Parameter-efficient Multimodal Sarcasm Detection
• Abstract summary: multimodal image-text sarcasm on social media poses challenges for opinion mining systems.<n>We propose AdS-CLIP, a lightweight framework that inserts adapters only in the upper layers to preserve low-level unimodal representations.<n> Experiments on two public benchmarks demonstrate that AdS-CLIP not only outperforms standard PEFT methods but also existing multimodal baselines with significantly fewer trainable parameters.
• Score: 2.497926557563177
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing prevalence of multimodal image-text sarcasm on social media poses challenges for opinion mining systems. Existing approaches rely on full fine-tuning of large models, making them unsuitable to adapt under resource-constrained settings. While recent parameter-efficient fine-tuning (PEFT) methods offer promise, their off-the-shelf use underperforms on complex tasks like sarcasm detection. We propose AdS-CLIP (Adapter-state Sharing in CLIP), a lightweight framework built on CLIP that inserts adapters only in the upper layers to preserve low-level unimodal representations in the lower layers and introduces a novel adapter-state sharing mechanism, where textual adapters guide visual ones to promote efficient cross-modal learning in the upper layers. Experiments on two public benchmarks demonstrate that AdS-CLIP not only outperforms standard PEFT methods but also existing multimodal baselines with significantly fewer trainable parameters.

Related papers

• When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning [31.20315349532034]
  We show that task-specific LoRA adapters in Continual Learning exhibit significant low-rank redundancy.<n>We propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations.
  arXiv  Detail & Related papers  (2026-06-26T14:19:03Z)
• PERL: Parameter Efficient Reasoning in CLIP Latent Space [11.607257085664727]
  We introduce PERL, a lightweight adaptation framework that augments a frozen CLIP model with a compact shared reasoning module applied recurrently.<n>PerL achieves the best parameter-performance trade-off among the compared methods under a fast-adaptation few-shot setting.<n>Overall, our results suggest that iterative latent reasoning provides a complementary adaptation mechanism to parameter scaling in disc-native vision-language models.
  arXiv  Detail & Related papers  (2026-05-18T14:25:25Z)
• BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning [84.56022893225422]
  Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge.<n>Applying vision-language models such as CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities.
  arXiv  Detail & Related papers  (2025-11-14T15:51:40Z)
• Regularizing Subspace Redundancy of Low-Rank Adaptation [54.473090597164834]
  We propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation.<n>Our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks.<n>As a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs.
  arXiv  Detail & Related papers  (2025-07-28T11:52:56Z)
• Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts [72.22148263683037]
  We study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures.<n>First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature.<n>Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks.
  arXiv  Detail & Related papers  (2025-07-09T03:25:45Z)
• FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts [47.35092228595656]
  FLoE is a novel PEFT framework that introduces two key innovations: (i) a Fisher information-guided importance scoring mechanism to dynamically identify task-critical transformer layers for MoE-based low-rank adaptation, enabling sparse adapter deployment; and (ii) a Bayesian optimization-driven rank allocator that automatically determines optimal LoRA ranks on specific datasets without exhaustive grid search.<n>Experiments across diverse LLMs and benchmarks reveal that FLoE achieves impressive efficiency-accuracy trade-offs, making FLoE particularly advantageous in resource-constrained environments that necessitate rapid adaptation.
  arXiv  Detail & Related papers  (2025-05-31T10:27:08Z)
• Efficient Multi-modal Long Context Learning for Training-free Adaptation [96.21248144937627]
  This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC)<n>It embeds demonstration examples directly into the model input.<n>It condenses long-context multimodal inputs into compact, task-specific memory representations.
  arXiv  Detail & Related papers  (2025-05-26T10:49:44Z)
• Enhancing Multi-modal Models with Heterogeneous MoE Adapters for Fine-tuning [3.8984478257737734]
  Multi-modal models excel in cross-modal tasks but are computationally expensive due to their billions of parameters.<n>Existing methods primarily focus on uni-modal processing, overlooking the critical modal fusion needed for multi-modal tasks.<n>We propose a mixture of experts that extend the traditional PEFT framework to support multi-modal expert combinations and improve information interaction.
  arXiv  Detail & Related papers  (2025-03-26T15:26:18Z)
• PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition [7.966499123076283]
  Vision-Language Models (VLMs) like CLIP offer promising solutions for Dynamic Facial Expression Recognition (DFER)<n>We propose PE-CLIP, a parameter-efficient fine-tuning framework that adapts CLIP for DFER while significantly reducing trainable parameters.<n>By balancing efficiency and accuracy, PE-CLIP sets a new benchmark in resource-efficient DFER.
  arXiv  Detail & Related papers  (2025-03-21T08:45:50Z)
• Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning [5.074620301447097]
  We propose a hierarchical scheme for expert allocation and rank configuration, HILO, for large language models (LLMs)<n>HILO dynamically adjusts the number and rank of adapter experts across layers, matching the varying representational complexity of model layers in adapter-granularity.<n>Experiments on multiple benchmark tasks demonstrate that HILO outperforms existing methods in accuracy while introducing fewer trainable parameters.
  arXiv  Detail & Related papers  (2025-02-06T08:58:03Z)
• FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network [19.466279425330857]
  We propose a novel multimodal object detector, named Low-rank Modal Adaptors (LMA) with a shared backbone. Our work was submitted to ACM MM in April 2024, but was rejected.
  arXiv  Detail & Related papers  (2024-07-23T02:27:52Z)
• CapS-Adapter: Caption-based MultiModal Adapter in Zero-Shot Classification [3.594351309950969]
  CapS-Adapter is an innovative method that harnesses both image and caption features to exceed existing state-of-the-art techniques in training-free scenarios. Our method achieves outstanding zero-shot classification results across 19 benchmark datasets, improving accuracy by 2.19% over the previous leading method.
  arXiv  Detail & Related papers  (2024-05-26T14:50:40Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval [4.4173427917548524]
  MultiWay-Adapter (MWA) is a novel framework featuring an 'Alignment Enhancer' This enhancer deepens inter-modal alignment, enabling high transferability with minimal tuning effort. Experiments show that unlike prior efficient tuning approaches, MWA maintains model effectiveness, while reducing training time by up-to 57%.
  arXiv  Detail & Related papers  (2023-09-04T10:48:29Z)
• Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models [106.19385911520652]
  Lottery Ticket Hypothesis (LTH) and its variants have been exploited to prune large pre-trained models generating parameterworks. LTH is enormously inhibited by repetitive full training and pruning routine of iterative magnitude pruning (IMP) We propose Instant Soup Pruning (ISP) to generate lottery ticket quality IMPworks.
  arXiv  Detail & Related papers  (2023-06-18T03:09:52Z)
• Parameter-efficient Tuning of Large-scale Multimodal Foundation Model [68.24510810095802]
  We propose A graceful prompt framework for cross-modal transfer (Aurora) to overcome these challenges. Considering the redundancy in existing architectures, we first utilize the mode approximation to generate 0.1M trainable parameters to implement the multimodal prompt tuning. A thorough evaluation on six cross-modal benchmarks shows that it not only outperforms the state-of-the-art but even outperforms the full fine-tuning approach.
  arXiv  Detail & Related papers  (2023-05-15T06:40:56Z)
• CLIP-guided Prototype Modulating for Few-shot Action Recognition [49.11385095278407]
  This work aims to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue. We present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of a video-text contrastive objective and a prototype modulation.
  arXiv  Detail & Related papers  (2023-03-06T09:17:47Z)
• To Adapt or to Fine-tune: A Case Study on Abstractive Summarization [7.353994554197792]
  Recent advances in the field of abstractive summarization leverage pre-trained language models rather than train a model from scratch. Such models are sluggish to train and accompanied by a massive overhead. It remains uncertain whether using adapters benefits the task of summarization, in terms of improved efficiency without an unpleasant sacrifice in performance.
  arXiv  Detail & Related papers  (2022-08-30T22:48:28Z)
• AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks [55.705355299065474]
  Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.
  arXiv  Detail & Related papers  (2022-04-30T16:49:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.