Knowledge-Decoupled Synergetic Learning: An MLLM based Collaborative Approach to Few-shot Multimodal Dialogue Intention Recognition

• URL: http://arxiv.org/abs/2503.04201v1
• Date: Thu, 06 Mar 2025 08:28:44 GMT
• Title: Knowledge-Decoupled Synergetic Learning: An MLLM based Collaborative Approach to Few-shot Multimodal Dialogue Intention Recognition
• Authors: Bin Chen, Yu Zhang, Hongfei Ye, Ziyi Huang, Hongyang Chen,
• Abstract summary: Training for few-shot multimodal dialogue intention recognition involves two interconnected tasks.<n>This phenomenon is attributed to knowledge interference stemming from the superposition of weight matrix updates during the training process.<n>We propose Knowledge-Decoupled Synergetic Learning, which transforms knowledge into interpretable rules, while applying the post-training of larger models.
• Score: 17.790383360652704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Few-shot multimodal dialogue intention recognition is a critical challenge in the e-commerce domainn. Previous methods have primarily enhanced model classification capabilities through post-training techniques. However, our analysis reveals that training for few-shot multimodal dialogue intention recognition involves two interconnected tasks, leading to a seesaw effect in multi-task learning. This phenomenon is attributed to knowledge interference stemming from the superposition of weight matrix updates during the training process. To address these challenges, we propose Knowledge-Decoupled Synergetic Learning (KDSL), which mitigates these issues by utilizing smaller models to transform knowledge into interpretable rules, while applying the post-training of larger models. By facilitating collaboration between the large and small multimodal large language models for prediction, our approach demonstrates significant improvements. Notably, we achieve outstanding results on two real Taobao datasets, with enhancements of 6.37\% and 6.28\% in online weighted F1 scores compared to the state-of-the-art method, thereby validating the efficacy of our framework.

Related papers

• Platonic Grounding for Efficient Multimodal Language Models [22.715168904364756]
  We motivate and propose a simple modification to existing multimodal frameworks that rely on aligning pretrained models. Our work also has implications for combining pretrained models into larger systems efficiently.
  arXiv  Detail & Related papers  (2025-04-27T18:56:26Z)
• Multi-Modal Self-Supervised Semantic Communication [52.76990720898666]
  We propose a multi-modal semantic communication system that leverages multi-modal self-supervised learning to enhance task-agnostic feature extraction. The proposed approach effectively captures both modality-invariant and modality-specific features while minimizing training-related communication overhead. The findings underscore the advantages of multi-modal self-supervised learning in semantic communication, paving the way for more efficient and scalable edge inference systems.
  arXiv  Detail & Related papers  (2025-03-18T06:13:02Z)
• Enhancing Multi-hop Reasoning in Vision-Language Models via Self-Distillation with Multi-Prompt Ensembling [11.462550020102935]
  Multi-modal large language models can leverage chain-of-thought prompting for zero or few-shot learning.<n>Similar prompting strategies are less effective for multi-modal LLMs due to modality gaps and task complexity.<n>We propose a self-distillation framework such that the model can improve itself without any annotated data.
  arXiv  Detail & Related papers  (2025-03-03T17:24:42Z)
• Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning [79.46570165281084]
  We propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks.
  arXiv  Detail & Related papers  (2024-11-11T07:36:19Z)
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training [55.54020926284334]
  Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. Retrieval augmentation techniques have proven to be effective plugins for both LLMs and MLLMs. In this study, we propose multimodal adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training (RA-BLIP), a novel retrieval-augmented framework for various MLLMs.
  arXiv  Detail & Related papers  (2024-10-18T03:45:19Z)
• Efficient Multi-Model Fusion with Adversarial Complementary Representation Learning [26.393644289860084]
  Single-model systems often suffer from deficiencies in tasks such as speaker verification (SV) and image classification. We propose an adversarial complementary representation learning (ACoRL) framework that enables newly trained models to avoid previously acquired knowledge.
  arXiv  Detail & Related papers  (2024-04-24T07:47:55Z)
• Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning [49.92517970237088]
  We tackle the problem of training a robot to understand multimodal prompts. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. We introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts.
  arXiv  Detail & Related papers  (2023-10-14T22:24:58Z)
• Derivative Free Weight-space Ensembling [0.0]
  We introduce Derivative Free Weight-space Ensembling (DFWE), a new few-sample task transfer approach for open-domain dialogue. We finetune each of the expert models on the target task, approaching the target task from several distinct knowledge bases. We linearly interpolate between the model weights using a gradient-free-optimization algorithm, to efficiently find a good weighting.
  arXiv  Detail & Related papers  (2023-07-07T10:42:44Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Pre-training Multi-party Dialogue Models with Latent Discourse Inference [85.9683181507206]
  We pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model.
  arXiv  Detail & Related papers  (2023-05-24T14:06:27Z)
• Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis [19.07020276666615]
  We propose a novel framework named MultiModal Contrastive Learning (MMCL) for multimodal representation to capture intra- and inter-modality dynamics simultaneously. We also design two contrastive learning tasks, instance- and sentiment-based contrastive learning, to promote the process of prediction and learn more interactive information related to sentiment.
  arXiv  Detail & Related papers  (2022-10-26T08:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.