Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility

• URL: http://arxiv.org/abs/2609.16875v1
• Date: Tue, 15 Sep 2026 09:06:00 GMT
• Title: Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility
• Abstract summary: We introduce Multi-modal Knowledge Preserving Adapter (MKP-Adapter), the first adapter-only BCT approach for Multi-modal Large Language Models (MLLMs)<n>MKP-Adapter is trained solely on pre-extracted embeddings and requires only negligible additional latency relative to the original backbone forward pass.
• Score: 13.870015478036906
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Upgrading embedding models typically requires expensive database re-indexing, as new query embeddings are incompatible with existing database embeddings. While Backward Compatible Training (BCT) mitigates this by enforcing compatibility during training, existing approaches often require updating the backbone model. This is impractical because of significant training cost, the risk of performance regression, and limited access to proprietary model weights. We introduce Multi-modal Knowledge Preserving Adapter (MKP-Adapter), the first adapter-only BCT approach for Multi-modal Large Language Models (MLLMs) that requires no backbone updates. We identified that the primary challenge in adapter-only BCT is preserving the knowledge of the new embeddings while enforcing backward compatibility. Hence, we propose a multi-level preservation loss that maintains the geometric structure of the embedding spaces throughout BCT. Furthermore, a focal re-weighting strategy is integrated to prioritize learning from challenging samples. Experiments demonstrate that our method achieves strong backward compatibility across diverse multi-modal benchmarks (image, text, visual document, and video retrieval tasks) and model types. Notably, MKP-Adapter is trained solely on pre-extracted embeddings and requires only negligible additional latency relative to the original backbone forward pass, highlighting its efficiency.

Related papers

• Don't Start Over: A Cost-Effective Framework for Migrating Personalized Prompts Between LLMs [51.79252689855809]
  Personalization in Large Language Models (LLMs) often relies on user-specific soft prompts.<n>We propose the Prompt-level User Migration Adapter (PUMA), a framework to efficiently migrate personalized prompts across incompatible models.<n>Experiments on three large-scale datasets show our method matches or even surpasses the performance of retraining from scratch, reducing computational cost by up to 98%.
  arXiv  Detail & Related papers  (2026-01-17T12:30:31Z)
• BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning [84.56022893225422]
  Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge.<n>Applying vision-language models such as CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities.
  arXiv  Detail & Related papers  (2025-11-14T15:51:40Z)
• DATA: Decomposed Attention-based Task Adaptation for Rehearsal-Free Continual Learning [22.386864304549285]
  Continual learning (CL) is essential for Large Language Models (LLMs) to adapt to evolving real-world demands.<n>Recent rehearsal-free methods employ model-based and regularization-based strategies to address this issue.<n>We propose a $textbfD$e $textbfA$ttention-based $textbfTask $textbfA$daptation ( DATA)<n> DATA explicitly decouples and learns both task-specific and task-shared knowledge using high-rank and low-rank task adapters.
  arXiv  Detail & Related papers  (2025-02-17T06:35:42Z)
• Towards Compatible Fine-tuning for Vision-Language Model Updates [114.25776195225494]
  Class-conditioned Context Optimization (ContCoOp) integrates learnable prompts with class embeddings using an attention layer before inputting them into the text encoder.<n>Our experiments over 15 datasets show that our ContCoOp achieves the highest compatibility over the baseline methods, and exhibits robust out-of-distribution generalization.
  arXiv  Detail & Related papers  (2024-12-30T12:06:27Z)
• Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models [53.25790521873174]
  Backward-compatible Training (BT) has been proposed to ensure that the new model aligns with the old model's embeddings.<n>This paper extends the concept of vision-only BT to the field of cross-modal retrieval.<n>We propose a projection module that maps the new model's embeddings to those of the old model.
  arXiv  Detail & Related papers  (2024-05-23T15:46:35Z)
• MixBCT: Towards Self-Adapting Backward-Compatible Training [66.52766344751635]
  We propose MixBCT, a simple yet highly effective backward-compatible training method. We conduct experiments on the large-scale face recognition datasets MS1Mv3 and IJB-C.
  arXiv  Detail & Related papers  (2023-08-14T05:55:38Z)
• TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter [21.41170708560114]
  A growing number of applications based on visual foundation models are emerging. In situations involving system upgrades, it becomes essential to re-train all downstream modules to adapt to the new foundation model. We introduce a parameter-efficient and task-agnostic adapter, dubbed TaCA, that facilitates compatibility across distinct foundation models.
  arXiv  Detail & Related papers  (2023-06-22T03:00:24Z)
• Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval [17.995993499100017]
  Backward-compatible training (BCT) improves the compatibility of two models with less negative impact on retrieval performance. We introduce AdvBCT, an Adversarial Backward-Training method with an elastic boundary constraint. Our method outperforms other BCT methods on both compatibility and discrimination.
  arXiv  Detail & Related papers  (2023-05-04T07:37:07Z)
• Forward Compatible Training for Representation Learning [53.300192863727226]
  backward compatible training (BCT) modifies training of the new model to make its representations compatible with those of the old model. BCT can significantly hinder the performance of the new model. In this work, we propose a new learning paradigm for representation learning: forward compatible training (FCT)
  arXiv  Detail & Related papers  (2021-12-06T06:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.