Darwinian Model Upgrades: Model Evolving with Selective Compatibility

• URL: http://arxiv.org/abs/2210.06954v1
• Date: Thu, 13 Oct 2022 12:28:48 GMT
• Title: Darwinian Model Upgrades: Model Evolving with Selective Compatibility
• Authors: Binjie Zhang, Shupeng Su, Yixiao Ge, Xuyuan Xu, Yexin Wang, Chun Yuan, Mike Zheng Shou, Ying Shan
• Abstract summary: BCT presents the first step towards backward-compatible model upgrades to get rid of backfilling. We propose Darwinian Model Upgrades (DMU) which disentangle the inheritance and variation in the model evolving with selective backward compatibility and forward adaptation. DMU effectively alleviates the new-to-new degradation and improves new-to-old compatibility, rendering a more proper model upgrading paradigm in large-scale retrieval systems.
• Score: 29.920204547961696
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The traditional model upgrading paradigm for retrieval requires recomputing all gallery embeddings before deploying the new model (dubbed as "backfilling"), which is quite expensive and time-consuming considering billions of instances in industrial applications. BCT presents the first step towards backward-compatible model upgrades to get rid of backfilling. It is workable but leaves the new model in a dilemma between new feature discriminativeness and new-to-old compatibility due to the undifferentiated compatibility constraints. In this work, we propose Darwinian Model Upgrades (DMU), which disentangle the inheritance and variation in the model evolving with selective backward compatibility and forward adaptation, respectively. The old-to-new heritable knowledge is measured by old feature discriminativeness, and the gallery features, especially those of poor quality, are evolved in a lightweight manner to become more adaptive in the new latent space. We demonstrate the superiority of DMU through comprehensive experiments on large-scale landmark retrieval and face recognition benchmarks. DMU effectively alleviates the new-to-new degradation and improves new-to-old compatibility, rendering a more proper model upgrading paradigm in large-scale retrieval systems.

Related papers

• Exploring Model Kinship for Merging Large Language Models [52.01652098827454]
  We introduce model kinship, the degree of similarity or relatedness between Large Language Models. We find that there is a certain relationship between model kinship and the performance gains after model merging. We propose a new model merging strategy: Top-k Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-16T14:29:29Z)
• Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer [20.96380700548786]
  Visual retrieval systems face challenges when updating models with improved representations due to misalignment between the old and new representations. Prior research has explored backward-compatible training methods that enable direct comparisons between new and old representations without backfilling. In this paper, we address achieving a balance between backward compatibility and the performance of independently trained models.
  arXiv  Detail & Related papers  (2024-08-16T15:05:28Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• MixBCT: Towards Self-Adapting Backward-Compatible Training [66.52766344751635]
  We propose MixBCT, a simple yet highly effective backward-compatible training method. We conduct experiments on the large-scale face recognition datasets MS1Mv3 and IJB-C.
  arXiv  Detail & Related papers  (2023-08-14T05:55:38Z)
• Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion [8.173078054056337]
  When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression errors. We propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models.
  arXiv  Detail & Related papers  (2023-02-04T03:40:35Z)
• Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval [28.268764435617975]
  We propose a new model upgrade paradigm, termed Bidirectional Compatible Training (BiCT) BiCT upgrades the old gallery embeddings by forward-compatible training towards the embedding space of the backward-compatible new model. We conduct comprehensive experiments to verify the prominent improvement by BiCT and observe that the inconspicuous loss weight of backward compatibility actually plays an essential role for both backward and forward retrieval performance.
  arXiv  Detail & Related papers  (2022-04-29T07:38:09Z)
• Forward Compatible Training for Representation Learning [53.300192863727226]
  backward compatible training (BCT) modifies training of the new model to make its representations compatible with those of the old model. BCT can significantly hinder the performance of the new model. In this work, we propose a new learning paradigm for representation learning: forward compatible training (FCT)
  arXiv  Detail & Related papers  (2021-12-06T06:18:54Z)
• Neighborhood Consensus Contrastive Learning for Backward-Compatible Representation [46.86784621137665]
  backward-compatible representation is proposed to enable the "new" features compatible with "old"' features. We propose a Neighborhood Consensus Contrastive Learning (NCCL) method, which learns backward-compatible representation from a neighborhood consensus perspective. Our method ensures backward compatibility without impairing the accuracy of the new model.
  arXiv  Detail & Related papers  (2021-08-07T05:50:47Z)
• Towards Backward-Compatible Representation Learning [86.39292571306395]
  We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning.
  arXiv  Detail & Related papers  (2020-03-26T14:34:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.