$BT^2$: Backward-compatible Training with Basis Transformation

• URL: http://arxiv.org/abs/2211.03989v3
• Date: Mon, 28 Aug 2023 05:56:47 GMT
• Title: $BT^2$: Backward-compatible Training with Basis Transformation
• Authors: Yifei Zhou, Zilu Li, Abhinav Shrivastava, Hengshuang Zhao, Antonio Torralba, Taipeng Tian, Ser-Nam Lim
• Abstract summary: Retrieval system often requires recomputing the representation of every piece of data in the gallery when updating to a better representation model. This process is known as backfilling and can be especially costly in the real world where the gallery often contains billions of samples. Recently, researchers have proposed the idea of Backward compatible Training (BCT) where the new representation model can be trained with an auxiliary loss to make it backward compatible with the old representation.
• Score: 107.37014712361788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern retrieval system often requires recomputing the representation of every piece of data in the gallery when updating to a better representation model. This process is known as backfilling and can be especially costly in the real world where the gallery often contains billions of samples. Recently, researchers have proposed the idea of Backward Compatible Training (BCT) where the new representation model can be trained with an auxiliary loss to make it backward compatible with the old representation. In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling. However, followup work shows that there is an inherent tradeoff where a backward compatible representation model cannot simultaneously maintain the performance of the new model itself. This paper reports our ``not-so-surprising'' finding that adding extra dimensions to the representation can help here. However, we also found that naively increasing the dimension of the representation did not work. To deal with this, we propose Backward-compatible Training with a novel Basis Transformation ($BT^2$). A basis transformation (BT) is basically a learnable set of parameters that applies an orthonormal transformation. Such a transformation possesses an important property whereby the original information contained in its input is retained in its output. We show in this paper how a BT can be utilized to add only the necessary amount of additional dimensions. We empirically verify the advantage of $BT^2$ over other state-of-the-art methods in a wide range of settings. We then further extend $BT^2$ to other challenging yet more practical settings, including significant change in model architecture (CNN to Transformers), modality change, and even a series of updates in the model architecture mimicking the evolution of deep learning models.

Related papers

• Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer [20.96380700548786]
  Visual retrieval systems face challenges when updating models with improved representations due to misalignment between the old and new representations. Prior research has explored backward-compatible training methods that enable direct comparisons between new and old representations without backfilling. In this paper, we address achieving a balance between backward compatibility and the performance of independently trained models.
  arXiv  Detail & Related papers  (2024-08-16T15:05:28Z)
• Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements [20.96380700548786]
  Learning compatible representations enables the interchangeable use of semantic features as models are updated over time. This is particularly relevant in search and retrieval systems where it is crucial to avoid reprocessing of the gallery images with the updated model. We show that the stationary representations learned by the $d$-Simplex fixed classifier optimally approximate compatibility representation according to the two inequality constraints of its formal definition.
  arXiv  Detail & Related papers  (2024-05-04T06:31:38Z)
• ReFT: Representation Finetuning for Language Models [74.51093640257892]
  We develop a family of Representation Finetuning (ReFT) methods. ReFTs operate on a frozen base model and learn task-specific interventions on hidden representations. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE.
  arXiv  Detail & Related papers  (2024-04-04T17:00:37Z)
• FastFill: Efficient Compatible Model Update [40.27741553705222]
  FastFill is a compatible model update process using feature alignment and policy based partial backfilling. We show that previous backfilling strategies suffer from decreased performance and demonstrate the importance of both the training objective and the ordering in online partial backfilling.
  arXiv  Detail & Related papers  (2023-03-08T18:03:51Z)
• Online Backfilling with No Regret for Large-Scale Image Retrieval [50.162438586686356]
  Backfilling is the process of re-extracting all gallery embeddings from upgraded models in image retrieval systems. We propose an online backfilling algorithm, which enables us to achieve a progressive performance improvement during the backfilling process. We incorporate a reverse transformation module for more effective and efficient merging, which is further enhanced by adopting a metric-compatible contrastive learning approach.
  arXiv  Detail & Related papers  (2023-01-10T03:10:32Z)
• Towards Universal Backward-Compatible Representation Learning [29.77801805854168]
  backward-compatible representation learning is introduced to support backfill-free model upgrades. We first introduce a new problem of universal backward-compatible representation learning, covering all possible data split in model upgrades. We propose a simple yet effective method, dubbed Universal Backward- Training (UniBCT) with a novel structural prototype refinement algorithm.
  arXiv  Detail & Related papers  (2022-03-03T09:23:51Z)
• Forward Compatible Training for Representation Learning [53.300192863727226]
  backward compatible training (BCT) modifies training of the new model to make its representations compatible with those of the old model. BCT can significantly hinder the performance of the new model. In this work, we propose a new learning paradigm for representation learning: forward compatible training (FCT)
  arXiv  Detail & Related papers  (2021-12-06T06:18:54Z)
• Visformer: The Vision-friendly Transformer [105.52122194322592]
  We propose a new architecture named Visformer, which is abbreviated from the Vision-friendly Transformer' With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy.
  arXiv  Detail & Related papers  (2021-04-26T13:13:03Z)
• Towards Backward-Compatible Representation Learning [86.39292571306395]
  We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning.
  arXiv  Detail & Related papers  (2020-03-26T14:34:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.