Learning Backward Compatible Embeddings

• URL: http://arxiv.org/abs/2206.03040v1
• Date: Tue, 7 Jun 2022 06:30:34 GMT
• Title: Learning Backward Compatible Embeddings
• Authors: Weihua Hu, Rajas Bansal, Kaidi Cao, Nikhil Rao, Karthik Subbian, Jure Leskovec
• Abstract summary: We study the problem of embedding version updates and their backward compatibility. We develop a solution based on learning backward compatible embeddings. We show that the best method, which we call BC-Aligner, maintains backward compatibility with existing unintended tasks even after multiple model version updates.
• Score: 74.74171220055766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embeddings, low-dimensional vector representation of objects, are fundamental in building modern machine learning systems. In industrial settings, there is usually an embedding team that trains an embedding model to solve intended tasks (e.g., product recommendation). The produced embeddings are then widely consumed by consumer teams to solve their unintended tasks (e.g., fraud detection). However, as the embedding model gets updated and retrained to improve performance on the intended task, the newly-generated embeddings are no longer compatible with the existing consumer models. This means that historical versions of the embeddings can never be retired or all consumer teams have to retrain their models to make them compatible with the latest version of the embeddings, both of which are extremely costly in practice. Here we study the problem of embedding version updates and their backward compatibility. We formalize the problem where the goal is for the embedding team to keep updating the embedding version, while the consumer teams do not have to retrain their models. We develop a solution based on learning backward compatible embeddings, which allows the embedding model version to be updated frequently, while also allowing the latest version of the embedding to be quickly transformed into any backward compatible historical version of it, so that consumer teams do not have to retrain their models. Under our framework, we explore six methods and systematically evaluate them on a real-world recommender system application. We show that the best method, which we call BC-Aligner, maintains backward compatibility with existing unintended tasks even after multiple model version updates. Simultaneously, BC-Aligner achieves the intended task performance similar to the embedding model that is solely optimized for the intended task.

Related papers

• Backward-Compatible Aligned Representations via an Orthogonal Transformation Layer [20.96380700548786]
  Visual retrieval systems face challenges when updating models with improved representations due to misalignment between the old and new representations. Prior research has explored backward-compatible training methods that enable direct comparisons between new and old representations without backfilling. In this paper, we address achieving a balance between backward compatibility and the performance of independently trained models.
  arXiv  Detail & Related papers  (2024-08-16T15:05:28Z)
• MUSCLE: A Model Update Strategy for Compatible LLM Evolution [29.032461144831053]
  Large Language Models (LLMs) are regularly updated to enhance performance. Instance-level degradation (instance regression) of performance from one model version to the next can interfere with a user's mental model of the capabilities of a particular language model. We propose a training strategy to minimize the extent of instance regression in model updates.
  arXiv  Detail & Related papers  (2024-07-12T17:12:48Z)
• Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning [65.57123249246358]
  We propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. We train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. Our prototype complement strategy synthesizes old classes' new features without using any old class instance.
  arXiv  Detail & Related papers  (2024-03-18T17:58:13Z)
• Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval [17.995993499100017]
  Backward-compatible training (BCT) improves the compatibility of two models with less negative impact on retrieval performance. We introduce AdvBCT, an Adversarial Backward-Training method with an elastic boundary constraint. Our method outperforms other BCT methods on both compatibility and discrimination.
  arXiv  Detail & Related papers  (2023-05-04T07:37:07Z)
• Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization [99.6826401545377]
  Foundation models are redefining how AI systems are built. Practitioners now follow a standard procedure to build their machine learning solutions. We propose model ratatouille, a new strategy to recycle the multiple fine-tunings of the same foundation model on diverse auxiliary tasks.
  arXiv  Detail & Related papers  (2022-12-20T17:21:46Z)
• Forward Compatible Few-Shot Class-Incremental Learning [71.2459746681805]
  A machine learning model should recognize new classes without forgetting old ones. Current methods handle incremental learning retrospectively. We propose ForwArd Compatible Training (FACT) for FSCIL.
  arXiv  Detail & Related papers  (2022-03-14T09:36:35Z)
• Forward Compatible Training for Representation Learning [53.300192863727226]
  backward compatible training (BCT) modifies training of the new model to make its representations compatible with those of the old model. BCT can significantly hinder the performance of the new model. In this work, we propose a new learning paradigm for representation learning: forward compatible training (FCT)
  arXiv  Detail & Related papers  (2021-12-06T06:18:54Z)
• Towards Backward-Compatible Representation Learning [86.39292571306395]
  We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning.
  arXiv  Detail & Related papers  (2020-03-26T14:34:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.