Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers

• URL: http://arxiv.org/abs/2608.08809v1
• Date: Sun, 09 Aug 2026 16:45:28 GMT
• Title: Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers
• Abstract summary: A transformer-based model can be made smaller in three ways.<n>A single object names any size the model can run at, and a short schedule lists the sizes to train.<n>The same abstraction covers both retrievers and rerankers and both encoder and decoder models.
• Score: 71.99661189378442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A single model scale challenges the flexibility of a production retrieval system: some settings need it faster, others need a smaller index, and the right trade-off changes with the workload. In the context of information retrieval (IR), a transformer-based model can be made smaller in three ways---using fewer layers, passing fewer tokens through the upper layers, or producing a shorter embedding---and each way saves a different compute resource. These options have been studied one at a time, each as its own method with its own code and training setup, which makes them hard to combine or adapt to a new model. We present~\ours to bring all three under one simple abstraction: a single object names any size the model can run at, and a short schedule lists the sizes to train. Training then produces one checkpoint that serves all of those sizes, and at deployment the user picks any of them. The same abstraction covers both retrievers and rerankers and both encoder and decoder models, as it works through interfaces that Hugging Face transformers already expose; a new backbone is a configuration change, not new modeling code. Prior methods---Matryoshka embeddings, early exit, 2D~Matryoshka (e.g., Starbucks), and layerwise token compression---become special cases of our unified abstraction. The same interface also enables Matryoshka~LTC (MLTC), which jointly trains several token-compression ratios in one retriever checkpoint. To validate our framework, we train 20 checkpoints across three backbones and two tasks: the quality curves are smooth, one checkpoint costs little over a model trained for a single size, and a controlled study confirms the wallclock speedups. We release the framework and all checkpoints as a resource for building elastic retrieval systems.

Related papers

• Update Your Transformer to the Latest Release: Re-Basin of Task Vectors [27.63078324151366]
  Foundation models serve as the backbone for numerous specialized models developed through fine-tuning.<n>When the underlying pretrained model is updated or retrained, the fine-tuned model becomes obsolete.<n>This raises the question: is it possible to transfer fine-tuning to a new release of the model?<n>In this work, we investigate how to transfer fine-tuning to a new checkpoint without having to re-train, in a data-free manner.
  arXiv  Detail & Related papers  (2025-05-28T13:55:12Z)
• POA: Pre-training Once for Models of All Sizes [33.72644336390202]
  We propose a novel tri-branch self-supervised training framework, termed as POA (Pre-training Once for All) Our approach introduces an innovative elastic student branch into a modern self-distillation paradigm. It achieves state-of-the-art performance using ViT, Swin Transformer and ResNet backbones.
  arXiv  Detail & Related papers  (2024-08-02T06:13:29Z)
• CerberusDet: Unified Multi-Dataset Object Detection [0.0]
  CerberusDet is a framework with a multi-headed model designed for handling multiple object detection tasks. Proposed model is built on the YOLO architecture and efficiently shares visual features from both backbone and neck components. CerberusDet achieved state-of-the-art results with 36% less inference time.
  arXiv  Detail & Related papers  (2024-07-17T15:00:35Z)
• MatFormer: Nested Transformer for Elastic Inference [91.45687988953435]
  MatFormer is a novel Transformer architecture designed to provide elastic inference across diverse deployment constraints.<n>MatFormer achieves this by incorporating a nested Feed Forward Network (FFN) block structure within a standard Transformer model.<n>We show that a 850M decoder-only MatFormer language model (MatLM) allows us to extract multiple smaller models spanning from 582M to 850M parameters.
  arXiv  Detail & Related papers  (2023-10-11T17:57:14Z)
• $\Delta$-Patching: A Framework for Rapid Adaptation of Pre-trained Convolutional Networks without Base Performance Loss [71.46601663956521]
  Models pre-trained on large-scale datasets are often fine-tuned to support newer tasks and datasets that arrive over time. We propose $Delta$-Patching for fine-tuning neural network models in an efficient manner, without the need to store model copies. Our experiments show that $Delta$-Networks outperform earlier model patching work while only requiring a fraction of parameters to be trained.
  arXiv  Detail & Related papers  (2023-03-26T16:39:44Z)
• AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers [94.11915008006483]
  We present a new method that reformulates point cloud completion as a set-to-set translation problem. We design a new model, called PoinTr, which adopts a Transformer encoder-decoder architecture for point cloud completion. Our method attains 6.53 CD on PCN, 0.81 CD on ShapeNet-55 and 0.392 MMD on real-world KITTI.
  arXiv  Detail & Related papers  (2023-01-11T16:14:12Z)
• Cross-Modal Adapter for Vision-Language Retrieval [60.59577149733934]
  We present a novel Cross-Modal Adapter for parameter-efficient transfer learning.<n>Inspired by adapter-based methods, we adjust the pre-trained model with a few parameterization layers.<n>Our approach has three notable benefits: (1) reduces the vast majority of fine-tuned parameters, (2) saves training time, and (3) allows all the pre-trained parameters to be fixed.
  arXiv  Detail & Related papers  (2022-11-17T16:15:30Z)
• AdaMix: Mixture-of-Adapter for Parameter-efficient Tuning of Large Language Models [119.7093605087114]
  Fine-tuning large-scale pre-trained language models to downstream tasks require updating hundreds of millions of parameters. This not only increases the serving cost to store a large copy of the model weights for every task, but also exhibits instability during few-shot task adaptation. We introduce a new mechanism to improve adapter capacity without increasing parameters or computational cost by two key techniques.
  arXiv  Detail & Related papers  (2022-05-24T23:41:22Z)
• Structural Dropout for Model Width Compression [1.52292571922932]
  Existing ML models are known to be highly over-parametrized, and use significantly more resources than required for a given task. We propose a method that requires only a single training session for the original model and a set of compressed models. The proposed approach is a "structural" dropout that prunes all elements in the hidden state above a randomly chosen index, forcing the model to learn an importance ordering over its features.
  arXiv  Detail & Related papers  (2022-05-13T21:50:57Z)
• A Fast Post-Training Pruning Framework for Transformers [74.59556951906468]
  Pruning is an effective way to reduce the huge inference cost of large Transformer models. Prior work on model pruning requires retraining the model. We propose a fast post-training pruning framework for Transformers that does not require any retraining.
  arXiv  Detail & Related papers  (2022-03-29T07:41:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.