Reverse Distillation: Consistently Scaling Protein Language Model Representations

• URL: http://arxiv.org/abs/2603.07710v1
• Date: Sun, 08 Mar 2026 16:24:05 GMT
• Title: Reverse Distillation: Consistently Scaling Protein Language Model Representations
• Abstract summary: We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into subspaces guided by smaller models of the same family.<n>A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features.<n>On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance.
• Score: 0.6066775336291516
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first k dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.

Related papers

• Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models [55.908141398092646]
  Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs.<n>We study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in the smaller models.<n>We show that small models such as $textttGPT2$ and $textttQwen3-0.6B$ exhibit severe condensation, whereas the larger models such as $textttGPT2-xl$ and $textttQwen3-32B
  arXiv  Detail & Related papers  (2026-01-30T16:07:03Z)
• Towards Reversible Model Merging For Low-rank Weights [5.100622189286672]
  Model merging aims to combine multiple fine-tuned models into a single set of weights that performs well across all source tasks.<n>We show that applying conventional merging methods to low-rank weights leads to severe performance degradation in the merged model.<n>We propose a fundamentally different approach: instead of collapsing all adapters into one set of weights, we construct a compact basis.<n>This reframes merging as generating a reconstruction-capable model space rather than producing a single merged model.
  arXiv  Detail & Related papers  (2025-10-15T23:22:38Z)
• Boomerang Distillation Enables Zero-Shot Model Size Interpolation [30.74896485280105]
  Existing approaches build model families by training each size independently, which is expensive and provides only coarse-grained size options.<n>In this work, we identify a novel phenomenon that we call boomerang distillation.<n>One first distills down to a small student and then progressively reconstructs intermediate-sized models by re-incorporating blocks of teacher layers into the student.
  arXiv  Detail & Related papers  (2025-10-06T17:41:20Z)
• SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery [54.866490321241905]
  Model merging-based multitask learning (MTL) offers a promising approach for performing MTL by merging multiple expert models. In this paper, we examine the merged model's representation distribution and uncover a critical issue of "representation bias" This bias arises from a significant distribution gap between the representations of the merged and expert models, leading to the suboptimal performance of the merged MTL model.
  arXiv  Detail & Related papers  (2024-10-18T11:49:40Z)
• PLeaS -- Merging Models with Permutations and Least Squares [43.17620198572947]
  We propose a new two-step algorithm to merge models -- termed PLeaS -- which relaxes constraints.<n>PLeaS partially matches nodes in each layer by maximizing alignment.<n>We also demonstrate how our method can be extended to address a challenging scenario where no data is available from the fine-tuning domains.
  arXiv  Detail & Related papers  (2024-07-02T17:24:04Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• When Do We Not Need Larger Vision Models? [55.957626371697785]
  Scaling up the size of vision models has been the de facto standard to obtain more powerful visual representations. We demonstrate the power of Scaling on Scales (S$2$), whereby a pre-trained and frozen smaller vision model can outperform larger models. We release a Python package that can apply S$2$ on any vision model with one line of code.
  arXiv  Detail & Related papers  (2024-03-19T17:58:39Z)
• Training Trajectories of Language Models Across Scales [99.38721327771208]
  Scaling up language models has led to unprecedented performance gains. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors?
  arXiv  Detail & Related papers  (2022-12-19T19:16:29Z)
• When Ensembling Smaller Models is More Efficient than Single Large Models [52.38997176317532]
  We show that ensembles can outperform single models with both higher accuracy and requiring fewer total FLOPs to compute. This presents an interesting observation that output diversity in ensembling can often be more efficient than training larger models.
  arXiv  Detail & Related papers  (2020-05-01T18:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.