Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models

• URL: http://arxiv.org/abs/2607.05533v1
• Date: Mon, 06 Jul 2026 18:14:37 GMT
• Title: Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
• Abstract summary: We propose MuCoDi, a pretraining framework that distills frozen tile embeddings from multiple PFMs into compact edge-oriented encoders.<n>We pretrain students on 14.3M TCGA tiles from only 11.8K WSIs and evaluate frozen encoders on 23 clinically curated downstream classification tasks.
• Score: 1.9573380763700714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computational pathology foundation models (PFMs) have advanced whole-slide image analysis. However, their size and inference cost hinder local deployment in pathology departments. We propose MuCoDi, a pretraining framework that distills frozen tile embeddings from multiple PFMs into compact edge-oriented encoders. Instead of regressing individual teacher features, MuCoDi trains lightweight MobileOne and RepViT students with a contrastive distillation objective adapted from MoCo v3, where cached Virchow2, UNI2, and H-Optimus-1 embeddings replace momentum-encoder keys. We pretrain students on 14.3M TCGA tiles from only 11.8K WSIs and evaluate frozen encoders on 23 clinically curated downstream classification tasks. RepViT-based MuCoEdge students retain near-teacher performance while reducing model size by orders of magnitude: MuCoEdge-R2.3 and MuCoEdge-R1.5 reach 71.0% external AUROC, within 0.8 percentage points of the best teacher (Virchow2, 71.8%), while MuCoEdge-R2.3 obtains the best external F1 and the second-best AUPRC (51.8% and 53.3%). MuCoEdge-R1.0 reaches 70.9% AUROC with only 6.4M parameters and 1.12 GFLOPs. On a Raspberry Pi 5, sub-million-parameter MobileOne students achieve up to 605-fold single-tile speedup over Virchow2 while retaining 66.5% to 66.9% external AUROC, demonstrating that PFM-quality pathology representations can be moved toward practical edge deployment. Code is available at https://anonymous.4open.science/r/mucodi-6243.

Related papers

• Distilling Foundation Models for Agentic What-If Reasoning:Cost, Latency, and Governance in a Hybrid LLM+SLM Architecture [4.205733518246009]
  Tabular foundation models deliver strong zero-training predictive performance via in-context learning, but their high inference latency makes them impractical as hot-path decision backends in interactive agentic loops.<n>We distill a TabPFN teacher into a compact feed-forward student across a business-decision simulation on UCI Adult and five OpenML benchmarks.<n>The student retains 95.4-100.5% accuracy and 96.8-100.0% AUC, with the lowest accuracy retention on credit-g at 95.4%.
  arXiv  Detail & Related papers  (2026-09-14T11:26:21Z)
• Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification [10.122947222129108]
  Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment.<n>We study cross-architecture knowledge distillation from a fine-tuned DINOv2 teacher to a compact bidirectional Visual State Space Model.<n>We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data.
  arXiv  Detail & Related papers  (2026-08-27T08:02:05Z)
• Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding [1.2132884030603328]
  We introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B for offline, edge-deployable vision-language retrieval.<n>Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and low latency is essential.<n>The released model contains 1,926,188,032 parameters (3.85 GB bf16), representing approximately 9.5% fewer parameters than the 2.13B teacher model.
  arXiv  Detail & Related papers  (2026-07-15T04:36:19Z)
• Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification [5.584060970507507]
  We present a benchmark jointly evaluating federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification.<n>It spans 13 FL algorithms and 10 KD objectives (a 130-pair cross-product) across 504 training runs, evaluated on ModelNet40 and a clinical craniosynostosis dataset.
  arXiv  Detail & Related papers  (2026-06-30T20:12:27Z)
• Single-Channel Tissue Segmentation via Cross-Modal Distillation from Foundation Models [0.0]
  Multiplexed fluorescence microscopy improves tissue segmentation by providing complementary channels including nuclear (DAPI) and membrane (E-cadherin)<n>This work proposes a cross-modal knowledge distillation framework that transfers semantic information from a frozen foundation model teacher processing multiplexed input to a lightweight student operating on the nuclear channel only.
  arXiv  Detail & Related papers  (2026-05-30T23:34:49Z)
• Less is More in Semantic Space: Intrinsic Decoupling via Clifford-M for Fundus Image Classification [13.23226865033351]
  We propose Clifford-M, a lightweight backbone that replaces feed-forward expansion and frequency-splitting modules with sparse geometric interaction.<n>Without pre-training, Clifford-M achieves a mean AUC-ROC of 0.8142 and a mean macro-F1 of 0.5481 on ODIR-5K using only 0.85M parameters.
  arXiv  Detail & Related papers  (2026-03-21T13:00:05Z)
• STEP3-VL-10B Technical Report [115.89015065130127]
  STEP3-VL-10B is a lightweight foundation model designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence.<n>We implement Parallel Coordinated Reasoning (PaCoRe) to scale test-time compute, allocating resources to scalable perceptual reasoning.<n>It records 92.2% on MMBench and 80.11% on MMMU, while excelling in complex reasoning with 94.43% on AIME2025 and 75.95% on MathVision.
  arXiv  Detail & Related papers  (2026-01-14T17:58:24Z)
• MI-to-Mid Distilled Compression (M2M-DC): An Hybrid-Information-Guided-Block Pruning with Progressive Inner Slicing Approach to Model Compression [2.351601888896043]
  We introduce MI-to-Mid Distilled Compression (M2M-DC), a two-scale, shape-safe compression framework.<n>M2M-DC ranks residual blocks by a label-aware mutual information signal.<n>It then alternates short KD phases with stage-coherent, residual-safe channel slicing.
  arXiv  Detail & Related papers  (2025-11-10T08:36:29Z)
• CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework [1.2172320168050466]
  We introduce Consensus-oriented Masked Distillation (CoMAD)<n>It unifies knowledge from self-supervised Vision Transformers into a compact student network.<n>On ImageNet-1K, CoMAD's ViT-Tiny achieves 75.4 percent Top-1, an increment of 0.4 percent over the previous state-of-the-art.
  arXiv  Detail & Related papers  (2025-08-06T18:55:14Z)
• CoMP: Continual Multimodal Pre-training for Vision Foundation Models [72.3323674291719]
  We continually pre-train prevailing Vision Foundation Models (VFMs) in a multimodal manner.<n>We introduce CoMP, a carefully designed multimodal pre-training pipeline.<n>Leading VFMs like DINOv2, SigLIP and AIMv2 achieve remarkable improvements in multimodal understanding tasks.
  arXiv  Detail & Related papers  (2025-03-24T17:52:47Z)
• TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance [97.01406871579525]
  We propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. We show that TinyCLIP can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. Our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet.
  arXiv  Detail & Related papers  (2023-09-21T17:59:53Z)
• EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm [111.17100512647619]
  This paper explains the rationality of Vision Transformer by analogy with the proven practical evolutionary algorithm (EA) We propose a novel pyramid EATFormer backbone that only contains the proposed EA-based transformer (EAT) block. Massive quantitative and quantitative experiments on image classification, downstream tasks, and explanatory experiments demonstrate the effectiveness and superiority of our approach.
  arXiv  Detail & Related papers  (2022-06-19T04:49:35Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• UniFormer: Unifying Convolution and Self-attention for Visual Recognition [69.68907941116127]
  Convolution neural networks (CNNs) and vision transformers (ViTs) have been two dominant frameworks in the past few years. We propose a novel Unified transFormer (UniFormer) which seamlessly integrates the merits of convolution and self-attention in a concise transformer format. Our UniFormer achieves 86.3 top-1 accuracy on ImageNet-1K classification.
  arXiv  Detail & Related papers  (2022-01-24T04:39:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.