Foundry: Distilling 3D Foundation Models for the Edge

• URL: http://arxiv.org/abs/2511.20721v1
• Date: Tue, 25 Nov 2025 07:53:56 GMT
• Title: Foundry: Distilling 3D Foundation Models for the Edge
• Abstract summary: Foundation Model Distillation (FMD) is a new paradigm for compressing large SSL models into compact proxies.<n>We present Foundry, the first implementation of FMD for 3D point clouds.
• Score: 15.177160184417575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation models pre-trained with self-supervised learning (SSL) on large-scale datasets have become powerful general-purpose feature extractors. However, their immense size and computational cost make them prohibitive for deployment on edge devices such as robots and AR/VR headsets. Existing compression techniques like standard knowledge distillation create efficient 'specialist' models but sacrifice the crucial, downstream-agnostic generality that makes foundation models so valuable. In this paper, we introduce Foundation Model Distillation (FMD), a new paradigm for compressing large SSL models into compact, efficient, and faithful proxies that retain their general-purpose representational power. We present Foundry, the first implementation of FMD for 3D point clouds. Our approach, Foundry, trains a student to learn a compressed set of SuperTokens that reconstruct the teacher's token-level representations, capturing a compact basis of its latent space. A single distilled model maintains strong transferability across diverse downstream tasks-classification, part segmentation, and few-shot scenarios-approaching full foundation-model performance while using significantly fewer tokens and FLOPs, making such models more practical for deployment on resourceconstrained hardware.

Related papers

• Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction [0.13999481573773068]
  Large 3D foundation models such as MASt3R achieve state-of-the-art stereo reconstruction but are computationally demanding for deployment under strict hardware constraints.<n>We study how far such models can be compressed through knowledge distillation, using lunar stereo reconstruction as a challenging and practically relevant case study.
  arXiv  Detail & Related papers  (2026-07-02T08:11:32Z)
• S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models [24.103531000455003]
  S-SONDO is the first framework to distill general audio models using only their output embeddings.<n>We demonstrate its effectiveness by distilling two audio foundation models into three efficient students.
  arXiv  Detail & Related papers  (2026-04-27T19:20:47Z)
• Transport and Merge: Cross-Architecture Merging for Large Language Models [59.53629883788284]
  Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data.<n>Many real-world deployments rely on smaller models trained or adapted from low-resource data.<n>This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets.
  arXiv  Detail & Related papers  (2026-02-05T09:57:57Z)
• SnapGen++: Unleashing Diffusion Transformers for Efficient High-Fidelity Image Generation on Edge Devices [72.0937240883345]
  Recent advances in diffusion transformers (DiTs) have set new standards in image generation, yet remain impractical for on-device deployment.<n>We present an efficient DiT framework tailored for mobile and edge devices that achieves transformer-level generation quality under strict resource constraints.
  arXiv  Detail & Related papers  (2026-01-13T07:46:46Z)
• TAP-CT: 3D Task-Agnostic Pretraining of Computed Tomography Foundation Models [39.00742360251856]
  Existing foundation models (FMs) in the medical domain often require extensive fine-tuning or rely on training resource-intensive decoders.<n>We introduce a suite of task-agnostic pretraining of CT foundation models (TAP-CT)<n>Our approach incorporates targeted modifications to patch embeddings, positional encodings, and volumetric augmentations, making the architecture depth-aware.
  arXiv  Detail & Related papers  (2025-11-30T12:43:15Z)
• EoS-FM: Can an Ensemble of Specialist Models act as a Generalist Feature Extractor? [8.178030486012437]
  We present an Ensemble-of-Specialists framework for building Remote Sensing Foundation Models (RSFMs)<n>Our method decomposes the training process into lightweight, task-specific ConvNeXtV2 specialists that can be frozen and reused.<n>Our framework sets a new direction for building scalable and efficient RSFMs.
  arXiv  Detail & Related papers  (2025-11-26T15:52:56Z)
• Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation [43.68215777330875]
  We introduce a systematic post-training pipeline that efficiently enhances small model accuracy.<n>The resulting instruction-tuned model achieves state-of-the-art performance.<n>This work provides a practical and efficient solution for developing high-performance language models on Ascend edge devices.
  arXiv  Detail & Related papers  (2025-09-30T16:40:55Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models [16.83959862897466]
  Federated Learning (FL) is an established paradigm for training deep learning models on decentralized data.<n>We introduce FedPromo, a novel framework that enables efficient adaptation of large-scale foundation models stored on a central server to new domains encountered only by remote clients.
  arXiv  Detail & Related papers  (2025-08-05T12:00:49Z)
• KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model [63.13906424204078]
  We propose KaLM-Embedding-V2, a series of versatile and compact embedding models.<n>For model architecture, we implement the models on a 0.5B compact size with simple mean-pooling to produce fixed-length embeddings.<n>For training data, we curate over 20 categories for pre-training and 100 categories for fine-tuning and contrastive distillation.
  arXiv  Detail & Related papers  (2025-06-26T01:09:44Z)
• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approaches [64.42735183056062]
  Large language models (LLMs) have evolved from specialized deep models to versatile foundation models.<n>LLMs require fine-tuning on local datasets and substantial memory for deployment over the network edges.<n>LLMs have been expanded beyond text generation to create images, audio, video, and multi-modal content.<n>Model fine-tuning and model-compression techniques have been developed to support the sustainable growth of LLMs.
  arXiv  Detail & Related papers  (2024-08-20T09:42:17Z)
• Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model [20.054342930450055]
  This paper introduces a novel method of Progressive Low Rank Decomposition (PLRD) tailored for the compression of large language models. PLRD allows for significant reductions in computational overhead and energy consumption. Our findings suggest that PLRD could set a new standard for the efficient scaling of LLMs.
  arXiv  Detail & Related papers  (2024-06-28T15:27:57Z)
• Towards Efficient Task-Driven Model Reprogramming with Foundation Models [52.411508216448716]
  Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task.
  arXiv  Detail & Related papers  (2023-04-05T07:28:33Z)
• Generic-to-Specific Distillation of Masked Autoencoders [119.21281960831651]
  We propose generic-to-specific distillation (G2SD) to tap the potential of small ViT models under the supervision of large models pre-trained by masked autoencoders. With G2SD, the vanilla ViT-Small model achieves 98.7%, 98.1% and 99.3% the performance of its teacher for image classification, object detection, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-02-28T17:13:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.