Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding

• URL: http://arxiv.org/abs/2607.16316v1
• Date: Wed, 15 Jul 2026 04:36:19 GMT
• Title: Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding
• Abstract summary: We introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B for offline, edge-deployable vision-language retrieval.<n>Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and low latency is essential.<n>The released model contains 1,926,188,032 parameters (3.85 GB bf16), representing approximately 9.5% fewer parameters than the 2.13B teacher model.
• Score: 1.2132884030603328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this report, we introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B for offline, edge-deployable vision-language retrieval. Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and low latency is essential. Compression combines (i) probe-driven structural pruning that removes four redundant text-decoder layers (28 to 24) ranked by adjacent-layer linear CKA, and (ii) layered knowledge distillation with hole-covering teacher-student mappings, mid-layer attention-map 1-CKA, and final-layer MSE and cosine losses with Matryoshka dimensions {128, 256, 512, 1024, 2048}. The released model contains 1,926,188,032 parameters (3.85 GB bf16), representing approximately 9.5% fewer parameters than the 2.13B teacher model. Empirical evaluations on MMEB-V2 (78 tasks, VLM2Vec protocol) show that Eddy-VL achieves an overall score of 63.2 compared with 68.9 for the teacher, retaining 91.7% of the teacher's performance while recovering 6.4 of the 12.1 points lost through pruning alone (56.8). Compositional reasoning performance remains close to the teacher on SugarCrepe (86.1 vs. 86.4), MR2-Bench (24.5 vs. 24.7), and ARO (59.5 vs. 60.4), while Winoground group performance (6.8 vs. 8.5) remains the primary limitation. Depth pruning also reduces forward latency by approximately 10% (150.0 to 136.4 ms per image on NVIDIA DGX Spark using FlashAttention-2). We present the architecture, compression methodology, training procedures, and evaluation results, demonstrating the effectiveness of Eddy-VL for multimodal retrieval under constrained edge deployment. Model weights and inference code are publicly available on Hugging Face.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models [1.9573380763700714]
  We propose MuCoDi, a pretraining framework that distills frozen tile embeddings from multiple PFMs into compact edge-oriented encoders.<n>We pretrain students on 14.3M TCGA tiles from only 11.8K WSIs and evaluate frozen encoders on 23 clinically curated downstream classification tasks.
  arXiv  Detail & Related papers  (2026-07-06T18:14:37Z)
• Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification [5.584060970507507]
  We present a benchmark jointly evaluating federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification.<n>It spans 13 FL algorithms and 10 KD objectives (a 130-pair cross-product) across 504 training runs, evaluated on ModelNet40 and a clinical craniosynostosis dataset.
  arXiv  Detail & Related papers  (2026-06-30T20:12:27Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2 [0.0]
  We present Density Field State Space Models (DF-SSM), a framework for compressing SSMs to a 1-bit scaffold with int8 low-rank correction.<n>We achieve a 278 MB model (9.7x smaller than the 2.7 GB FP16 teacher) that runs at 21.4x faster inference on GPU.
  arXiv  Detail & Related papers  (2026-04-28T14:25:10Z)
• Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction [0.0]
  We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) VT.<n>SCT achieves up to 199x memory reduction per layer at rank 32, enabling full training steps on a Steam Deck handheld.<n>Rank-sweep experiments on SmolLM2-1.7B show that all tested ranks converge to the same loss floor (4.2-4.5), identifying the learning rate schedule -- not rank -- as the primary bottleneck.
  arXiv  Detail & Related papers  (2026-04-01T10:53:56Z)
• Hybrid Gated Flow (HGF): Stabilizing 1.58-bit LLMs via Selective Low-Rank Correction [0.766310831583367]
  Hybrid Gated Flow (HGF) is a dual-stream architecture that couples a 1.58-bit ternary backbone with a learnable, low-rank FP16 correction path.<n>We show that HGF 5.4 achieves a validation loss of 0.9306 compared to BitNet's 1.0294, recovering approximately 55% of the quality gap between pure ternary quantization and the FP16 baseline.
  arXiv  Detail & Related papers  (2026-02-05T03:47:17Z)
• Skywork Open Reasoner 1 Technical Report [51.403686909760914]
  We present Skywork-OR1, an effective and scalable reinforcement learning (RL) implementation for long Chain-of-Thought (CoT) models.<n>Building on the DeepSeek-R1-Distill model series, our RL approach achieves notable performance gains.<n>Our Skywork-OR1-32B model surpasses both DeepSeek-R1 and Qwen3-32B on the AIME24 and AIME25 benchmarks.
  arXiv  Detail & Related papers  (2025-05-28T12:56:04Z)
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model [118.06260386652778]
  We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs.
  arXiv  Detail & Related papers  (2024-05-07T15:56:43Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• LGD: Label-guided Self-distillation for Object Detection [59.9972914042281]
  We propose the first self-distillation framework for general object detection, termed LGD (Label-Guided self-Distillation) Our framework involves sparse label-appearance encoding, inter-object relation adaptation and intra-object knowledge mapping to obtain the instructive knowledge. Compared with a classical teacher-based method FGFI, LGD not only performs better without requiring pretrained teacher but also with 51% lower training cost beyond inherent student learning.
  arXiv  Detail & Related papers  (2021-09-23T16:55:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.