MI-to-Mid Distilled Compression (M2M-DC): An Hybrid-Information-Guided-Block Pruning with Progressive Inner Slicing Approach to Model Compression

• URL: http://arxiv.org/abs/2511.06842v2
• Date: Tue, 18 Nov 2025 01:20:04 GMT
• Title: MI-to-Mid Distilled Compression (M2M-DC): An Hybrid-Information-Guided-Block Pruning with Progressive Inner Slicing Approach to Model Compression
• Abstract summary: We introduce MI-to-Mid Distilled Compression (M2M-DC), a two-scale, shape-safe compression framework.<n>M2M-DC ranks residual blocks by a label-aware mutual information signal.<n>It then alternates short KD phases with stage-coherent, residual-safe channel slicing.
• Score: 2.351601888896043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce MI-to-Mid Distilled Compression (M2M-DC), a two-scale, shape-safe compression framework that interleaves information-guided block pruning with progressive inner slicing and staged knowledge distillation (KD). First, M2M-DC ranks residual (or inverted-residual) blocks by a label-aware mutual information (MI) signal and removes the least informative units (structured prune-after-training). It then alternates short KD phases with stage-coherent, residual-safe channel slicing: (i) stage "planes" (co-slicing conv2 out-channels with the downsample path and next-stage inputs), and (ii) an optional mid-channel trim (conv1 out / bn1 / conv2 in). This targets complementary redundancy, whole computational motifs and within-stage width while preserving residual shape invariants. On CIFAR-100, M2M-DC yields a clean accuracy-compute frontier. For ResNet-18, we obtain 85.46% Top-1 with 3.09M parameters and 0.0139 GMacs (72% params, 63% GMacs vs. teacher; mean final 85.29% over three seeds). For ResNet-34, we reach 85.02% Top-1 with 5.46M params and 0.0195 GMacs (74% / 74% vs. teacher; mean final 84.62%). Extending to inverted-residuals, MobileNetV2 achieves a mean final 68.54% Top-1 at 1.71M params (27%) and 0.0186 conv GMacs (24%), improving over the teacher's 66.03% by +2.5 points across three seeds. Because M2M-DC exposes only a thin, architecture-aware interface (blocks, stages, and down sample/skip wiring), it generalizes across residual CNNs and extends to inverted-residual families with minor legalization rules. The result is a compact, practical recipe for deployment-ready models that match or surpass teacher accuracy at a fraction of the compute.

Related papers

• Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification [10.122947222129108]
  Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment.<n>We study cross-architecture knowledge distillation from a fine-tuned DINOv2 teacher to a compact bidirectional Visual State Space Model.<n>We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data.
  arXiv  Detail & Related papers  (2026-08-27T08:02:05Z)
• Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models [1.9573380763700714]
  We propose MuCoDi, a pretraining framework that distills frozen tile embeddings from multiple PFMs into compact edge-oriented encoders.<n>We pretrain students on 14.3M TCGA tiles from only 11.8K WSIs and evaluate frozen encoders on 23 clinically curated downstream classification tasks.
  arXiv  Detail & Related papers  (2026-07-06T18:14:37Z)
• VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use [0.0]
  We present VectraYX-Nano, a 41.95M- parameter decoder-only language model trained from scratch in Spanish for cybersecurity.<n>A 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at $25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M)<n> Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/-
  arXiv  Detail & Related papers  (2026-05-13T18:03:07Z)
• BoostLoRA: Growing Effective Rank by Boosting Adapters [3.022272657109967]
  BoostLoRA is a gradient-boosting framework that overcomes the limit by iteratively training and merging minimal adapters.<n>It is the first PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity.
  arXiv  Detail & Related papers  (2026-04-30T01:43:00Z)
• THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture [0.0]
  THEIA is a 2.75M modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives.<n> Transformer baselines also reach >99% on all 39 rules, and flats match THEIA on Phase-1 accuracy within 0.04pp.
  arXiv  Detail & Related papers  (2026-04-13T10:44:15Z)
• MEMENTO: Teaching LLMs to Manage Their Own Context [50.3558738319336]
  Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state.<n>We introduce MEMENTO: a method that teaches models to segment reasoning into blocks, compress each block into a memento, and reason forward by attending only to mementos.<n>We show that a two-stage SFT recipe on OpenMementos is effective across different model families and scales.
  arXiv  Detail & Related papers  (2026-04-10T19:30:29Z)
• Parameter Reduction Improves Vision Transformers: A Comparative Study of Sharing and Width Reduction [0.0]
  We study two simple-reduction strategies applied to ViT-B/16 on ImageNet-1K.<n>Our findings suggest that architectural constraints such as parameter sharing and reduced width may act as useful inductive biases.
  arXiv  Detail & Related papers  (2025-11-30T20:04:30Z)
• Time-Series at the Edge: Tiny Separable CNNs for Wearable Gait Detection and Optimal Sensor Placement [3.7765281299298015]
  We study on-device time-series analysis for gait detection in Parkinson's disease (PD) from short windows of triaxial acceleration, targeting resource-latency wearables and edge nodes.<n>We compare magnitude thresholding to three 1D CNNs for time-series analysis: a literature baseline (separable convolutions) and two ultra-light models - one purely separable and one with residual connections.
  arXiv  Detail & Related papers  (2025-11-29T08:52:41Z)
• Temporal Zoom Networks: Distance Regression and Continuous Depth for Efficient Action Localization [6.908972852063454]
  Boundary Distance Regression (BDR) replaces classification with signed-distance regression and zero-crossing extraction.<n>BDR retrofits to existing methods with about 50 lines of code, improving mAP@0.7 by 1.8 to 3.1 percent (average +2.4)<n>On THUMOS14, ATR achieves 56.5 percent mAP@0.7 at 151G FLOPs versus 53.6 percent at 198G for the Uniform-6 baseline.
  arXiv  Detail & Related papers  (2025-11-06T00:41:54Z)
• Exploring the Hierarchical Reasoning Model for Small Natural-Image Classification Without Augmentation [51.56484100374058]
  It is evaluated on MNIST, CIFAR-10, and CIFAR-100 under a deliberately raw regime.<n>It is concluded that, for small-resolution image classification without augmentation, HRM is not competitive with even simple convolutional architectures.
  arXiv  Detail & Related papers  (2025-10-04T01:22:41Z)
• IMAGINE: An 8-to-1b 22nm FD-SOI Compute-In-Memory CNN Accelerator With an End-to-End Analog Charge-Based 0.15-8POPS/W Macro Featuring Distribution-Aware Data Reshaping [0.6071203743728119]
  We present IMAGINE, a workload-adaptive 1-to-8b CIM-CNN accelerator in 22nm FD-SOI.<n>It introduces a 1152x256 end-to-end charge-based macro with a multi-bit DP based on an input-serial, weight-parallel accumulation that avoids power-hungry DACs.<n>Measurement results showcase an 8b system-level energy efficiency of 40TOPS/W at 0.3/0.6V, with competitive accuracies on MNIST and CIFAR-10.
  arXiv  Detail & Related papers  (2024-12-27T17:18:15Z)
• Multi-Feature Fusion and Compressed Bi-LSTM for Memory-Efficient Heartbeat Classification on Wearable Devices [0.0]
  We present a resource-efficient approach for electrocardiogram (ECG) based heartbeat classification using multi-feature fusion and bidirectional long short-term memory (Bi-LSTM)<n>The dataset comprises five original classes from the MIT-BIH Arrhythmia Database: Normal (N), Left Bundle Branch Block (LBBB), Right Bundle Branch Block (RBBB), Premature Ventricular Contraction (PVC), and Paced Beat (PB)
  arXiv  Detail & Related papers  (2024-05-24T07:53:27Z)
• SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation [53.675725490807615]
  We introduce SDPose, a new self-distillation method for improving the performance of small transformer-based models. SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs, while SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset.
  arXiv  Detail & Related papers  (2024-04-04T15:23:14Z)
• Filter Pruning For CNN With Enhanced Linear Representation Redundancy [3.853146967741941]
  We present a data-driven loss function term calculated from the correlation matrix of different feature maps in the same layer, named CCM-loss. CCM-loss provides us with another universal transcendental mathematical tool besides L*-norm regularization. In our new strategy, we mainly focus on the consistency and integrality of the information flow in the network.
  arXiv  Detail & Related papers  (2023-10-10T06:27:30Z)
• Self-Supervised Learning based on Heat Equation [75.61974935666446]
  This paper presents a new perspective of self-supervised learning based on extending heat equation into high dimensional feature space. We simplify it by splitting x and y axes as two first-order linear differential equations. This introduces a very simple masked image modeling (MIM) method, named QB-Heat.
  arXiv  Detail & Related papers  (2022-11-23T18:59:59Z)
• SplitMixer: Fat Trimmed From MLP-like Models [53.12472550578278]
  We present SplitMixer, a simple and lightweight isotropic-like architecture, for visual recognition. It contains two types of interleaving convolutional operations to mix information across locations (spatial mixing) and channels (channel mixing)
  arXiv  Detail & Related papers  (2022-07-21T01:37:07Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion [139.71852076031962]
  We present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF) We derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47%.
  arXiv  Detail & Related papers  (2021-07-14T18:07:38Z)
• Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic model [6.338178373376447]
  We investigate if the wav2vec 2.0 self-supervised pretraining helps mitigate the overfitting issues with connectionist temporal classification. For supervised adaptation of the wav2vec 2.0 model, both E2E-LFMMI and CTC achieve similar results. For Babel languages, we obtain relative improvements of 26% and 23% on Swahili (38h) and 18% and 17% on Tagalog (84h) respectively.
  arXiv  Detail & Related papers  (2021-04-06T14:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.