GhostNetV3-Small: A Tailored Architecture and Comparative Study of Distillation Strategies for Tiny Images

• URL: http://arxiv.org/abs/2509.12380v1
• Date: Mon, 15 Sep 2025 19:19:09 GMT
• Title: GhostNetV3-Small: A Tailored Architecture and Comparative Study of Distillation Strategies for Tiny Images
• Abstract summary: This paper explores strategies for compressing and adapting models to enable efficient inference in resource-constrained environments.<n>We focus on GhostNetV3, a state-of-the-art architecture for mobile applications, and propose GhostNetV3-Small, a modified variant designed to perform better on low-resolution inputs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks have achieved remarkable success across a range of tasks, however their computational demands often make them unsuitable for deployment on resource-constrained edge devices. This paper explores strategies for compressing and adapting models to enable efficient inference in such environments. We focus on GhostNetV3, a state-of-the-art architecture for mobile applications, and propose GhostNetV3-Small, a modified variant designed to perform better on low-resolution inputs such as those in the CIFAR-10 dataset. In addition to architectural adaptation, we provide a comparative evaluation of knowledge distillation techniques, including traditional knowledge distillation, teacher assistants, and teacher ensembles. Experimental results show that GhostNetV3-Small significantly outperforms the original GhostNetV3 on CIFAR-10, achieving an accuracy of 93.94%. Contrary to expectations, all examined distillation strategies led to reduced accuracy compared to baseline training. These findings indicate that architectural adaptation can be more impactful than distillation in small-scale image classification tasks, highlighting the need for further research on effective model design and advanced distillation techniques for low-resolution domains.

Related papers

• PSViT: A Methodology for Structurally Pruning Spiking Vision Transformers [2.9583391652768314]
  Spiking Vision Transformer (SViT) models are promising low-power ViT models for solving vision-based tasks.<n>We propose PSViT, a novel methodology to perform structured pruning on SViT models.
  arXiv  Detail & Related papers  (2026-06-02T07:18:57Z)
• Topology-Guided Knowledge Distillation for Efficient Point Cloud Processing [3.3903891679981593]
  This work introduces a novel distillation framework to transfer knowledge from a high-capacity teacher to a lightweight student model.<n>Our approach captures the underlying geometric structures of point clouds while selectively guiding the student model's learning process.<n>Our method achieves state-of-the-art performance among knowledge distillation techniques trained solely on LiDAR data.
  arXiv  Detail & Related papers  (2025-05-12T22:15:54Z)
• Comparative Analysis of Lightweight Deep Learning Models for Memory-Constrained Devices [0.0]
  Five state-of-the-art architectures are benchmarked across three diverse datasets: CIFAR-10, CIFAR-100, and Tiny ImageNet.<n>The models are assessed using four key performance metrics: classification accuracy, inference time, floating-point operations (FLOPs), and model size.
  arXiv  Detail & Related papers  (2025-05-06T08:36:01Z)
• A Light Perspective for 3D Object Detection [46.23578780480946]
  This paper introduces a novel approach that incorporates cutting-edge Deep Learning techniques into the feature extraction process.<n>Our model, NextBEV, surpasses established feature extractors like ResNet50 and MobileNetV3.<n>By fusing these lightweight proposals, we have enhanced the accuracy of the VoxelNet-based model by 2.93% and improved the F1-score of the PointPillar-based model by approximately 20%.
  arXiv  Detail & Related papers  (2025-03-10T10:03:23Z)
• Numerical Pruning for Efficient Autoregressive Models [87.56342118369123]
  This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning.<n>Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and modules, respectively.<n>To verify the effectiveness of our method, we provide both theoretical support and extensive experiments.
  arXiv  Detail & Related papers  (2024-12-17T01:09:23Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• TSCM: A Teacher-Student Model for Vision Place Recognition Using Cross-Metric Knowledge Distillation [6.856317526681759]
  Visual place recognition plays a pivotal role in autonomous exploration and navigation of mobile robots.<n>Existing methods overcome this by exploiting powerful yet large networks.<n>We propose a high-performance teacher and lightweight student distillation framework called TSCM.
  arXiv  Detail & Related papers  (2024-04-02T02:29:41Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer [76.2625311630021]
  Vision transformers (ViTs) have shown very impressive empirical performance in various computer vision tasks. To mitigate this challenging problem, structured pruning is a promising solution to compress model size and enable practical efficiency. We propose GOHSP, a unified framework of Graph and Optimization-based Structured Pruning for ViT models.
  arXiv  Detail & Related papers  (2023-01-13T00:40:24Z)
• Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition [124.80263629921498]
  We propose Pixel Distillation that extends knowledge distillation into the input level while simultaneously breaking architecture constraints. Such a scheme can achieve flexible cost control for deployment, as it allows the system to adjust both network architecture and image quality according to the overall requirement of resources.
  arXiv  Detail & Related papers  (2021-12-17T14:31:40Z)
• Towards Practical Lipreading with Distilled and Efficient Models [57.41253104365274]
  Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. There is still a significant gap between the current methodologies and the requirements for an effective deployment of lipreading in practical scenarios. We propose a series of innovations that significantly bridge that gap: first, we raise the state-of-the-art performance by a wide margin on LRW and LRW-1000 to 88.5% and 46.6%, respectively using self-distillation.
  arXiv  Detail & Related papers  (2020-07-13T16:56:27Z)
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
  We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking. We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints. FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
  arXiv  Detail & Related papers  (2020-06-03T05:20:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.