Faster and Accurate Neural Networks with Semantic Inference

• URL: http://arxiv.org/abs/2310.01259v2
• Date: Tue, 3 Oct 2023 15:08:22 GMT
• Title: Faster and Accurate Neural Networks with Semantic Inference
• Abstract summary: We leverage the intrinsic redundancy in latent representations to reduce the computational load with limited loss in performance. New framework called Semantic Inference (SINF) identifies the semantic cluster the object belongs to and executes the subgraph extracted from that semantic cluster for inference. New approach named Discriminative Capability Score (DCS) that finds the subgraph with the capability to discriminate among the members of a specific semantic cluster.
• Score: 9.372908891132772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNN) usually come with a significant computational burden. While approaches such as structured pruning and mobile-specific DNNs have been proposed, they incur drastic accuracy loss. In this paper we leverage the intrinsic redundancy in latent representations to reduce the computational load with limited loss in performance. We show that semantically similar inputs share many filters, especially in the earlier layers. Thus, semantically similar classes can be clustered to create cluster-specific subgraphs. To this end, we propose a new framework called Semantic Inference (SINF). In short, SINF (i) identifies the semantic cluster the object belongs to using a small additional classifier and (ii) executes the subgraph extracted from the base DNN related to that semantic cluster for inference. To extract each cluster-specific subgraph, we propose a new approach named Discriminative Capability Score (DCS) that finds the subgraph with the capability to discriminate among the members of a specific semantic cluster. DCS is independent from SINF and can be applied to any DNN. We benchmark the performance of DCS on the VGG16, VGG19, and ResNet50 DNNs trained on the CIFAR100 dataset against 6 state-of-the-art pruning approaches. Our results show that (i) SINF reduces the inference time of VGG19, VGG16, and ResNet50 respectively by up to 35%, 29% and 15% with only 0.17%, 3.75%, and 6.75% accuracy loss (ii) DCS achieves respectively up to 3.65%, 4.25%, and 2.36% better accuracy with VGG16, VGG19, and ResNet50 with respect to existing discriminative scores (iii) when used as a pruning criterion, DCS achieves up to 8.13% accuracy gain with 5.82% less parameters than the existing state of the art work published at ICLR 2023 (iv) when considering per-cluster accuracy, SINF performs on average 5.73%, 8.38% and 6.36% better than the base VGG16, VGG19, and ResNet50.

Related papers

• Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency [0.0]
  This study compares nine lightweight CNN model packages across CIFAR-10, CIFAR-100, and Tiny ImageNet.<n>We report top-1 accuracy, macro F1, top-5 accuracy, parameter count, FP32 storage, GMACs, batch-size-1 latency on an NVIDIA L4 and AMD 5 5500U CPU.
  arXiv  Detail & Related papers  (2026-07-02T10:14:07Z)
• SOLAR: Switchable Output Layer for Accuracy and Robustness in Once-for-All Training [5.872858426296439]
  Once-for-All (OFA) training enables a single super-net to generate multiple sub-nets tailored to diverse deployment scenarios.<n>We propose SOLAR (Switchable Output Layer for Accuracy and Robustness in Once-for-All Training), a technique that assigns each sub-net a separate classification head.
  arXiv  Detail & Related papers  (2025-09-20T23:15:28Z)
• Time Frequency Analysis of EMG Signal for Gesture Recognition using Fine grained Features [3.9440964696313485]
  This paper proposes a novel approach to EMG-based hand gesture recognition that uses fine-grained classification.<n> XMANet unifies low-level local and high level semantic cues through cross layer mutual attention among shallow to deep CNN experts.
  arXiv  Detail & Related papers  (2025-04-20T18:51:10Z)
• Graph Neural Networks Need Cluster-Normalize-Activate Modules [19.866482154218374]
  Graph Neural Networks (GNNs) are non-Euclidean deep learning models for graph-structured data. We propose a plug-and-play module consisting of three steps: Cluster-Normalize-Activate (CNA) CNA significantly improves the accuracy over the state-of-the-art in node classification and property prediction tasks.
  arXiv  Detail & Related papers  (2024-12-05T10:59:20Z)
• Towards Generalized Entropic Sparsification for Convolutional Neural Networks [0.0]
  Convolutional neural networks (CNNs) are reported to be overparametrized. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint.
  arXiv  Detail & Related papers  (2024-04-06T21:33:39Z)
• Touch Analysis: An Empirical Evaluation of Machine Learning Classification Algorithms on Touch Data [7.018254711671888]
  We propose a novel Deep Neural Net (DNN) architecture to classify the individuals correctly. When we combine the new features with the existing ones, SVM and kNN achieved the classification accuracy of 94.7% and 94.6%, respectively.
  arXiv  Detail & Related papers  (2023-11-23T20:31:48Z)
• Patch-Level Contrasting without Patch Correspondence for Accurate and Dense Contrastive Representation Learning [79.43940012723539]
  ADCLR is a self-supervised learning framework for learning accurate and dense vision representation. Our approach achieves new state-of-the-art performance for contrastive methods.
  arXiv  Detail & Related papers  (2023-06-23T07:38:09Z)
• Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement [68.44100784364987]
  We propose a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no additional training cost for users. We create a reinforced version of the ImageNet training dataset, called ImageNet+, as well as reinforced datasets CIFAR-100+, Flowers-102+, and Food-101+. Models trained with ImageNet+ are more accurate, robust, and calibrated, and transfer well to downstream tasks.
  arXiv  Detail & Related papers  (2023-03-15T23:10:17Z)
• Upcycling Models under Domain and Category Shift [95.22147885947732]
  We introduce an innovative global and local clustering learning technique (GLC) We design a novel, adaptive one-vs-all global clustering algorithm to achieve the distinction across different target classes. Remarkably, in the most challenging open-partial-set DA scenario, GLC outperforms UMAD by 14.8% on the VisDA benchmark.
  arXiv  Detail & Related papers  (2023-03-13T13:44:04Z)
• EGRC-Net: Embedding-induced Graph Refinement Clustering Network [66.44293190793294]
  We propose a novel graph clustering network called Embedding-Induced Graph Refinement Clustering Network (EGRC-Net) EGRC-Net effectively utilizes the learned embedding to adaptively refine the initial graph and enhance the clustering performance. Our proposed methods consistently outperform several state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-11-19T09:08:43Z)
• Focal Modulation Networks [105.93086472906765]
  Self-attention (SA) is completely replaced by focal modulation network (FocalNet) FocalNets with tiny and base sizes achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. FocalNets exhibit remarkable superiority when transferred to downstream tasks.
  arXiv  Detail & Related papers  (2022-03-22T17:54:50Z)
• BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing [48.11023234245863]
  We propose a new framework called BottleFit, which includes a novel training strategy to achieve high accuracy even with strong compression rates. BottleFit achieves 77.1% data compression with up to 0.6% accuracy loss on ImageNet dataset. We show that BottleFit decreases power consumption and latency respectively by up to 49% and 89% with respect to (w.r.t.) local computing and by 37% and 55% w.r.t. edge offloading.
  arXiv  Detail & Related papers  (2022-01-07T22:08:07Z)
• Problem-dependent attention and effort in neural networks with applications to image resolution and model selection [0.0]
  This paper introduces two new ensemble-based methods to reduce the data and computation costs of image classification. They can be used with any set of classifiers and do not require additional training.
  arXiv  Detail & Related papers  (2022-01-05T02:27:35Z)
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion [139.71852076031962]
  We present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF) We derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47%.
  arXiv  Detail & Related papers  (2021-07-14T18:07:38Z)
• Post-training deep neural network pruning via layer-wise calibration [70.65691136625514]
  We propose a data-free extension of the approach for computer vision models based on automatically-generated synthetic fractal images. When using real data, we are able to get a ResNet50 model on ImageNet with 65% sparsity rate in 8-bit precision in a post-training setting.
  arXiv  Detail & Related papers  (2021-04-30T14:20:51Z)
• Deep learning-based Edge-aware pre and post-processing methods for JPEG compressed images [7.803786205476811]
  We propose a learning-based compression scheme that envelopes a standard between pre and post-processing deep CNNs. We demonstrate improvements over prior approaches utilizing a compression-decompression network by introducing: (a) an edge-aware loss function to prevent blurring that is commonly occurred in prior works & (b) a super-resolution convolutional neural network (CNN) for post-processing along with a corresponding pre-processing network for improved rate-distortion performance in the low rate regime.
  arXiv  Detail & Related papers  (2021-04-11T05:50:31Z)
• Noise Injection-based Regularization for Point Cloud Processing [17.742343473850063]
  Noise injection-based regularization has been widely used in image domain to improve the performance of deep neural networks (DNNs) We propose a series of regularization techniques, namely DropFeat, DropPoint and DropCluster, to perform noise injection on the point feature maps. Experimental results show our approaches enable significant performance improvement.
  arXiv  Detail & Related papers  (2021-03-28T00:55:25Z)
• ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN [6.662639002101124]
  convolutional neural network (CNN) gets deeper and wider in recent years. Existing magnitude-based pruning methods are efficient, but the performance of the compressed network is unpredictable. We propose a novel automatic channel pruning method (ACP) ACP is evaluated against several state-of-the-art CNNs on three different classification datasets.
  arXiv  Detail & Related papers  (2021-01-16T08:56:38Z)
• Inception Convolution with Efficient Dilation Search [121.41030859447487]
  Dilation convolution is a critical mutant of standard convolution neural network to control effective receptive fields and handle large scale variance of objects. We propose a new mutant of dilated convolution, namely inception (dilated) convolution where the convolutions have independent dilation among different axes, channels and layers. We explore a practical method for fitting the complex inception convolution to the data, a simple while effective dilation search algorithm(EDO) based on statistical optimization is developed.
  arXiv  Detail & Related papers  (2020-12-25T14:58:35Z)
• Revisiting Batch Normalization for Improving Corruption Robustness [85.20742045853738]
  We interpret corruption robustness as a domain shift and propose to rectify batch normalization statistics for improving model robustness. We find that simply estimating and adapting the BN statistics on a few representation samples, without retraining the model, improves the corruption robustness by a large margin.
  arXiv  Detail & Related papers  (2020-10-07T19:56:47Z)
• Method for classifying a noisy Raman spectrum based on a wavelet transform and a deep neural network [0.0]
  The framework consists of two main engines. Wavelet transform is proposed as the framework front-end for transforming 1-D noise Raman spectrum to two-dimensional data. The optimum classifier is chosen by implementing several traditional machine learning (ML) and deep learning (DL) algorithms.
  arXiv  Detail & Related papers  (2020-09-09T02:31:23Z)
• Second-Order Provable Defenses against Adversarial Attacks [63.34032156196848]
  We show that if the eigenvalues of the network are bounded, we can compute a certificate in the $l$ norm efficiently using convex optimization. We achieve certified accuracy of 5.78%, and 44.96%, and 43.19% on 2,59% and 4BP-based methods respectively.
  arXiv  Detail & Related papers  (2020-06-01T05:55:18Z)
• Fixing the train-test resolution discrepancy: FixEfficientNet [98.64315617109344]
  This paper provides an analysis of the performance of the EfficientNet image classifiers with several recent training procedures. The resulting network, called FixEfficientNet, significantly outperforms the initial architecture with the same number of parameters.
  arXiv  Detail & Related papers  (2020-03-18T14:22:58Z)
• Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks [9.409651543514615]
  This work introduces convolutional layers with pre-defined sparse 2D kernels that have support sets that repeat periodically within and across filters. Due to the efficient storage of our periodic sparse kernels, the parameter savings can translate into considerable improvements in energy efficiency.
  arXiv  Detail & Related papers  (2020-01-29T07:10:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.