WSEBP: A Novel Width-depth Synchronous Extension-based Basis Pursuit Algorithm for Multi-Layer Convolutional Sparse Coding

• URL: http://arxiv.org/abs/2203.14856v2
• Date: Wed, 30 Mar 2022 02:22:24 GMT
• Title: WSEBP: A Novel Width-depth Synchronous Extension-based Basis Pursuit Algorithm for Multi-Layer Convolutional Sparse Coding
• Authors: Haitong Tang, Shuang He, Lingbin Bian, Zhiming Cui, Nizhuan Wang
• Abstract summary: Multi-layer convolutional sparse coding (ML-CSC) can interpret the convolutional neural networks (CNNs) Many current state-of-art (SOTA) pursuit algorithms require multiple iterations to optimize the solution of ML-CSC. We propose a novel width-depth synchronous extension-based basis pursuit (WSEBP) algorithm which solves the ML-CSC problem without the limitation of the number of iterations.
• Score: 4.521915878576165
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The pursuit algorithms integrated in multi-layer convolutional sparse coding (ML-CSC) can interpret the convolutional neural networks (CNNs). However, many current state-of-art (SOTA) pursuit algorithms require multiple iterations to optimize the solution of ML-CSC, which limits their applications to deeper CNNs due to high computational cost and large number of resources for getting very tiny gain of performance. In this study, we focus on the 0th iteration in pursuit algorithm by introducing an effective initialization strategy for each layer, by which the solution for ML-CSC can be improved. Specifically, we first propose a novel width-depth synchronous extension-based basis pursuit (WSEBP) algorithm which solves the ML-CSC problem without the limitation of the number of iterations compared to the SOTA algorithms and maximizes the performance by an effective initialization in each layer. Then, we propose a simple and unified ML-CSC-based classification network (ML-CSC-Net) which consists of an ML-CSC-based feature encoder and a fully-connected layer to validate the performance of WSEBP on image classification task. The experimental results show that our proposed WSEBP outperforms SOTA algorithms in terms of accuracy and consumption resources. In addition, the WSEBP integrated in CNNs can improve the performance of deeper CNNs and make them interpretable. Finally, taking VGG as an example, we propose WSEBP-VGG13 to enhance the performance of VGG13, which achieves competitive results on four public datasets, i.e., 87.79% vs. 86.83% on Cifar-10 dataset, 58.01% vs. 54.60% on Cifar-100 dataset, 91.52% vs. 89.58% on COVID-19 dataset, and 99.88% vs. 99.78% on Crack dataset, respectively. The results show the effectiveness of the proposed WSEBP, the improved performance of ML-CSC with WSEBP, and interpretation of the CNNs or deeper CNNs.

Related papers

• Adaptive Spatial Goodness Encoding: Advancing and Scaling Forward-Forward Learning Without Backpropagation [5.092009068303438]
  We propose a new Forward-Forward (FF)-based training framework tailored for convolutional neural networks (CNNs)<n>ASGE features maps to compute spatially-aware goodness rep- resentations at each layer, enabling layer-wise supervision.<n>We present the first successful ap- plication of FF-based training to ImageNet datasets, with Top-1 and Top-5 accuracies of 26.21% and 47.49%.
  arXiv  Detail & Related papers  (2025-09-15T19:38:32Z)
• Efficient Traffic Classification using HW-NAS: Advanced Analysis and Optimization for Cybersecurity on Resource-Constrained Devices [1.3124513975412255]
  This paper presents a hardware-efficient deep neural network (DNN) optimized through hardware-aware neural architecture search (HW-NAS)<n>It supports the classification of session-level encrypted traffic on resource-constrained Internet of Things (IoT) and edge devices.<n>The optimized model attains an accuracy of 96.59% with just 88.26K parameters, 10.08M FLOPs, and a maximum tensor size of 20.12K.
  arXiv  Detail & Related papers  (2025-06-12T21:37:45Z)
• Efficient training for large-scale optical neural network using an evolutionary strategy and attention pruning [14.20309603187239]
  MZI-based block optical neural networks (BONNs) can achieve large-scale network models.<n>We propose an attention-based pruning (CAP) algorithm for large-scale BONNs.<n>Our proposed CAP algorithm show excellent potential for larger-scale network models and more complex tasks.
  arXiv  Detail & Related papers  (2025-05-19T09:41:11Z)
• Efficient Fault Detection in WSN Based on PCA-Optimized Deep Neural Network Slicing Trained with GOA [0.6827423171182154]
  Traditional fault detection methods often struggle with optimizing deep neural networks (DNNs) for efficient performance.<n>This study proposes a novel hybrid method combining Principal Component Analysis (PCA) with a DNN optimized by the Grasshopper Optimization Algorithm (GOA) to address these limitations.<n>Our approach achieves a remarkable 99.72% classification accuracy, with exceptional precision and recall, outperforming conventional methods.
  arXiv  Detail & Related papers  (2025-05-11T15:51:56Z)
• GDSG: Graph Diffusion-based Solution Generator for Optimization Problems in MEC Networks [109.17835015018532]
  We present a Graph Diffusion-based Solution Generation (GDSG) method. This approach is designed to work with suboptimal datasets while converging to the optimal solution large probably. We build GDSG as a multi-task diffusion model utilizing a Graph Neural Network (GNN) to acquire the distribution of high-quality solutions.
  arXiv  Detail & Related papers  (2024-12-11T11:13:43Z)
• Hyperspectral Image Classification Based on Faster Residual Multi-branch Spiking Neural Network [6.166929138912052]
  This paper builds a spiking neural network (SNN) based on the leaky integrate-and-fire (LIF) neuron model for HSI classification tasks. SNN-SWMR requires a time step reduction of about 84%, training time, and testing time reduction of about 63% and 70% at the same accuracy.
  arXiv  Detail & Related papers  (2024-09-18T00:51:01Z)
• Hybrid-Task Meta-Learning: A GNN Approach for Scalable and Transferable Bandwidth Allocation [50.96751567777229]
  We develop a deep learning-based bandwidth allocation policy that is scalable with the number of users and transferable to different communication scenarios.<n>To support scalability, the bandwidth allocation policy is represented by a graph neural network (GNN)<n>We develop a hybrid-task meta-learning (HML) algorithm that trains the initial parameters of the GNN with different communication scenarios.
  arXiv  Detail & Related papers  (2023-12-23T04:25:12Z)
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference [13.924924047051782]
  Deep convolution architectures for Spiking Neural Networks (SNNs) have significantly enhanced image classification performance and reduced computational burdens. This research explores a new pathway, drawing inspiration from the progress made in Multi-Layer Perceptrons (MLPs) We propose an innovative spiking architecture that uses batch normalization to retain MFI compatibility. We establish an efficient multi-stage spiking network that blends effectively global receptive fields with local feature extraction.
  arXiv  Detail & Related papers  (2023-06-21T16:52:20Z)
• ConvBLS: An Effective and Efficient Incremental Convolutional Broad Learning System for Image Classification [63.49762079000726]
  We propose a convolutional broad learning system (ConvBLS) based on the spherical K-means (SKM) algorithm and two-stage multi-scale (TSMS) feature fusion. Our proposed ConvBLS method is unprecedentedly efficient and effective.
  arXiv  Detail & Related papers  (2023-04-01T04:16:12Z)
• Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
  We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter. We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures'' Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
  arXiv  Detail & Related papers  (2022-10-23T18:37:22Z)
• Efficient Dataset Distillation Using Random Feature Approximation [109.07737733329019]
  We propose a novel algorithm that uses a random feature approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel. Our algorithm provides at least a 100-fold speedup over KIP and can run on a single GPU. Our new method, termed an RFA Distillation (RFAD), performs competitively with KIP and other dataset condensation algorithms in accuracy over a range of large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-21T15:56:13Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Large-scale Optimization of Partial AUC in a Range of False Positive Rates [51.12047280149546]
  The area under the ROC curve (AUC) is one of the most widely used performance measures for classification models in machine learning. We develop an efficient approximated gradient descent method based on recent practical envelope smoothing technique. Our proposed algorithm can also be used to minimize the sum of some ranked range loss, which also lacks efficient solvers.
  arXiv  Detail & Related papers  (2022-03-03T03:46:18Z)
• Learning Enhancement of CNNs via Separation Index Maximizing at the First Convolutional Layer [1.6244541005112747]
  The Separation Index (SI) as a supervised complexity measure is explained its usage in better learning of CNNs for classification problems illustrate. A learning strategy proposes through which the first layer of a CNN is optimized by maximizing the SI, and the further layers are trained through the backpropagation algorithm to learn further layers.
  arXiv  Detail & Related papers  (2022-01-13T21:32:14Z)
• CSC-Unet: A Novel Convolutional Sparse Coding Strategy Based Neural Network for Semantic Segmentation [0.44289311505645573]
  We propose a novel strategy that reformulated the popularly-used convolution operation to multi-layer convolutional sparse coding block. We show that the multi-layer convolutional sparse coding block enables semantic segmentation model to converge faster, can extract finer semantic and appearance information of images, and improve the ability to recover spatial detail information.
  arXiv  Detail & Related papers  (2021-08-01T09:16:31Z)
• Greedy Network Enlarging [53.319011626986004]
  We propose a greedy network enlarging method based on the reallocation of computations. With step-by-step modifying the computations on different stages, the enlarged network will be equipped with optimal allocation and utilization of MACs. With application of our method on GhostNet, we achieve state-of-the-art 80.9% and 84.3% ImageNet top-1 accuracies.
  arXiv  Detail & Related papers  (2021-07-31T08:36:30Z)
• ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN [6.662639002101124]
  convolutional neural network (CNN) gets deeper and wider in recent years. Existing magnitude-based pruning methods are efficient, but the performance of the compressed network is unpredictable. We propose a novel automatic channel pruning method (ACP) ACP is evaluated against several state-of-the-art CNNs on three different classification datasets.
  arXiv  Detail & Related papers  (2021-01-16T08:56:38Z)
• What Deep CNNs Benefit from Global Covariance Pooling: An Optimization Perspective [102.37204254403038]
  We make an attempt to understand what deep CNNs benefit from GCP in a viewpoint of optimization. We show that GCP can make the optimization landscape more smooth and the gradients more predictive. We conduct extensive experiments using various deep CNN models on diversified tasks, and the results provide strong support to our findings.
  arXiv  Detail & Related papers  (2020-03-25T07:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.