Self-supervised Feature-Gate Coupling for Dynamic Network Pruning

• URL: http://arxiv.org/abs/2111.14302v2
• Date: Fri, 31 May 2024 01:32:49 GMT
• Title: Self-supervised Feature-Gate Coupling for Dynamic Network Pruning
• Authors: Mengnan Shi, Chang Liu, Jianbin Jiao, Qixiang Ye,
• Abstract summary: We propose a feature-gate coupling (FGC) approach to align distributions of features and gates. FGC is a plug-and-play module, which consists of two steps carried out in an iterative self-supervised manner. Experimental results validate that the proposed FGC method improves the baseline approach with significant margins.
• Score: 46.14789934991936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gating modules have been widely explored in dynamic network pruning to reduce the run-time computational cost of deep neural networks while preserving the representation of features. Despite the substantial progress, existing methods remain ignoring the consistency between feature and gate distributions, which may lead to distortion of gated features. In this paper, we propose a feature-gate coupling (FGC) approach aiming to align distributions of features and gates. FGC is a plug-and-play module, which consists of two steps carried out in an iterative self-supervised manner. In the first step, FGC utilizes the $k$-Nearest Neighbor method in the feature space to explore instance neighborhood relationships, which are treated as self-supervisory signals. In the second step, FGC exploits contrastive learning to regularize gating modules with generated self-supervisory signals, leading to the alignment of instance neighborhood relationships within the feature and gate spaces. Experimental results validate that the proposed FGC method improves the baseline approach with significant margins, outperforming the state-of-the-arts with better accuracy-computation trade-off. Code is publicly available.

Related papers

• Fluid Antenna System-assisted Physical Layer Secret Key Generation [64.92952968689636]
  This paper investigates physical-layer generation (PLKG) in multiant base station systems by leveraging a fluid antenna system (FAS) to dynamically radio environments.<n>We propose an assisted PLKG model that integrates transmit beamforming and port selection under independent and spatially correlated environments.<n>It is shown that the sliding window-based port selection method introduced in this paper achieves higher KGR with fewer chains through dynamic port selection.
  arXiv  Detail & Related papers  (2025-09-19T03:01:29Z)
• VFM-Guided Semi-Supervised Detection Transformer under Source-Free Constraints for Remote Sensing Object Detection [9.029534000674388]
  VG-DETR integrates a Vision Foundation Model (VFM) into the training pipeline in a "free lunch" manner.<n>We introduce a VFM-guided pseudo-label mining strategy that leverages the VFM's semantic priors to assess the reliability of the generated pseudo-labels.<n>In addition, a dual-level VFM-guided alignment method is proposed, which aligns detector features with VFM embeddings at both the instance and image levels.
  arXiv  Detail & Related papers  (2025-08-15T02:35:56Z)
• Enhancing Transferability and Consistency in Cross-Domain Recommendations via Supervised Disentanglement [13.553355329509243]
  Cross-domain recommendation aims to alleviate the data sparsity by transferring knowledge across domains.<n>Disentangled representation learning provides an effective solution to model complex user preferences.<n>We propose DGCDR, a GNN-enhanced encoder-decoder framework.
  arXiv  Detail & Related papers  (2025-07-23T01:29:45Z)
• Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking [15.052244821404079]
  We introduce Adaptive-Free Guidance (A-CFG), a novel method that tailors unconditional input by leveraging the model's predictive confidence.<n>A-CFG focuses on areas of ambiguity leading to more effective guidance.<n> Experiments on diverse language generation benchmarks show that A-CFG yields substantial improvements over standard CFG.
  arXiv  Detail & Related papers  (2025-05-26T16:40:22Z)
• From Limited Labels to Open Domains:An Efficient Learning Method for Drone-view Geo-Localization [12.785100004522059]
  Traditional supervised drone-view geo-localization (DVGL) methods heavily depend on paired training data.<n>DVGL methods require obtaining the new paired data and subsequent retraining for model adaptation.<n>We propose a novel cross-domain invariant knowledge transfer network (CDIKTNet) with limited supervision.
  arXiv  Detail & Related papers  (2025-03-10T16:46:43Z)
• Without Paired Labeled Data: An End-to-End Self-Supervised Paradigm for UAV-View Geo-Localization [2.733505168507872]
  UAV-View Geo-Localization (UVGL) aims to achieve accurate localization of unmanned aerial vehicles (UAVs) by retrieving the most relevant GPS-tagged satellite images. Existing methods heavily rely on pre-paired UAV-satellite images for supervised learning. We propose an end-to-end self-supervised UVGL method to overcome these limitations.
  arXiv  Detail & Related papers  (2025-02-17T02:53:08Z)
• Scalable spectral representations for multi-agent reinforcement learning in network MDPs [13.782868855372774]
  A popular model for multi-agent control, Network Markov Decision Processes (MDPs) pose a significant challenge to efficient learning. We first derive scalable spectral local representations for network MDPs, which induces a network linear subspace for the local $Q$-function of each agent. We design a scalable algorithmic framework for continuous state-action network MDPs, and provide end-to-end guarantees for the convergence of our algorithm.
  arXiv  Detail & Related papers  (2024-10-22T17:45:45Z)
• Fourier Test-time Adaptation with Multi-level Consistency for Robust Classification [10.291631977766672]
  We propose a novel approach called Fourier Test-time Adaptation (FTTA) to integrate input and model tuning. FTTA builds a reliable multi-level consistency measurement of paired inputs for achieving self-supervised of prediction. It was extensively validated on three large classification datasets with different modalities and organs.
  arXiv  Detail & Related papers  (2023-06-05T02:29:38Z)
• Boundary-semantic collaborative guidance network with dual-stream feedback mechanism for salient object detection in optical remote sensing imagery [22.21644705244091]
  We propose boundary-semantic collaborative guidance network (BSCGNet) with dual-stream feedback mechanism. BSCGNet exhibits distinct advantages in challenging scenarios and outperforms the 17 state-of-the-art (SOTA) approaches proposed in recent years.
  arXiv  Detail & Related papers  (2023-03-06T03:36:06Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• Stagewise Unsupervised Domain Adaptation with Adversarial Self-Training for Road Segmentation of Remote Sensing Images [93.50240389540252]
  Road segmentation from remote sensing images is a challenging task with wide ranges of application potentials. We propose a novel stagewise domain adaptation model called RoadDA to address the domain shift (DS) issue in this field. Experiment results on two benchmarks demonstrate that RoadDA can efficiently reduce the domain gap and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-28T09:29:14Z)
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search [55.164053971213576]
  convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead. Structured (channel) pruning is usually applied to reduce the model redundancy while preserving the network structure. Existing structured pruning methods require hand-crafted rules which may lead to tremendous pruning space.
  arXiv  Detail & Related papers  (2020-11-04T07:43:01Z)
• Self-Guided Adaptation: Progressive Representation Alignment for Domain Adaptive Object Detection [86.69077525494106]
  Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness of object detection models. Existing UDA methods largely ignore the instantaneous data distribution during model learning, which could deteriorate the feature representation given large domain shift. We propose a Self-Guided Adaptation (SGA) model, target at aligning feature representation and transferring object detection models across domains.
  arXiv  Detail & Related papers  (2020-03-19T13:30:45Z)
• Contradictory Structure Learning for Semi-supervised Domain Adaptation [67.89665267469053]
  Current adversarial adaptation methods attempt to align the cross-domain features. Two challenges remain unsolved: 1) the conditional distribution mismatch and 2) the bias of the decision boundary towards the source domain. We propose a novel framework for semi-supervised domain adaptation by unifying the learning of opposite structures.
  arXiv  Detail & Related papers  (2020-02-06T22:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.