Graph Modularity: Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Neural Networks

• URL: http://arxiv.org/abs/2111.12485v1
• Date: Wed, 24 Nov 2021 13:29:17 GMT
• Title: Graph Modularity: Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Neural Networks
• Authors: Yao Lu, Wen Yang, Yunzhe Zhang, Jinhuan Wang, Shengbo Gong, Zhuangzhi Chen, Zuohui Chen, Qi Xuan, Xiaoniu Yang
• Abstract summary: We move a tiny step towards understanding the transition of feature representations in deep neural networks (DNNs) We first characterize this transition by analyzing the class separation in intermediate layers, and next model the process of class separation as community evolution in dynamic graphs. We find that modularity tends to rise as the layer goes deeper, but descends or reaches a plateau at particular layers.
• Score: 7.187240308034312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There are good arguments to support the claim that feature representations eventually transition from general to specific in deep neural networks (DNNs), but this transition remains relatively underexplored. In this work, we move a tiny step towards understanding the transition of feature representations. We first characterize this transition by analyzing the class separation in intermediate layers, and next model the process of class separation as community evolution in dynamic graphs. Then, we introduce modularity, a common metric in graph theory, to quantify the evolution of communities. We find that modularity tends to rise as the layer goes deeper, but descends or reaches a plateau at particular layers. Through an asymptotic analysis, we show that modularity can provide quantitative analysis of the transition of the feature representations. With the insight on feature representations, we demonstrate that modularity can also be used to identify and locate redundant layers in DNNs, which provides theoretical guidance for layer pruning. Based on this inspiring finding, we propose a layer-wise pruning method based on modularity. Further experiments show that our method can prune redundant layers with minimal impact on performance. The codes are available at https://github.com/yaolu-zjut/Dynamic-Graphs-Construction.

Related papers

• Manifold GCN: Diffusion-based Convolutional Neural Network for Manifold-valued Graphs [2.685668802278156]
  We propose two graph neural network layers for graphs with features in a Riemannian manifold. First, based on a manifold-valued graph diffusion equation, we construct a diffusion layer that can be applied to an arbitrary number of nodes. Second, we model a multilayer tangent perceptron by transferring ideas from the vector neuron framework to our general setting.
  arXiv  Detail & Related papers  (2024-01-25T18:36:10Z)
• Learning to Approximate Adaptive Kernel Convolution on Graphs [4.434835769977399]
  We propose a diffusion learning framework, where the range of feature aggregation is controlled by the scale of a diffusion kernel. Our model is tested on various standard for node-wise classification for the state-of-the-art datasets performance. It is also validated on a real-world brain network data for graph classifications to demonstrate its practicality for Alzheimer classification.
  arXiv  Detail & Related papers  (2024-01-22T10:57:11Z)
• A Fractional Graph Laplacian Approach to Oversmoothing [15.795926248847026]
  We generalize the concept of oversmoothing from undirected to directed graphs. We propose fractional graph Laplacian neural ODEs, which describe non-local dynamics. Our method is more flexible with respect to the convergence of the graph's Dirichlet energy, thereby mitigating oversmoothing.
  arXiv  Detail & Related papers  (2023-05-22T14:52:33Z)
• Deep Manifold Learning with Graph Mining [80.84145791017968]
  We propose a novel graph deep model with a non-gradient decision layer for graph mining. The proposed model has achieved state-of-the-art performance compared to the current models.
  arXiv  Detail & Related papers  (2022-07-18T04:34:08Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities [4.142375560633827]
  Graph neural networks (GNNs) have been widely used to learn vector representation of graph-structured data. A potential cause is that deep GNN models tend to lose the nodes' local information through many message passing steps. We propose a multi-level attention pooling architecture to solve this so-called oversmoothing problem.
  arXiv  Detail & Related papers  (2021-03-02T05:58:12Z)
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition [126.51241919472356]
  We design a simple and highly modularized graph convolutional network architecture for skeleton-based action recognition. Our network is constructed by repeating a building block that aggregates multi-granularity information from both the spatial and temporal paths.
  arXiv  Detail & Related papers  (2020-11-26T14:43:04Z)
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers [84.57980167400513]
  Neural Function Modules (NFM) aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm.
  arXiv  Detail & Related papers  (2020-10-15T20:43:17Z)
• Towards Deeper Graph Neural Networks [63.46470695525957]
  Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Several recent studies attribute this performance deterioration to the over-smoothing issue. We propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields.
  arXiv  Detail & Related papers  (2020-07-18T01:11:14Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.