Power Normalizations in Fine-grained Image, Few-shot Image and Graph Classification

• URL: http://arxiv.org/abs/2012.13975v1
• Date: Sun, 27 Dec 2020 17:06:06 GMT
• Title: Power Normalizations in Fine-grained Image, Few-shot Image and Graph Classification
• Authors: Piotr Koniusz and Hongguang Zhang
• Abstract summary: We study Power Normalizations (PN) in the deep learning setup via a novel PN layer pooling feature maps. We investigate the role and meaning of MaxExp and Gamma, two popular PN functions. We show that SPN on the autocorrelation/covariance matrix and the Heat Diffusion Process (HDP) on a graph Laplacian matrix are closely related.
• Score: 38.84294567166725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Power Normalizations (PN) are useful non-linear operators which tackle feature imbalances in classification problems. We study PNs in the deep learning setup via a novel PN layer pooling feature maps. Our layer combines the feature vectors and their respective spatial locations in the feature maps produced by the last convolutional layer of CNN into a positive definite matrix with second-order statistics to which PN operators are applied, forming so-called Second-order Pooling (SOP). As the main goal of this paper is to study Power Normalizations, we investigate the role and meaning of MaxExp and Gamma, two popular PN functions. To this end, we provide probabilistic interpretations of such element-wise operators and discover surrogates with well-behaved derivatives for end-to-end training. Furthermore, we look at the spectral applicability of MaxExp and Gamma by studying Spectral Power Normalizations (SPN). We show that SPN on the autocorrelation/covariance matrix and the Heat Diffusion Process (HDP) on a graph Laplacian matrix are closely related, thus sharing their properties. Such a finding leads us to the culmination of our work, a fast spectral MaxExp which is a variant of HDP for covariances/autocorrelation matrices. We evaluate our ideas on fine-grained recognition, scene recognition, and material classification, as well as in few-shot learning and graph classification.

Related papers

• On the dynamical Lie algebras of quantum approximate optimization algorithms [4.987686869768721]
  Dynamical Lie algebras (DLAs) have emerged as a valuable tool in the study of parameterized quantum circuits. In this work, we investigate DLAs for the quantum approximate optimization algorithm (QAOA) We show that the dimension of the DLA is $O(n3)$ and give an explicit basis for the DLA.
  arXiv  Detail & Related papers  (2024-07-17T14:12:30Z)
• Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry [63.694184882697435]
  Global Covariance Pooling (GCP) has been demonstrated to improve the performance of Deep Neural Networks (DNNs) by exploiting second-order statistics of high-level representations.
  arXiv  Detail & Related papers  (2024-07-15T07:11:44Z)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion [83.90492831583997]
  We show that a batch-normalized network can keep the optimal signal propagation properties, but avoid exploding gradients in depth. We use a Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded depth. We also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
  arXiv  Detail & Related papers  (2023-10-03T12:35:02Z)
• Graph Positional Encoding via Random Feature Propagation [39.84324765957645]
  Two main families of node feature augmentation schemes have been explored for enhancing GNNs. We propose a novel family of positional encoding schemes which draws a link between the above two approaches. We empirically demonstrate that RFP significantly outperforms both spectral PE and random features in multiple node classification and graph classification benchmarks.
  arXiv  Detail & Related papers  (2023-03-06T06:28:20Z)
• Curvature-informed multi-task learning for graph networks [56.155331323304]
  State-of-the-art graph neural networks attempt to predict multiple properties simultaneously. We investigate a potential explanation for this phenomenon: the curvature of each property's loss surface significantly varies, leading to inefficient learning.
  arXiv  Detail & Related papers  (2022-08-02T18:18:41Z)
• Graph Kernel Neural Networks [53.91024360329517]
  We propose to use graph kernels, i.e. kernel functions that compute an inner product on graphs, to extend the standard convolution operator to the graph domain. This allows us to define an entirely structural model that does not require computing the embedding of the input graph. Our architecture allows to plug-in any type of graph kernels and has the added benefit of providing some interpretability.
  arXiv  Detail & Related papers  (2021-12-14T14:48:08Z)
• High-order Tensor Pooling with Attention for Action Recognition [39.22510412349891]
  We capture high-order statistics of feature vectors formed by a neural network. We propose end-to-end second- and higher-order pooling to form a tensor descriptor.
  arXiv  Detail & Related papers  (2021-10-11T12:32:56Z)
• Graph Convolutional Network with Generalized Factorized Bilinear Aggregation [31.674649135019386]
  We propose a novel generalization of Factorized Bilinear (FB) layer to model the feature interactions in Graph Convolutional Networks (GCNs) Our experimental results on multiple datasets demonstrate that the GFB-GCN is competitive with other methods for text classification.
  arXiv  Detail & Related papers  (2021-07-24T17:57:06Z)
• Finite-Function-Encoding Quantum States [52.77024349608834]
  We introduce finite-function-encoding (FFE) states which encode arbitrary $d$-valued logic functions. We investigate some of their structural properties.
  arXiv  Detail & Related papers  (2020-12-01T13:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.