Neural Eigenfunctions Are Structured Representation Learners

• URL: http://arxiv.org/abs/2210.12637v3
• Date: Fri, 8 Dec 2023 05:21:47 GMT
• Title: Neural Eigenfunctions Are Structured Representation Learners
• Authors: Zhijie Deng, Jiaxin Shi, Hao Zhang, Peng Cui, Cewu Lu, Jun Zhu
• Abstract summary: This paper introduces a structured, adaptive-length deep representation called Neural Eigenmap. We show that, when the eigenfunction is derived from positive relations in a data augmentation setup, applying NeuralEF results in an objective function. We demonstrate using such representations as adaptive-length codes in image retrieval systems.
• Score: 93.53445940137618
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces a structured, adaptive-length deep representation called Neural Eigenmap. Unlike prior spectral methods such as Laplacian Eigenmap that operate in a nonparametric manner, Neural Eigenmap leverages NeuralEF to parametrically model eigenfunctions using a neural network. We show that, when the eigenfunction is derived from positive relations in a data augmentation setup, applying NeuralEF results in an objective function that resembles those of popular self-supervised learning methods, with an additional symmetry-breaking property that leads to \emph{structured} representations where features are ordered by importance. We demonstrate using such representations as adaptive-length codes in image retrieval systems. By truncation according to feature importance, our method requires up to $16\times$ shorter representation length than leading self-supervised learning ones to achieve similar retrieval performance. We further apply our method to graph data and report strong results on a node representation learning benchmark with more than one million nodes.

Related papers

• Learning local discrete features in explainable-by-design convolutional neural networks [0.0]
  We introduce an explainable-by-design convolutional neural network (CNN) based on the lateral inhibition mechanism. The model consists of the predictor, that is a high-accuracy CNN with residual or dense skip connections. By collecting observations and directly calculating probabilities, we can explain causal relationships between motifs of adjacent levels.
  arXiv  Detail & Related papers  (2024-10-31T18:39:41Z)
• Visualising Feature Learning in Deep Neural Networks by Diagonalizing the Forward Feature Map [4.776836972093627]
  We present a method for analysing feature learning by decomposing deep neural networks (DNNs) We find that DNNs converge to a minimal feature (MF) regime dominated by a number of eigenfunctions equal to the number of classes. We recast the phenomenon of neural collapse into a kernel picture which can be extended to broader tasks such as regression.
  arXiv  Detail & Related papers  (2024-10-05T18:53:48Z)
• Noise-Resilient Unsupervised Graph Representation Learning via Multi-Hop Feature Quality Estimation [53.91958614666386]
  Unsupervised graph representation learning (UGRL) based on graph neural networks (GNNs) We propose a novel UGRL method based on Multi-hop feature Quality Estimation (MQE)
  arXiv  Detail & Related papers  (2024-07-29T12:24:28Z)
• Learning Single-Index Models with Shallow Neural Networks [43.6480804626033]
  We introduce a natural class of shallow neural networks and study its ability to learn single-index models via gradient flow. We show that the corresponding optimization landscape is benign, which in turn leads to generalization guarantees that match the near-optimal sample complexity of dedicated semi-parametric methods.
  arXiv  Detail & Related papers  (2022-10-27T17:52:58Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based Gesture Recognition [73.64451471862613]
  We propose an efficient hierarchical self-attention network (HAN) for skeleton-based gesture recognition. Joint self-attention module is used to capture spatial features of fingers, the finger self-attention module is designed to aggregate features of the whole hand. Experiments show that our method achieves competitive results on three gesture recognition datasets with much lower computational complexity.
  arXiv  Detail & Related papers  (2021-06-25T02:15:53Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)
• Learning the Implicit Semantic Representation on Graph-Structured Data [57.670106959061634]
  Existing representation learning methods in graph convolutional networks are mainly designed by describing the neighborhood of each node as a perceptual whole. We propose a Semantic Graph Convolutional Networks (SGCN) that explores the implicit semantics by learning latent semantic-paths in graphs.
  arXiv  Detail & Related papers  (2021-01-16T16:18:43Z)
• GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability [14.31120627384789]
  We propose an efficient algorithm to visualise symmetries in neural networks. Our proposed method, GENNI, allows us to efficiently identify parameters that are functionally equivalent and then visualise the subspace of the resulting equivalence class.
  arXiv  Detail & Related papers  (2020-11-14T22:53:13Z)
• SCG-Net: Self-Constructing Graph Neural Networks for Semantic Segmentation [23.623276007011373]
  We propose a module that learns a long-range dependency graph directly from the image and uses it to propagate contextual information efficiently. The module is optimised via a novel adaptive diagonal enhancement method and a variational lower bound. When incorporated into a neural network (SCG-Net), semantic segmentation is performed in an end-to-end manner and competitive performance.
  arXiv  Detail & Related papers  (2020-09-03T12:13:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.