Neural Eigenfunctions Are Structured Representation Learners
- URL: http://arxiv.org/abs/2210.12637v3
- Date: Fri, 8 Dec 2023 05:21:47 GMT
- Title: Neural Eigenfunctions Are Structured Representation Learners
- Authors: Zhijie Deng, Jiaxin Shi, Hao Zhang, Peng Cui, Cewu Lu, Jun Zhu
- Abstract summary: This paper introduces a structured, adaptive-length deep representation called Neural Eigenmap.
We show that, when the eigenfunction is derived from positive relations in a data augmentation setup, applying NeuralEF results in an objective function.
We demonstrate using such representations as adaptive-length codes in image retrieval systems.
- Score: 93.53445940137618
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper introduces a structured, adaptive-length deep representation
called Neural Eigenmap. Unlike prior spectral methods such as Laplacian
Eigenmap that operate in a nonparametric manner, Neural Eigenmap leverages
NeuralEF to parametrically model eigenfunctions using a neural network. We show
that, when the eigenfunction is derived from positive relations in a data
augmentation setup, applying NeuralEF results in an objective function that
resembles those of popular self-supervised learning methods, with an additional
symmetry-breaking property that leads to \emph{structured} representations
where features are ordered by importance. We demonstrate using such
representations as adaptive-length codes in image retrieval systems. By
truncation according to feature importance, our method requires up to
$16\times$ shorter representation length than leading self-supervised learning
ones to achieve similar retrieval performance. We further apply our method to
graph data and report strong results on a node representation learning
benchmark with more than one million nodes.
Related papers
- Symmetric Single Index Learning [46.7352578439663]
One popular model is the single-index model, in which labels are produced by an unknown linear projection with a possibly unknown link function.
We consider single index learning in the setting of symmetric neural networks.
arXiv Detail & Related papers (2023-10-03T14:59:00Z) - Learning Single-Index Models with Shallow Neural Networks [43.6480804626033]
We introduce a natural class of shallow neural networks and study its ability to learn single-index models via gradient flow.
We show that the corresponding optimization landscape is benign, which in turn leads to generalization guarantees that match the near-optimal sample complexity of dedicated semi-parametric methods.
arXiv Detail & Related papers (2022-10-27T17:52:58Z) - Dynamic Inference with Neural Interpreters [72.90231306252007]
We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules.
inputs to the model are routed through a sequence of functions in a way that is end-to-end learned.
We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
arXiv Detail & Related papers (2021-10-12T23:22:45Z) - HAN: An Efficient Hierarchical Self-Attention Network for Skeleton-Based
Gesture Recognition [73.64451471862613]
We propose an efficient hierarchical self-attention network (HAN) for skeleton-based gesture recognition.
Joint self-attention module is used to capture spatial features of fingers, the finger self-attention module is designed to aggregate features of the whole hand.
Experiments show that our method achieves competitive results on three gesture recognition datasets with much lower computational complexity.
arXiv Detail & Related papers (2021-06-25T02:15:53Z) - PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive
Learning [109.84770951839289]
We present PredRNN, a new recurrent network for learning visual dynamics from historical context.
We show that our approach obtains highly competitive results on three standard datasets.
arXiv Detail & Related papers (2021-03-17T08:28:30Z) - Learning the Implicit Semantic Representation on Graph-Structured Data [57.670106959061634]
Existing representation learning methods in graph convolutional networks are mainly designed by describing the neighborhood of each node as a perceptual whole.
We propose a Semantic Graph Convolutional Networks (SGCN) that explores the implicit semantics by learning latent semantic-paths in graphs.
arXiv Detail & Related papers (2021-01-16T16:18:43Z) - Representation Learning of Reconstructed Graphs Using Random Walk Graph
Convolutional Network [12.008472517000651]
We propose wGCN -- a novel framework that utilizes random walk to obtain the node-specific mesoscopic structures of the graph.
We believe that combining high-order local structural information can more efficiently explore the potential of the network.
arXiv Detail & Related papers (2021-01-02T10:31:14Z) - Deep Reinforcement Learning of Graph Matching [63.469961545293756]
Graph matching (GM) under node and pairwise constraints has been a building block in areas from optimization to computer vision.
We present a reinforcement learning solver for GM i.e. RGM that seeks the node correspondence between pairwise graphs.
Our method differs from the previous deep graph matching model in the sense that they are focused on the front-end feature extraction and affinity function learning.
arXiv Detail & Related papers (2020-12-16T13:48:48Z) - GENNI: Visualising the Geometry of Equivalences for Neural Network
Identifiability [14.31120627384789]
We propose an efficient algorithm to visualise symmetries in neural networks.
Our proposed method, GENNI, allows us to efficiently identify parameters that are functionally equivalent and then visualise the subspace of the resulting equivalence class.
arXiv Detail & Related papers (2020-11-14T22:53:13Z) - SCG-Net: Self-Constructing Graph Neural Networks for Semantic
Segmentation [23.623276007011373]
We propose a module that learns a long-range dependency graph directly from the image and uses it to propagate contextual information efficiently.
The module is optimised via a novel adaptive diagonal enhancement method and a variational lower bound.
When incorporated into a neural network (SCG-Net), semantic segmentation is performed in an end-to-end manner and competitive performance.
arXiv Detail & Related papers (2020-09-03T12:13:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.