Topologically Regularized Data Embeddings
- URL: http://arxiv.org/abs/2110.09193v1
- Date: Mon, 18 Oct 2021 11:25:47 GMT
- Title: Topologically Regularized Data Embeddings
- Authors: Robin Vandaele, Bo Kang, Jefrey Lijffijt, Tijl De Bie, Yvan Saeys
- Abstract summary: We introduce a new set of topological losses, and propose their usage as a way for topologically regularizing data embeddings to naturally represent a prespecified model.
We include experiments on synthetic and real data that highlight the usefulness and versatility of this approach.
- Score: 22.222311627054875
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Unsupervised feature learning often finds low-dimensional embeddings that
capture the structure of complex data. For tasks for which expert prior
topological knowledge is available, incorporating this into the learned
representation may lead to higher quality embeddings. For example, this may
help one to embed the data into a given number of clusters, or to accommodate
for noise that prevents one from deriving the distribution of the data over the
model directly, which can then be learned more effectively. However, a general
tool for integrating different prior topological knowledge into embeddings is
lacking. Although differentiable topology layers have been recently developed
that can (re)shape embeddings into prespecified topological models, they have
two important limitations for representation learning, which we address in this
paper. First, the currently suggested topological losses fail to represent
simple models such as clusters and flares in a natural manner. Second, these
losses neglect all original structural (such as neighborhood) information in
the data that is useful for learning. We overcome these limitations by
introducing a new set of topological losses, and proposing their usage as a way
for topologically regularizing data embeddings to naturally represent a
prespecified model. We include thorough experiments on synthetic and real data
that highlight the usefulness and versatility of this approach, with
applications ranging from modeling high-dimensional single cell data, to graph
embedding.
Related papers
- Dissecting embedding method: learning higher-order structures from data [0.0]
Geometric deep learning methods for data learning often include set of assumptions on the geometry of the feature space.
These assumptions together with data being discrete and finite can cause some generalisations, which are likely to create wrong interpretations of the data and models outputs.
arXiv Detail & Related papers (2024-10-14T08:19:39Z) - On Characterizing the Evolution of Embedding Space of Neural Networks
using Algebraic Topology [9.537910170141467]
We study how the topology of feature embedding space changes as it passes through the layers of a well-trained deep neural network (DNN) through Betti numbers.
We demonstrate that as depth increases, a topologically complicated dataset is transformed into a simple one, resulting in Betti numbers attaining their lowest possible value.
arXiv Detail & Related papers (2023-11-08T10:45:12Z) - Homological Convolutional Neural Networks [4.615338063719135]
We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations.
We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
arXiv Detail & Related papers (2023-08-26T08:48:51Z) - Topologically Regularized Data Embeddings [15.001598256750619]
We introduce a generic approach based on algebraic topology to incorporate topological prior knowledge into low-dimensional embeddings.
We show that jointly optimizing an embedding loss with such a topological loss function as a regularizer yields embeddings that reflect not only local proximities but also the desired topological structure.
We empirically evaluate the proposed approach on computational efficiency, robustness, and versatility in combination with linear and non-linear dimensionality reduction and graph embedding methods.
arXiv Detail & Related papers (2023-01-09T13:49:47Z) - Towards a mathematical understanding of learning from few examples with
nonlinear feature maps [68.8204255655161]
We consider the problem of data classification where the training set consists of just a few data points.
We reveal key relationships between the geometry of an AI model's feature space, the structure of the underlying data distributions, and the model's generalisation capabilities.
arXiv Detail & Related papers (2022-11-07T14:52:58Z) - Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
Federated learning aims to train models across different clients without the sharing of data for privacy considerations.
We study how data heterogeneity affects the representations of the globally aggregated models.
We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
arXiv Detail & Related papers (2022-10-01T09:04:17Z) - CHALLENGER: Training with Attribution Maps [63.736435657236505]
We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance.
In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
arXiv Detail & Related papers (2022-05-30T13:34:46Z) - Learning from few examples with nonlinear feature maps [68.8204255655161]
We explore the phenomenon and reveal key relationships between dimensionality of AI model's feature space, non-degeneracy of data distributions, and the model's generalisation capabilities.
The main thrust of our present analysis is on the influence of nonlinear feature transformations mapping original data into higher- and possibly infinite-dimensional spaces on the resulting model's generalisation capabilities.
arXiv Detail & Related papers (2022-03-31T10:36:50Z) - Towards Open-World Feature Extrapolation: An Inductive Graph Learning
Approach [80.8446673089281]
We propose a new learning paradigm with graph representation and learning.
Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
arXiv Detail & Related papers (2021-10-09T09:02:45Z) - A Topological Framework for Deep Learning [0.7310043452300736]
We show that the classification problem in machine learning is always solvable under very mild conditions.
In particular, we show that a softmax classification network acts on an input topological space by a finite sequence of topological moves to achieve the classification task.
arXiv Detail & Related papers (2020-08-31T15:56:42Z) - Self-Learning with Rectification Strategy for Human Parsing [73.06197841003048]
We propose a trainable graph reasoning method to correct two typical errors in the pseudo-labels.
The reconstructed features have a stronger ability to represent the topology structure of the human body.
Our method outperforms other state-of-the-art methods in supervised human parsing tasks.
arXiv Detail & Related papers (2020-04-17T03:51:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.