Relative Representations: Topological and Geometric Perspectives

• URL: http://arxiv.org/abs/2409.10967v1
• Date: Tue, 17 Sep 2024 08:09:22 GMT
• Title: Relative Representations: Topological and Geometric Perspectives
• Authors: Alejandro García-Castellanos, Giovanni Luca Marchetti, Danica Kragic, Martina Scolamiero,
• Abstract summary: Relative representations are an established approach to zero-shot model stitching. We introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes.
• Score: 53.88896255693922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Relative representations are an established approach to zero-shot model stitching, consisting of a non-trainable transformation of the latent space of a deep neural network. Based on insights of topological and geometric nature, we propose two improvements to relative representations. First, we introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. The latter coincides with the symmetries in parameter space induced by common activation functions. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes. We provide an empirical investigation on a natural language task, where both the proposed variations yield improved performance on zero-shot model stitching.

Related papers

• Unified theory for joint covariance properties under geometric image transformations for spatio-temporal receptive fields according to the generalized Gaussian derivative model for visual receptive fields [0.5439020425819]
  We show how the parameters of receptive fields need to be transformed, in order to match the output from receptive fields under atemporal image relations. We conclude with geometric analysis, showing how derived joint covariance properties make it possible to relate or match receptive field responses.
  arXiv  Detail & Related papers  (2023-11-17T14:10:55Z)
• Affine Invariance in Continuous-Domain Convolutional Neural Networks [6.019182604573028]
  This research studies affine invariance on continuous-domain convolutional neural networks. We introduce a new criterion to assess the similarity of two input signals under affine transformations. Our research could eventually extend the scope of geometrical transformations that practical deep-learning pipelines can handle.
  arXiv  Detail & Related papers  (2023-11-13T14:17:57Z)
• Distributed Bayesian Learning of Dynamic States [65.7870637855531]
  The proposed algorithm is a distributed Bayesian filtering task for finite-state hidden Markov models. It can be used for sequential state estimation, as well as for modeling opinion formation over social networks under dynamic environments.
  arXiv  Detail & Related papers  (2022-12-05T19:40:17Z)
• Deep neural networks on diffeomorphism groups for optimal shape reparameterization [44.99833362998488]
  We propose an algorithm for constructing approximations of orientation-preserving diffeomorphisms by composition of elementary diffeomorphisms. The algorithm is implemented using PyTorch, and is applicable for both unparametrized curves and surfaces.
  arXiv  Detail & Related papers  (2022-07-22T15:25:59Z)
• Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime [9.184987303791292]
  We show that in permuted MNIST image classification tasks, the performance of multilayer perceptrons trained by vanilla gradient descent can be improved. We provide a theoretical explanation of this effect by studying a qualitatively similar two-task linear regression problem. We show that when a model is trained on the two tasks in sequence without any additional regularization, the risk gain on the first task is small.
  arXiv  Detail & Related papers  (2022-06-01T18:04:33Z)
• Counting Phases and Faces Using Bayesian Thermodynamic Integration [77.34726150561087]
  We introduce a new approach to reconstruction of the thermodynamic functions and phase boundaries in two-parametric statistical mechanics systems. We use the proposed approach to accurately reconstruct the partition functions and phase diagrams of the Ising model and the exactly solvable non-equilibrium TASEP.
  arXiv  Detail & Related papers  (2022-05-18T17:11:23Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Self-Supervised Graph Representation Learning via Topology Transformations [61.870882736758624]
  We present the Topology Transformation Equivariant Representation learning, a general paradigm of self-supervised learning for node representations of graph data. In experiments, we apply the proposed model to the downstream node and graph classification tasks, and results show that the proposed method outperforms the state-of-the-art unsupervised approaches.
  arXiv  Detail & Related papers  (2021-05-25T06:11:03Z)
• Joint Network Topology Inference via Structured Fusion Regularization [70.30364652829164]
  Joint network topology inference represents a canonical problem of learning multiple graph Laplacian matrices from heterogeneous graph signals. We propose a general graph estimator based on a novel structured fusion regularization. We show that the proposed graph estimator enjoys both high computational efficiency and rigorous theoretical guarantee.
  arXiv  Detail & Related papers  (2021-03-05T04:42:32Z)
• Addressing the Topological Defects of Disentanglement via Distributed Operators [10.29148285032989]
  A popular approach to disentanglement consists in learning to map each of these factors to distinct subspaces of a model's latent representation. Here, we show that for a broad family of transformations acting on images, this approach introduces topological defects. Motivated by classical results from group representation theory, we study an alternative, more flexible approach to disentanglement.
  arXiv  Detail & Related papers  (2021-02-10T18:34:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.