Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks

• URL: http://arxiv.org/abs/2310.06369v1
• Date: Tue, 10 Oct 2023 07:11:25 GMT
• Title: Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks
• Authors: Sung Moon Ko, Sumin Lee, Dae-Woong Jeong, Woohyung Lim, Sehui Han
• Abstract summary: We propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer (GATE) We find a proper diffeomorphism between pairs of tasks to ensure that every arbitrary point maps to a locally flat coordinate in the overlapping region, allowing the transfer of knowledge from the source to the target data. GATE outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.
• Score: 5.038936775643437
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Transfer learning is a crucial technique for handling a small amount of data that is potentially related to other abundant data. However, most of the existing methods are focused on classification tasks using images and language datasets. Therefore, in order to expand the transfer learning scheme to regression tasks, we propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer Encoder (GATE). In this method, we interpret the latent vectors from the model to exist on a Riemannian curved manifold. We find a proper diffeomorphism between pairs of tasks to ensure that every arbitrary point maps to a locally flat coordinate in the overlapping region, allowing the transfer of knowledge from the source to the target data. This also serves as an effective regularizer for the model to behave in extrapolation regions. In this article, we demonstrate that GATE outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.

Related papers

• Symmetry Discovery for Different Data Types [52.2614860099811]
  Equivariant neural networks incorporate symmetries into their architecture, achieving higher generalization performance. We propose LieSD, a method for discovering symmetries via trained neural networks which approximate the input-output mappings of the tasks. We validate the performance of LieSD on tasks with symmetries such as the two-body problem, the moment of inertia matrix prediction, and top quark tagging.
  arXiv  Detail & Related papers  (2024-10-13T13:39:39Z)
• Disentangled Representation Learning with the Gromov-Monge Gap [65.73194652234848]
  Learning disentangled representations from unlabelled data is a fundamental challenge in machine learning. We introduce a novel approach to disentangled representation learning based on quadratic optimal transport. We demonstrate the effectiveness of our approach for quantifying disentanglement across four standard benchmarks.
  arXiv  Detail & Related papers  (2024-07-10T16:51:32Z)
• Trivialized Momentum Facilitates Diffusion Generative Modeling on Lie Groups [37.78638937228254]
  This article demonstrates how a technique called trivialization' can transfer the effectiveness of diffusion models in Euclidean spaces to Lie groups. A momentum variable was algorithmically introduced to help transport the position variable between data distribution and a fixed, easy-to-sample distribution. The resulting method achieves state-of-the-art performance on protein and RNA torsion angle generation and sophisticated torus datasets.
  arXiv  Detail & Related papers  (2024-05-25T23:53:07Z)
• Improving embedding of graphs with missing data by soft manifolds [51.425411400683565]
  The reliability of graph embeddings depends on how much the geometry of the continuous space matches the graph structure. We introduce a new class of manifold, named soft manifold, that can solve this situation. Using soft manifold for graph embedding, we can provide continuous spaces to pursue any task in data analysis over complex datasets.
  arXiv  Detail & Related papers  (2023-11-29T12:48:33Z)
• Single Domain Generalization via Normalised Cross-correlation Based Convolutions [14.306250516592304]
  Single Domain Generalization aims to train robust models using data from a single source. We propose a novel operator called XCNorm that computes the normalized cross-correlation between weights and an input feature patch. We show that deep neural networks composed of this operator are robust to common semantic distribution shifts.
  arXiv  Detail & Related papers  (2023-07-12T04:15:36Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• The Manifold Scattering Transform for High-Dimensional Point Cloud Data [16.500568323161563]
  We present practical schemes for implementing the manifold scattering transform to datasets arising in naturalistic systems. We show that our methods are effective for signal classification and manifold classification tasks.
  arXiv  Detail & Related papers  (2022-06-21T02:15:00Z)
• On Hypothesis Transfer Learning of Functional Linear Models [8.557392136621894]
  We study the transfer learning (TL) for the functional linear regression (FLR) under the Reproducing Kernel Space (RKHS) framework. We measure the similarity across tasks using RKHS distance, allowing the type of information being transferred tied to the properties of the imposed RKHS. Two algorithms are proposed: one conducts the transfer when positive sources are known, while the other leverages aggregation to achieve robust transfer without prior information about the sources.
  arXiv  Detail & Related papers  (2022-06-09T04:50:16Z)
• Geometry-Contrastive Transformer for Generalized 3D Pose Transfer [95.56457218144983]
  The intuition of this work is to perceive the geometric inconsistency between the given meshes with the powerful self-attention mechanism. We propose a novel geometry-contrastive Transformer that has an efficient 3D structured perceiving ability to the global geometric inconsistencies. We present a latent isometric regularization module together with a novel semi-synthesized dataset for the cross-dataset 3D pose transfer task.
  arXiv  Detail & Related papers  (2021-12-14T13:14:24Z)
• An Information-Geometric Distance on the Space of Tasks [31.359578768463752]
  This paper prescribes a distance between learning tasks modeled as joint distributions on data and labels. We develop an algorithm to compute the distance which iteratively transports the marginal on the data of the source task to that of the target task. We perform thorough empirical validation and analysis across diverse image classification datasets to show that the coupled transfer distance correlates strongly with the difficulty of fine-tuning.
  arXiv  Detail & Related papers  (2020-11-01T19:48:39Z)
• Spatial Pyramid Based Graph Reasoning for Semantic Segmentation [67.47159595239798]
  We apply graph convolution into the semantic segmentation task and propose an improved Laplacian. The graph reasoning is directly performed in the original feature space organized as a spatial pyramid. We achieve comparable performance with advantages in computational and memory overhead.
  arXiv  Detail & Related papers  (2020-03-23T12:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.