$\boldsymbolλ$-Orthogonality Regularization for Compatible Representation Learning

• URL: http://arxiv.org/abs/2509.16664v2
• Date: Mon, 20 Oct 2025 18:17:23 GMT
• Title: $\boldsymbolλ$-Orthogonality Regularization for Compatible Representation Learning
• Authors: Simone Ricci, Niccolò Biondi, Federico Pernici, Ioannis Patras, Alberto Del Bimbo,
• Abstract summary: Retrieval systems rely on representations learned by increasingly powerful models.<n>Due to the high training cost and inconsistencies in learned representations, there is significant interest in facilitating communication between representations.
• Score: 48.264642951728085
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval systems rely on representations learned by increasingly powerful models. However, due to the high training cost and inconsistencies in learned representations, there is significant interest in facilitating communication between representations and ensuring compatibility across independently trained neural networks. In the literature, two primary approaches are commonly used to adapt different learned representations: affine transformations, which adapt well to specific distributions but can significantly alter the original representation, and orthogonal transformations, which preserve the original structure with strict geometric constraints but limit adaptability. A key challenge is adapting the latent spaces of updated models to align with those of previous models on downstream distributions while preserving the newly learned representation spaces. In this paper, we impose a relaxed orthogonality constraint, namely $\lambda$-Orthogonality regularization, while learning an affine transformation, to obtain distribution-specific adaptation while retaining the original learned representations. Extensive experiments across various architectures and datasets validate our approach, demonstrating that it preserves the model's zero-shot performance and ensures compatibility across model updates. Code available at: \href{https://github.com/miccunifi/lambda_orthogonality.git}{https://github.com/miccunifi/lambda\_orthogonality}.

Related papers

• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)
• Invariance Pair-Guided Learning: Enhancing Robustness in Neural Networks [0.0]
  We propose a technique to guide the neural network through the training phase.<n>We form a corrective gradient complementing the traditional gradient descent approach.<n>Experiments on ColoredMNIST, Waterbird-100, and CelebANIST datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-02-26T09:36:00Z)
• Equivariant Adaptation of Large Pretrained Models [20.687626756753563]
  We show that a canonicalization network can effectively be used to make a large pretrained network equivariant. Using dataset-dependent priors to inform the canonicalization function, we are able to make large pretrained models equivariant while maintaining their performance.
  arXiv  Detail & Related papers  (2023-10-02T21:21:28Z)
• Spatially-varying Regularization with Conditional Transformer for Unsupervised Image Registration [11.498623409184225]
  We introduce an end-to-end framework that uses neural networks to learn a deformation regularizer directly from data. The proposed method is built upon a Transformer-based model, but it can be readily adapted to any network architecture.
  arXiv  Detail & Related papers  (2023-03-10T19:11:16Z)
• Equivariance with Learned Canonicalization Functions [77.32483958400282]
  We show that learning a small neural network to perform canonicalization is better than using predefineds. Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
  arXiv  Detail & Related papers  (2022-11-11T21:58:15Z)
• Frame Averaging for Equivariant Shape Space Learning [85.42901997467754]
  A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries. We present a framework for incorporating equivariance in encoders and decoders by introducing two contributions.
  arXiv  Detail & Related papers  (2021-12-03T06:41:19Z)
• HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning [74.76431541169342]
  Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. We propose a novel hierarchical semantic-visual adaptation (HSVA) framework to align semantic and visual domains. Experiments on four benchmark datasets demonstrate HSVA achieves superior performance on both conventional and generalized ZSL.
  arXiv  Detail & Related papers  (2021-09-30T14:27:50Z)
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction [86.94309120789396]
  We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks.
  arXiv  Detail & Related papers  (2021-06-06T21:53:54Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.