The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

• URL: http://arxiv.org/abs/2606.16028v1
• Date: Sun, 14 Jun 2026 21:25:58 GMT
• Title: The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints
• Abstract summary: Empirically, exploiting similarity across different problems, instead of solving them individually, can significantly improve overall performance.<n>We prove lower and upper bounds on the description-length for separate and joint approximation classes, respectively, in uniform norm.<n>This provides theoretical insight into the description-length-efficiency of compositional multi-output architectures.
• Score: 4.192948150195333
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models. Empirically, exploiting similarity across different problems, instead of solving them individually, can significantly improve overall performance. While the generalization and sample complexity properties of multitask learning have been widely studied, the parametric complexity of joint approximation in comparison to separate approximation remains less well understood. The question is particularly relevant in modern deep learning, where models are increasingly required to satisfy structural constraints such as equivariance, conservation laws, or orthogonality. We prove lower and upper bounds on the description-length for separate and joint approximation classes, respectively, in uniform norm. We build a class of orthogonal functions by composing a shared hard feature, realized by a Rademacher-Haar wavelet series, with Sawtooth-Walsh readouts to enforce orthogonality of output coordinates. The dyadic tree structure of the Rademacher-Haar wavelet concentrates the approximation hardness in the common feature component, while the readouts act as task-specific heads. Using an information-theoretic framework, we obtain a sharp gap between the optimal approximation rates achievable by joint and separate coding. Finally, we realize this separation in a neural network model using Heaviside activations via reduction to triangle-wave approximation. Our results show that even under an orthogonality constraint joint approximation requires strictly fewer bits in compositional architectures, provided the tasks share a latent hard feature. This provides theoretical insight into the description-length-efficiency of compositional multi-output architectures and clarifies how neural networks can retain expressivity under geometric constraints.

Related papers

• Learning Sparse Compositional Functions with Norm-Constrained Neural Networks [8.967327976469653]
  We show that deep networks can exploit the compositional structure of the target functions, effectively avoiding the curse of dimensionality (CoD)<n>Our results have broad applicability since every function that is efficiently Turing computable admits sparse compositional representations.
  arXiv  Detail & Related papers  (2026-05-25T09:02:13Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• Quantifying Multimodal Capabilities: Formal Generalization Guarantees in Pairwise Metric Learning [16.068927747527606]
  This paper presents a theoretical analysis of the generalization properties of multimodal metric learning models.<n>We establish hierarchical relationships between function classes corresponding to different modality subsets and quantify the discrepancy between learned mappings and ground truth.<n>Our findings on both upper and lower bounds demonstrate that incorporating fine-grained modality features reduces the complexity of the hypothesis space by enhancing modality complementarity.
  arXiv  Detail & Related papers  (2026-05-02T12:44:43Z)
• KVNN: Learnable Multi-Kernel Volterra Neural Networks [6.550684951976901]
  kernelized Volterra Neural Network (kVNN)<n>Experiments on two representative tasks: video action recognition and image denoising.<n>Results are maintained even when trained from scratch without large-scale pretraining.
  arXiv  Detail & Related papers  (2026-04-16T15:18:48Z)
• GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning [51.677086019209554]
  We propose a Generalized Structural Sparse to capture powerful relationships across modalities for pair-wise similarity learning. The distance metric delicately encapsulates two formats of diagonal and block-diagonal terms. Experiments on cross-modal and two extra uni-modal retrieval tasks have validated its superiority and flexibility.
  arXiv  Detail & Related papers  (2024-10-20T03:45:50Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• On Neural Architecture Inductive Biases for Relational Tasks [76.18938462270503]
  We introduce a simple architecture based on similarity-distribution scores which we name Compositional Network generalization (CoRelNet) We find that simple architectural choices can outperform existing models in out-of-distribution generalizations.
  arXiv  Detail & Related papers  (2022-06-09T16:24:01Z)
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning [53.17258888552998]
  This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation. We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
  arXiv  Detail & Related papers  (2022-06-01T23:26:51Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• Approximation Theory of Convolutional Architectures for Time Series Modelling [15.42770933459534]
  We study the approximation properties of convolutional architectures applied to time series modelling. Recent results reveal an intricate connection between approximation efficiency and memory structures in the data generation process.
  arXiv  Detail & Related papers  (2021-07-20T09:19:26Z)
• Robust Submodular Minimization with Applications to Cooperative Modeling [0.0]
  This paper studies the problem of robust submodular Minimization subject to constraints. Constrained Submodular Minimization arises in several applications such as co-operative cuts in image segmentation, co-operative matchings in image correspondence, etc.
  arXiv  Detail & Related papers  (2020-01-25T20:40:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.