Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads

• URL: http://arxiv.org/abs/2608.11661v1
• Date: Wed, 12 Aug 2026 05:09:44 GMT
• Title: Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads
• Abstract summary: A multiplicative dual-encoder network computes a real-valued output for a pair of inputs as the inner product of their separate encodings.<n>No unified theory guides the basic design decisions.<n>We show that normalization is gauge fixing and that whitening pins the interaction modes up to permutation and sign.
• Score: 5.703301837612397
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A multiplicative dual-encoder network computes a real-valued output for a pair of inputs as the inner product of their separate encodings. This architecture has been developed independently in operator learning, bipartite matching, contrastive vision-language models, retrieval, and other areas, yet no unified theory guides the basic design decisions: how many interaction modes to represent, how to normalize the encoders, and when the architecture should be avoided. We provide such a foundation by introducing the class of functions of low interaction rank, a class whose intrinsic complexity is measured by its interaction spectrum. Within this framework, approximation error decomposes into a spectral truncation term and an encoder-realization term; sample complexity is governed by the sum of the two encoder complexities rather than their product; and a usability criterion based on spectral decay determines when the architecture can succeed. The same framework exposes a central identifiability problem: the encoders are defined only up to a linear gauge symmetry that leaves the learned coordinates arbitrary. We show that normalization is gauge fixing and that whitening pins the interaction modes up to permutation and sign, thereby explaining the uninterpretability of contrastive dimensions and providing a constructive remedy. Experiments on synthetic kernels, operator learning, and CLIP models validate the theoretical predictions: spectral decay rates match the predicted scaling, whitening recovers the true modes, and independently trained CLIP models are related by a single rotation which, after removal by whitening, exposes interpretable concept axes. The code of this paper is provided at https://github.com/RS2002/Mul-Net .

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• Reconstruction of detector error model for quantum error correction [6.505803823301456]
  We introduce a globally consistent framework to invert experimental syndrome statistics directly into discrete physical hypergraphs.<n>We show that exact continuous parameter extraction in dense codes is limited by a textitvariance cascade, where absolute statistical variance accumulates linearly from high to low-degree mechanisms.<n>This provides a practical approach for characterizing and decoding highly correlated noise in realistic quantum hardware.
  arXiv  Detail & Related papers  (2026-06-15T06:51:48Z)
• Selective Synergistic Learning for Video Object-Centric Learning [43.27149574230564]
  We propose Selective Synergistic Learning (SSync) for slot-based object-centric learning.<n>SSync prevents patchtopatch alignment by error propagation by only the most reliable cues.<n>Also, SSync improves decomposition quality and serves as a versatile, plugand-play module.
  arXiv  Detail & Related papers  (2026-06-14T00:51:33Z)
• Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning [56.24532075189964]
  We propose Zeta, a dual whitening that applies coordinate whitening and spectral whitening in a strictly ordered pipeline.<n>We prove that Zeta matches or surpasses strong baselines across language modeling (0.6B to 8B parameters), mixture-of-experts architectures, and vision tasks.
  arXiv  Detail & Related papers  (2026-06-12T07:10:17Z)
• Encoding Numerical Data for Generative Quantum Machine Learning [0.820994582033002]
  Real-world data is often numeric, requiring models to translate between binary and continuous representations.<n>We show that it requires the models to learn correlations that are solely an artifact of the way the data is encoded, and not related to the data itself.<n>We propose a strategy based on Gray-codes that can be implemented with essentially no overhead, conserves structures in the data, and avoids artificial correlations in situations in which the standard approach creates them.
  arXiv  Detail & Related papers  (2026-03-24T16:39:58Z)
• CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphs [13.488544043942495]
  We aim to investigate whether the model's neural dynamics encode internally decodable signals that are predictive of logical validity during code generation.<n>By decomposing complex residual flows, we aim to identify the structural signatures that distinguish sound reasoning from logical failure.<n>Analysis across Python, C++, and Java confirms that intrinsic correctness signals are robust across diverse syntaxes.
  arXiv  Detail & Related papers  (2026-02-06T03:49:15Z)
• TRACE: Learning to Compute on Graphs [15.34239150750753]
  We introduce textbfTRACE, a new paradigm built on an architecturally sound backbone and a principled learning objective.<n>First, TRACE employs a Hierarchical Transformer that mirrors the step-by-step flow of computation.<n>Second, we introduce textbffunction shift learning, a novel objective that decouples the learning problem.
  arXiv  Detail & Related papers  (2025-09-26T05:22:32Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• X-volution: On the unification of convolution and self-attention [52.80459687846842]
  We propose a multi-branch elementary module composed of both convolution and self-attention operation. The proposed X-volution achieves highly competitive visual understanding improvements.
  arXiv  Detail & Related papers  (2021-06-04T04:32:02Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.