Disentanglement with Holographic Reduced Representations

• URL: http://arxiv.org/abs/2606.09725v1
• Date: Mon, 08 Jun 2026 16:48:35 GMT
• Title: Disentanglement with Holographic Reduced Representations
• Abstract summary: Disentanglement, the separation of factors of variation in data using neural networks, remains a long-standing challenge in machine learning.<n>We introduce an unsupervised learning algorithm that uses holographic reduced representations (HRR) for neural disentanglement.<n>We show that the HRR unbinding operation provides an inductive bias for separating factors and yields competitive results against baselines.
• Score: 36.48134953186487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Disentanglement, the separation of factors of variation in data using neural networks, remains a long-standing challenge in machine learning. Prior work has addressed this problem with variational autoencoders and generative adversarial networks that incorporate ideas from variational inference and information-theoretic constraints. In contrast to methods that rely on continuous representations, we propose a design that treats disentangled representations as symbolic structures, motivated by the compositional relationships among the concepts that make up samples from a distribution. However, learning discrete symbolic structures with neural networks while maintaining differentiability is difficult and often requires complex architectures. To address this, we introduce an unsupervised learning algorithm that uses holographic reduced representations (HRR) for neural disentanglement. We show that the HRR unbinding operation provides an inductive bias for separating factors and yields competitive results against baselines, as measured by latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts can be reliably encoded, giving a quantitative account of the inductive bias toward disentanglement. The resulting representations differ from standard autoencoder-based models, in that their latent units are vectors that are summed together, rather than scalar dimensions of a low-dimensional latent vector. We show that this HRR representation is more robust to noise than other disentangled representations and maintains reconstruction quality across a range of SNRs.

Related papers

• Disentangled Representation Learning via Flow Matching [48.12507436294143]
  Disentangled representation learning aims to capture the underlying explanatory factors of observed data.<n>Existing diffusion-based methods encourage factor independence via inductive biases, yet frequently lack strong semantic alignment.<n>We propose a flow matching-based framework for disentangled representation learning, which casts disentanglement as learning factor-conditioned flows in a compact latent space.
  arXiv  Detail & Related papers  (2026-02-05T02:14:36Z)
• Twinning Complex Networked Systems: Data-Driven Calibration of the mABCD Synthetic Graph Generator [2.6776012440607784]
  We propose a method for estimating matching configurations and for quantifying the associated error.<n>Our results demonstrate that this task is non-trivial, as strong interdependencies between configuration parameters weaken independent estimation and instead favour a joint-prediction approach.
  arXiv  Detail & Related papers  (2026-02-02T12:40:19Z)
• Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement [58.9768112704998]
  Disentangled representation learning strives to extract the intrinsic factors within observed data. We introduce a new perspective and framework, demonstrating that diffusion models with cross-attention can serve as a powerful inductive bias. This is the first work to reveal the potent disentanglement capability of diffusion models with cross-attention, requiring no complex designs.
  arXiv  Detail & Related papers  (2024-02-15T05:07:54Z)
• Labeling Neural Representations with Inverse Recognition [25.867702786273586]
  Inverse Recognition (INVERT) is a scalable approach for connecting learned representations with human-understandable concepts. In contrast to prior work, INVERT is capable of handling diverse types of neurons, exhibits less computational complexity, and does not rely on the availability of segmentation masks. We demonstrate the applicability of INVERT in various scenarios, including the identification of representations affected by spurious correlations.
  arXiv  Detail & Related papers  (2023-11-22T18:55:25Z)
• Learning Disentangled Discrete Representations [22.5004558029479]
  We show the relationship between discrete latent spaces and disentangled representations by replacing the standard Gaussian variational autoencoder with a tailored categorical variational autoencoder. We provide both analytical and empirical findings that demonstrate the advantages of discrete VAEs for learning disentangled representations.
  arXiv  Detail & Related papers  (2023-07-26T12:29:58Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Structure-preserving GANs [6.438897276587413]
  We introduce structure-preserving GANs as a data-efficient framework for learning distributions. We show that we can reduce the discriminator space to its projection on the invariant discriminator space. We contextualize our framework by building symmetry-preserving GANs for distributions with intrinsic group symmetry.
  arXiv  Detail & Related papers  (2022-02-02T16:40:04Z)
• NOTMAD: Estimating Bayesian Networks with Sample-Specific Structures and Parameters [70.55488722439239]
  We present NOTMAD, which learns to mix archetypal networks according to sample context. We demonstrate the utility of NOTMAD and sample-specific network inference through analysis and experiments, including patient-specific gene expression networks.
  arXiv  Detail & Related papers  (2021-11-01T17:17:34Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Mixture Representation Learning with Coupled Autoencoders [1.589915930948668]
  We propose an unsupervised variational framework using multiple interacting networks called cpl-mixVAE. In this framework, the mixture representation of each network is regularized by imposing a consensus constraint on the discrete factor. We use the proposed method to jointly uncover discrete and continuous factors of variability describing gene expression in a single-cell transcriptomic dataset.
  arXiv  Detail & Related papers  (2020-07-20T04:12:04Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.