A Walsh Hadamard Derived Linear Vector Symbolic Architecture

• URL: http://arxiv.org/abs/2410.22669v1
• Date: Wed, 30 Oct 2024 03:42:59 GMT
• Title: A Walsh Hadamard Derived Linear Vector Symbolic Architecture
• Authors: Mohammad Mahmudul Alam, Alexander Oberle, Edward Raff, Stella Biderman, Tim Oates, James Holt,
• Abstract summary: Symbolic Vector Architectures (VSAs) are an approach to developing Neuro-symbolic AI. HLB is designed to have favorable computational efficiency, and efficacy in classic VSA tasks.
• Score: 83.27945465029167
• License:
• Abstract: Vector Symbolic Architectures (VSAs) are one approach to developing Neuro-symbolic AI, where two vectors in $\mathbb{R}^d$ are `bound' together to produce a new vector in the same space. VSAs support the commutativity and associativity of this binding operation, along with an inverse operation, allowing one to construct symbolic-style manipulations over real-valued vectors. Most VSAs were developed before deep learning and automatic differentiation became popular and instead focused on efficacy in hand-designed systems. In this work, we introduce the Hadamard-derived linear Binding (HLB), which is designed to have favorable computational efficiency, and efficacy in classic VSA tasks, and perform well in differentiable systems. Code is available at https://github.com/FutureComputing4AI/Hadamard-derived-Linear-Binding

Related papers

• Knowledge Composition using Task Vectors with Learned Anisotropic Scaling [51.4661186662329]
  We introduce aTLAS, an algorithm that linearly combines parameter blocks with different learned coefficients, resulting in anisotropic scaling at the task vector level. We show that such linear combinations explicitly exploit the low intrinsicity of pre-trained models, with only a few coefficients being the learnable parameters. We demonstrate the effectiveness of our method in task arithmetic, few-shot recognition and test-time adaptation, with supervised or unsupervised objectives.
  arXiv  Detail & Related papers  (2024-07-03T07:54:08Z)
• Householder Projector for Unsupervised Latent Semantics Discovery [58.92485745195358]
  Householder Projector helps StyleGANs to discover more disentangled and precise semantic attributes without sacrificing image fidelity. We integrate our projector into pre-trained StyleGAN2/StyleGAN3 and evaluate the models on several benchmarks.
  arXiv  Detail & Related papers  (2023-07-16T11:43:04Z)
• Understanding Hyperdimensional Computing for Parallel Single-Pass Learning [47.82940409267635]
  We show that HDC can outperform the state-of-the-art HDC model by up to 7.6% while maintaining hardware efficiency. We propose a new class of VSAs, finite group VSAs, which surpass the limits of HDC. Experimental results show that our RFF method and group VSA can both outperform the state-of-the-art HDC model by up to 7.6%.
  arXiv  Detail & Related papers  (2022-02-10T02:38:56Z)
• Fast Differentiable Matrix Square Root and Inverse Square Root [65.67315418971688]
  We propose two more efficient variants to compute the differentiable matrix square root and the inverse square root. For the forward propagation, one method is to use Matrix Taylor Polynomial (MTP), and the other method is to use Matrix Pad'e Approximants (MPA) A series of numerical tests show that both methods yield considerable speed-up compared with the SVD or the NS iteration.
  arXiv  Detail & Related papers  (2022-01-29T10:00:35Z)
• HyperSeed: Unsupervised Learning with Vector Symbolic Architectures [5.258404928739212]
  This paper presents a novel unsupervised machine learning approach named Hyperseed. It leverages Vector Symbolic Architectures (VSA) for fast learning a topology preserving feature map of unlabelled data. The two distinctive novelties of the Hyperseed algorithm are 1) Learning from only few input data samples and 2) A learning rule based on a single vector operation.
  arXiv  Detail & Related papers  (2021-10-15T20:05:43Z)
• Learning with Holographic Reduced Representations [28.462635977110413]
  Holographic Reduced Representations (HRR) are a method for performing symbolic AI on top of real-valued vectors. This paper revisits this approach to understand if it is viable for enabling a hybrid neural-symbolic approach to learning.
  arXiv  Detail & Related papers  (2021-09-05T19:37:34Z)
• A Differential Geometry Perspective on Orthogonal Recurrent Models [56.09491978954866]
  We employ tools and insights from differential geometry to offer a novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Motivated by this observation, we study a new recurrent model, which spans the entire space of vector fields.
  arXiv  Detail & Related papers  (2021-02-18T19:39:22Z)
• Variable Binding for Sparse Distributed Representations: Theory and Applications [4.150085009901543]
  Symbolic reasoning and neural networks are often considered incompatible approaches. Connectionist models known as Vector Symbolic Architectures (VSAs) can potentially bridge this gap. VSAs encode symbols by dense pseudo-random vectors, where information is distributed throughout the entire neuron population. We show that variable binding between dense vectors in VSAs is mathematically equivalent to tensor product binding between sparse vectors, an operation which increases dimensionality.
  arXiv  Detail & Related papers  (2020-09-14T20:40:09Z)
• Tensor Relational Algebra for Machine Learning System Design [7.764107702934616]
  We present an alternative implementation abstraction called the relational tensor algebra (TRA) TRA is a set-based algebra based on the relational algebra. Our empirical study shows that the optimized TRA-based back-end can significantly outperform alternatives for running ML in distributed clusters.
  arXiv  Detail & Related papers  (2020-09-01T15:51:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.