Developing a Foundation of Vector Symbolic Architectures Using Category Theory

• URL: http://arxiv.org/abs/2501.05368v2
• Date: Fri, 02 May 2025 18:22:53 GMT
• Title: Developing a Foundation of Vector Symbolic Architectures Using Category Theory
• Authors: Nolan P Shaw, P Michael Furlong, Britt Anderson, Jeff Orchard,
• Abstract summary: We present a first attempt at applying category theory to Vector Symbolic Architectures.<n> VSAs arose from the need to unify neural processing and the kind of symbolic reasoning that humans perform.<n>This formalisation involves a proof that the right Kan extension in such cases can be expressed as simple, element-wise operations.
• Score: 1.293702091862065
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Connectionist approaches to machine learning, \emph{i.e.} neural networks, are enjoying a considerable vogue right now. However, these methods require large volumes of data and produce models that are uninterpretable to humans. An alternative framework that is compatible with neural networks and gradient-based learning, but explicitly models compositionality, is Vector Symbolic Architectures (VSAs). VSAs are a family of algebras on high-dimensional vector representations. They arose in cognitive science from the need to unify neural processing and the kind of symbolic reasoning that humans perform. While machine learning methods have benefited from category-theoretical analyses, VSAs have not yet received similar treatment. In this paper, we present a first attempt at applying category theory to VSAs. Specifically, We generalise from vectors to co-presheaves, and describe VSA operations as the right Kan extensions of the external tensor product. This formalisation involves a proof that the right Kan extension in such cases can be expressed as simple, element-wise operations. We validate our formalisation with worked examples that connect to current VSA implementations, while suggesting new possible designs for VSAs.

Related papers

• Fast and Accurate Explanations of Distance-Based Classifiers by Uncovering Latent Explanatory Structures [13.600836585770134]
  We contribute by uncovering a hidden neural network structure in distance-based classifiers.<n>We show the overall usefulness of explaining distance-based models through two practical use cases.
  arXiv  Detail & Related papers  (2025-08-05T21:01:58Z)
• A Walsh Hadamard Derived Linear Vector Symbolic Architecture [83.27945465029167]
  Symbolic Vector Architectures (VSAs) are an approach to developing Neuro-symbolic AI. HLB is designed to have favorable computational efficiency, and efficacy in classic VSA tasks.
  arXiv  Detail & Related papers  (2024-10-30T03:42:59Z)
• Towards a Categorical Foundation of Deep Learning: A Survey [0.0]
  This thesis is a survey that covers some recent work attempting to study machine learning categorically. acting as a lingua franca of mathematics and science, category theory might be able to give a unifying structure to the field of machine learning.
  arXiv  Detail & Related papers  (2024-10-07T13:11:16Z)
• Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures [22.12114509953737]
  Abstract reasoning is a cornerstone of human intelligence, and replicating it with artificial intelligence (AI) presents an ongoing challenge. This study focuses on efficiently solving Raven's progressive matrices (RPM), a visual test for assessing abstract reasoning abilities. Instead of hard-coding the rule formulations associated with RPMs, our approach can learn the VSA rule formulations with just one pass through the training data.
  arXiv  Detail & Related papers  (2024-01-29T10:17:18Z)
• Meaning Representations from Trajectories in Autoregressive Models [106.63181745054571]
  We propose to extract meaning representations from autoregressive language models by considering the distribution of all possible trajectories extending an input text. This strategy is prompt-free, does not require fine-tuning, and is applicable to any pre-trained autoregressive model. We empirically show that the representations obtained from large models align well with human annotations, outperform other zero-shot and prompt-free methods on semantic similarity tasks, and can be used to solve more complex entailment and containment tasks that standard embeddings cannot handle.
  arXiv  Detail & Related papers  (2023-10-23T04:35:58Z)
• A Recursive Bateson-Inspired Model for the Generation of Semantic Formal Concepts from Spatial Sensory Data [77.34726150561087]
  This paper presents a new symbolic-only method for the generation of hierarchical concept structures from complex sensory data. The approach is based on Bateson's notion of difference as the key to the genesis of an idea or a concept. The model is able to produce fairly rich yet human-readable conceptual representations without training.
  arXiv  Detail & Related papers  (2023-07-16T15:59:13Z)
• Parametric Classification for Generalized Category Discovery: A Baseline Study [70.73212959385387]
  Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. We investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers.
  arXiv  Detail & Related papers  (2022-11-21T18:47:11Z)
• Neural Dependencies Emerging from Learning Massive Categories [94.77992221690742]
  This work presents two astonishing findings on neural networks learned for large-scale image classification. 1) Given a well-trained model, the logits predicted for some category can be directly obtained by linearly combining the predictions of a few other categories. 2) Neural dependencies exist not only within a single model, but even between two independently learned models.
  arXiv  Detail & Related papers  (2022-11-21T09:42:15Z)
• Learning Primitive-aware Discriminative Representations for Few-shot Learning [28.17404445820028]
  Few-shot learning aims to learn a classifier that can be easily adapted to recognize novel classes with only a few labeled examples. We propose a Primitive Mining and Reasoning Network (PMRN) to learn primitive-aware representations. Our method achieves state-of-the-art results on six standard benchmarks.
  arXiv  Detail & Related papers  (2022-08-20T16:22:22Z)
• Large-Margin Representation Learning for Texture Classification [67.94823375350433]
  This paper presents a novel approach combining convolutional layers (CLs) and large-margin metric learning for training supervised models on small datasets for texture classification. The experimental results on texture and histopathologic image datasets have shown that the proposed approach achieves competitive accuracy with lower computational cost and faster convergence when compared to equivalent CNNs.
  arXiv  Detail & Related papers  (2022-06-17T04:07:45Z)
• Fair Interpretable Representation Learning with Correction Vectors [60.0806628713968]
  We propose a new framework for fair representation learning that is centered around the learning of "correction vectors" We show experimentally that several fair representation learning models constrained in such a way do not exhibit losses in ranking or classification performance.
  arXiv  Detail & Related papers  (2022-02-07T11:19:23Z)
• Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning [109.21780441933164]
  We propose a hybrid approach to improve systematic generalization in reasoning. We showcase a prototype with algebraic representation for the abstract spatial-temporal task of Raven's Progressive Matrices (RPM) We show that the algebraic representation learned can be decoded by isomorphism to generate an answer.
  arXiv  Detail & Related papers  (2021-11-25T09:56:30Z)
• HyperSeed: Unsupervised Learning with Vector Symbolic Architectures [5.258404928739212]
  This paper presents a novel unsupervised machine learning approach named Hyperseed. It leverages Vector Symbolic Architectures (VSA) for fast learning a topology preserving feature map of unlabelled data. The two distinctive novelties of the Hyperseed algorithm are 1) Learning from only few input data samples and 2) A learning rule based on a single vector operation.
  arXiv  Detail & Related papers  (2021-10-15T20:05:43Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Variable Binding for Sparse Distributed Representations: Theory and Applications [4.150085009901543]
  Symbolic reasoning and neural networks are often considered incompatible approaches. Connectionist models known as Vector Symbolic Architectures (VSAs) can potentially bridge this gap. VSAs encode symbols by dense pseudo-random vectors, where information is distributed throughout the entire neuron population. We show that variable binding between dense vectors in VSAs is mathematically equivalent to tensor product binding between sparse vectors, an operation which increases dimensionality.
  arXiv  Detail & Related papers  (2020-09-14T20:40:09Z)
• Gradient-based Competitive Learning: Theory [1.6752712949948443]
  This paper introduces a novel perspective in this area by combining gradient-based and competitive learning. The theory is based on the intuition that neural networks are able to learn topological structures by working directly on the transpose of the input matrix. The proposed approach has a great potential as it can be generalized to a vast selection of topological learning tasks.
  arXiv  Detail & Related papers  (2020-09-06T19:00:51Z)
• Symbiotic Adversarial Learning for Attribute-based Person Search [86.7506832053208]
  We present a symbiotic adversarial learning framework, called SAL.Two GANs sit at the base of the framework in a symbiotic learning scheme. Specifically, two different types of generative adversarial networks learn collaboratively throughout the training process.
  arXiv  Detail & Related papers  (2020-07-19T07:24:45Z)
• Pairwise Supervision Can Provably Elicit a Decision Boundary [84.58020117487898]
  Similarity learning is a problem to elicit useful representations by predicting the relationship between a pair of patterns. We show that similarity learning is capable of solving binary classification by directly eliciting a decision boundary.
  arXiv  Detail & Related papers  (2020-06-11T05:35:16Z)
• Analyzing Knowledge Graph Embedding Methods from a Multi-Embedding Interaction Perspective [3.718476964451589]
  Real-world knowledge graphs are usually incomplete, so knowledge graph embedding methods have been proposed to address this issue. These methods represent entities and relations as embedding vectors in semantic space and predict the links between them. We propose a new multi-embedding model based on quaternion algebra and show that it achieves promising results using popular benchmarks.
  arXiv  Detail & Related papers  (2019-03-27T13:09:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.