Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures

• URL: http://arxiv.org/abs/2403.13218v1
• Date: Wed, 20 Mar 2024 00:37:19 GMT
• Title: Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures
• Authors: Calvin Yeung, Prathyush Poduval, Mohsen Imani,
• Abstract summary: We introduce a new variant of the resonator network, based on self-attention based update rules in iterative search problem. Our algorithm enables a larger capacity for associative memory, enabling applications in many tasks like perception based pattern recognition, scene decomposition, and object reasoning.
• Score: 6.473177443214531
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vector Symbolic Architectures (VSAs) have emerged as a novel framework for enabling interpretable machine learning algorithms equipped with the ability to reason and explain their decision processes. The basic idea is to represent discrete information through high dimensional random vectors. Complex data structures can be built up with operations over vectors such as the "binding" operation involving element-wise vector multiplication, which associates data together. The reverse task of decomposing the associated elements is a combinatorially hard task, with an exponentially large search space. The main algorithm for performing this search is the resonator network, inspired by Hopfield network-based memory search operations. In this work, we introduce a new variant of the resonator network, based on self-attention based update rules in the iterative search problem. This update rule, based on the Hopfield network with log-sum-exp energy function and norm-bounded states, is shown to substantially improve the performance and rate of convergence. As a result, our algorithm enables a larger capacity for associative memory, enabling applications in many tasks like perception based pattern recognition, scene decomposition, and object reasoning. We substantiate our algorithm with a thorough evaluation and comparisons to baselines.

Related papers

• RESOLVE: Relational Reasoning with Symbolic and Object-Level Features Using Vector Symbolic Processing [1.3049516752695616]
  We propose RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces. By leveraging this design, the model achieves both low compute latency and memory efficiency.
  arXiv  Detail & Related papers  (2024-11-13T02:17:03Z)
• Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations [15.59251297818324]
  We present an approach for analyzing grouping information contained within a neural network's activations. We exploit features from all layers and obviating the need to guess which part of the model contains relevant information.
  arXiv  Detail & Related papers  (2023-12-11T01:20:34Z)
• An Efficient Algorithm for Clustered Multi-Task Compressive Sensing [60.70532293880842]
  Clustered multi-task compressive sensing is a hierarchical model that solves multiple compressive sensing tasks. The existing inference algorithm for this model is computationally expensive and does not scale well in high dimensions. We propose a new algorithm that substantially accelerates model inference by avoiding the need to explicitly compute these covariance matrices.
  arXiv  Detail & Related papers  (2023-09-30T15:57:14Z)
• Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement [75.9289887536165]
  We present a hierarchical abstraction approach to uncover underlying entities. We show how to learn a correspondence between intervening on states of entities in the agent's model and acting on objects in the environment. We use this correspondence to develop a method for control that generalizes to different numbers and configurations of objects.
  arXiv  Detail & Related papers  (2023-03-20T18:19:36Z)
• Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation [102.25240608024063]
  Referring image segments an image from a language expression. We develop an algorithm that shifts from being localization-centric to segmentation-language. Compared to its counterparts, our method is more versatile yet effective.
  arXiv  Detail & Related papers  (2023-03-11T08:42:40Z)
• A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms [64.3064050603721]
  We generalize Runge-Kutta neural network to a recurrent neural network (R2N2) superstructure for the design of customized iterative algorithms. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields similar iterations to Krylov solvers for linear equation systems, Newton-Krylov solvers for nonlinear equation systems, and Runge-Kutta solvers for ordinary differential equations.
  arXiv  Detail & Related papers  (2022-11-22T16:30:33Z)
• Deep Equilibrium Assisted Block Sparse Coding of Inter-dependent Signals: Application to Hyperspectral Imaging [71.57324258813675]
  A dataset of inter-dependent signals is defined as a matrix whose columns demonstrate strong dependencies. A neural network is employed to act as structure prior and reveal the underlying signal interdependencies. Deep unrolling and Deep equilibrium based algorithms are developed, forming highly interpretable and concise deep-learning-based architectures.
  arXiv  Detail & Related papers  (2022-03-29T21:00:39Z)
• Sparsely ensembled convolutional neural network classifiers via reinforcement learning [0.0]
  We consider convolutional neural network (CNN) ensemble learning with the objective function inspired by least action principle. We teach an agent to perceive images through the set of pre-trained classifiers and want the resulting dynamically configured system to unfold the computational graph. Our experimental results prove, that if the agent exploits the dynamic (and context-dependent) structure of computations, it outperforms conventional ensemble learning.
  arXiv  Detail & Related papers  (2021-02-07T21:26:57Z)
• Towards Efficient Scene Understanding via Squeeze Reasoning [71.1139549949694]
  We propose a novel framework called Squeeze Reasoning. Instead of propagating information on the spatial map, we first learn to squeeze the input feature into a channel-wise global vector. We show that our approach can be modularized as an end-to-end trained block and can be easily plugged into existing networks.
  arXiv  Detail & Related papers  (2020-11-06T12:17:01Z)
• Resonator networks for factoring distributed representations of data structures [3.46969645559477]
  We show how data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. Re resonator networks open the possibility to apply VSAs to myriad artificial intelligence problems in real-world domains.
  arXiv  Detail & Related papers  (2020-07-07T19:24:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.