On the Equivalence of the Weighted Tsetlin Machine and the Perceptron

• URL: http://arxiv.org/abs/2212.13634v1
• Date: Tue, 27 Dec 2022 22:38:59 GMT
• Title: On the Equivalence of the Weighted Tsetlin Machine and the Perceptron
• Authors: Jivitesh Sharma, Ole-Christoffer Granmo and Lei Jiao
• Abstract summary: Tsetlin Machine (TM) has been gaining popularity as an inherently interpretable machine leaning method. Although possessing favorable properties, TM has not been the go-to method for AI applications.
• Score: 12.48513712803069
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tsetlin Machine (TM) has been gaining popularity as an inherently interpretable machine leaning method that is able to achieve promising performance with low computational complexity on a variety of applications. The interpretability and the low computational complexity of the TM are inherited from the Boolean expressions for representing various sub-patterns. Although possessing favorable properties, TM has not been the go-to method for AI applications, mainly due to its conceptual and theoretical differences compared with perceptrons and neural networks, which are more widely known and well understood. In this paper, we provide detailed insights for the operational concept of the TM, and try to bridge the gap in the theoretical understanding between the perceptron and the TM. More specifically, we study the operational concept of the TM following the analytical structure of perceptrons, showing the resemblance between the perceptrons and the TM. Through the analysis, we indicated that the TM's weight update can be considered as a special case of the gradient weight update. We also perform an empirical analysis of TM by showing the flexibility in determining the clause length, visualization of decision boundaries and obtaining interpretable boolean expressions from TM. In addition, we also discuss the advantages of TM in terms of its structure and its ability to solve more complex problems.

Related papers

• Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning [53.685764040547625]
  Transformer-based large language models (LLMs) have displayed remarkable creative prowess and emergence capabilities. This work provides a fine mathematical analysis to show how transformers leverage the multi-concept semantics of words to enable powerful ICL and excellent out-of-distribution ICL abilities.
  arXiv  Detail & Related papers  (2024-11-04T15:54:32Z)
• The Probabilistic Tsetlin Machine: A Novel Approach to Uncertainty Quantification [1.0499611180329802]
  This paper introduces the Probabilistic Tsetlin Machine (PTM) framework, aimed at providing a robust, reliable, and interpretable approach for uncertainty quantification. Unlike the original TM, the PTM learns the probability of staying on each state of each Tsetlin Automaton (TA) across all clauses. During inference, TAs decide their actions by sampling states based on learned probability distributions.
  arXiv  Detail & Related papers  (2024-10-23T13:20:42Z)
• Understanding Self-attention Mechanism via Dynamical System Perspective [58.024376086269015]
  Self-attention mechanism (SAM) is widely used in various fields of artificial intelligence. We show that intrinsic stiffness phenomenon (SP) in the high-precision solution of ordinary differential equations (ODEs) also widely exists in high-performance neural networks (NN) We show that the SAM is also a stiffness-aware step size adaptor that can enhance the model's representational ability to measure intrinsic SP.
  arXiv  Detail & Related papers  (2023-08-19T08:17:41Z)
• Understanding Masked Autoencoders via Hierarchical Latent Variable Models [109.35382136147349]
  Masked autoencoder (MAE) has recently achieved prominent success in a variety of vision tasks. Despite the emergence of intriguing empirical observations on MAE, a theoretically principled understanding is still lacking.
  arXiv  Detail & Related papers  (2023-06-08T03:00:10Z)
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning [55.048010996144036]
  We show that under some noise assumption, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free. We propose Spectral Dynamics Embedding (SPEDE), which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise.
  arXiv  Detail & Related papers  (2021-11-22T19:24:57Z)
• Pre-Trained Models: Past, Present and Future [126.21572378910746]
  Large-scale pre-trained models (PTMs) have recently achieved great success and become a milestone in the field of artificial intelligence (AI) By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch.
  arXiv  Detail & Related papers  (2021-06-14T02:40:32Z)
• Distributed Word Representation in Tsetlin Machine [14.62945824459286]
  Tsetlin Machine (TM) is an interpretable pattern recognition algorithm based on propositional logic. We propose a novel way of using pre-trained word representations for TM. The approach significantly enhances TM performance and maintains interpretability at the same time.
  arXiv  Detail & Related papers  (2021-04-14T14:48:41Z)
• On the Convergence of Tsetlin Machines for the XOR Operator [16.85020149902772]
  The Tsetlin Machine (TM) is a novel machine learning algorithm with several distinct properties. We analyze the convergence of the TM when input is non-linearly related to output by the XOR-operator.
  arXiv  Detail & Related papers  (2021-01-07T14:13:41Z)
• Generalization Properties of Optimal Transport GANs with Latent Distribution Learning [52.25145141639159]
  We study how the interplay between the latent distribution and the complexity of the pushforward map affects performance. Motivated by our analysis, we advocate learning the latent distribution as well as the pushforward map within the GAN paradigm.
  arXiv  Detail & Related papers  (2020-07-29T07:31:33Z)
• On the Convergence of Tsetlin Machines for the IDENTITY- and NOT Operators [14.186377732379045]
  We analyze the convergence of the Tsetlin Machine with only one clause involved for classification. Our analysis reveals that the TM, with just one clause, can converge correctly to the intended logical operator. The analysis of the convergence of the two basic operators lays the foundation for analyzing other logical operators.
  arXiv  Detail & Related papers  (2020-07-28T14:31:16Z)
• Complex Markov Logic Networks: Expressivity and Liftability [10.635097939284751]
  We study expressivity of Markov logic networks (MLNs) We introduce complex MLNs, which use complex-valued weights. We show that, unlike standard MLNs with real-valued weights, complex MLNs are fully expressive.
  arXiv  Detail & Related papers  (2020-02-24T13:50:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.