Extending the Tsetlin Machine With Integer-Weighted Clauses for Increased Interpretability

• URL: http://arxiv.org/abs/2005.05131v1
• Date: Mon, 11 May 2020 14:18:09 GMT
• Title: Extending the Tsetlin Machine With Integer-Weighted Clauses for Increased Interpretability
• Authors: K. Darshana Abeyrathna, Ole-Christoffer Granmo, Morten Goodwin
• Abstract summary: Building machine models that are both interpretable and accurate is an unresolved challenge for many pattern recognition problems. Using a linear combination of conjunctive clauses in propositional logic, Tsetlin Machines (TMs) have shown competitive performance on diverse benchmarks. Here, we address the accuracy-interpretability challenge by equipping the TM clauses with integer weights.
• Score: 9.432068833600884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite significant effort, building models that are both interpretable and accurate is an unresolved challenge for many pattern recognition problems. In general, rule-based and linear models lack accuracy, while deep learning interpretability is based on rough approximations of the underlying inference. Using a linear combination of conjunctive clauses in propositional logic, Tsetlin Machines (TMs) have shown competitive performance on diverse benchmarks. However, to do so, many clauses are needed, which impacts interpretability. Here, we address the accuracy-interpretability challenge in machine learning by equipping the TM clauses with integer weights. The resulting Integer Weighted TM (IWTM) deals with the problem of learning which clauses are inaccurate and thus must team up to obtain high accuracy as a team (low weight clauses), and which clauses are sufficiently accurate to operate more independently (high weight clauses). Since each TM clause is formed adaptively by a team of Tsetlin Automata, identifying effective weights becomes a challenging online learning problem. We address this problem by extending each team of Tsetlin Automata with a stochastic searching on the line (SSL) automaton. In our novel scheme, the SSL automaton learns the weight of its clause in interaction with the corresponding Tsetlin Automata team, which, in turn, adapts the composition of the clause by the adjusting weight. We evaluate IWTM empirically using five datasets, including a study of interpetability. On average, IWTM uses 6.5 times fewer literals than the vanilla TM and 120 times fewer literals than a TM with real-valued weights. Furthermore, in terms of average F1-Score, IWTM outperforms simple Multi-Layered Artificial Neural Networks, Decision Trees, Support Vector Machines, K-Nearest Neighbor, Random Forest, XGBoost, Explainable Boosting Machines, and standard and real-value weighted TMs.

Related papers

• ETHEREAL: Energy-efficient and High-throughput Inference using Compressed Tsetlin Machine [0.3121107735397556]
  The Tsetlin Machine (TM) is a novel alternative to deep neural networks (DNNs) We introduce a training approach that incorporates excluded automata states to sparsify TM logic patterns in both positive and negative clauses. Compared to standard TMs, ETHEREAL TM models can reduce model size by up to 87.54%, with only a minor accuracy compromise.
  arXiv  Detail & Related papers  (2025-02-08T16:58:43Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs) We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length. PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning [63.80739044622555]
  We introduce MuSR, a dataset for evaluating language models on soft reasoning tasks specified in a natural language narrative. This dataset has two crucial features. First, it is created through a novel neurosymbolic synthetic-to-natural generation algorithm. Second, our dataset instances are free text narratives corresponding to real-world domains of reasoning.
  arXiv  Detail & Related papers  (2023-10-24T17:59:20Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)
• SatLM: Satisfiability-Aided Language Models Using Declarative Prompting [68.40726892904286]
  We propose a new satisfiability-aided language modeling (SatLM) approach for improving the reasoning capabilities of large language models (LLMs) We use an LLM to generate a declarative task specification rather than an imperative program and leverage an off-the-shelf automated theorem prover to derive the final answer. We evaluate SATLM on 8 different datasets and show that it consistently outperforms program-aided LMs in the imperative paradigm.
  arXiv  Detail & Related papers  (2023-05-16T17:55:51Z)
• Building Concise Logical Patterns by Constraining Tsetlin Machine Clause Size [11.43224924974832]
  This paper introduces a novel variant of TM learning - Clause Size Constrained TMs (CSC-TMs) As soon as a clause includes more literals than the constraint allows, it starts expelling literals. Our results show that CSC-TM maintains accuracy with up to 80 times fewer literals.
  arXiv  Detail & Related papers  (2023-01-19T17:37:48Z)
• Improving Rare Word Recognition with LM-aware MWER Training [50.241159623691885]
  We introduce LMs in the learning of hybrid autoregressive transducer (HAT) models in the discriminative training framework. For the shallow fusion setup, we use LMs during both hypotheses generation and loss computation, and the LM-aware MWER-trained model achieves 10% relative improvement. For the rescoring setup, we learn a small neural module to generate per-token fusion weights in a data-dependent manner.
  arXiv  Detail & Related papers  (2022-04-15T17:19:41Z)
• Coalesced Multi-Output Tsetlin Machines with Clause Sharing [7.754230120409288]
  Using finite-state machines to learn patterns, Tsetlin machines (TMs) have obtained competitive accuracy and learning speed across several benchmarks. We introduce clause sharing, merging multiple TMs into a single one. Our empirical results on MNIST, Fashion-MNIST, and Kuzushiji-MNIST show that CoTM obtains significantly higher accuracy than TM on $50$- to $1$K-clause configurations.
  arXiv  Detail & Related papers  (2021-08-17T12:52:01Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• A Regression Tsetlin Machine with Integer Weighted Clauses for Compact Pattern Representation [9.432068833600884]
  The Regression Tsetlin Machine (RTM) addresses the lack of interpretability impeding state-of-the-art nonlinear regression models. We introduce integer weighted clauses to reduce computation cost N times and increase interpretability. We evaluate the potential of the integer weighted RTM using six artificial datasets.
  arXiv  Detail & Related papers  (2020-02-04T12:06:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.