BriLLM: Brain-inspired Large Language Model

• URL: http://arxiv.org/abs/2503.11299v2
• Date: Mon, 07 Apr 2025 11:09:39 GMT
• Title: BriLLM: Brain-inspired Large Language Model
• Authors: Hai Zhao, Hongqiu Wu, Dongjie Yang, Anni Zou, Jiale Hong,
• Abstract summary: BriLLM is a non-Transformer, non-GPT, non-traditional machine learning input-output controlled generative language model.<n>We release the first BriLLM version in Chinese, with 4000 tokens, 32-dimensional node width, 16-token long sequence prediction ability, and language model prediction performance comparable to GPT-1.
• Score: 51.849486186292914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper reports the first brain-inspired large language model (BriLLM). This is a non-Transformer, non-GPT, non-traditional machine learning input-output controlled generative language model. The model is based on the Signal Fully-connected flowing (SiFu) definition on the directed graph in terms of the neural network, and has the interpretability of all nodes on the graph of the whole model, instead of the traditional machine learning model that only has limited interpretability at the input and output ends. In the language model scenario, the token is defined as a node in the graph. A randomly shaped or user-defined signal flow flows between nodes on the principle of "least resistance" along paths. The next token or node to be predicted or generated is the target of the signal flow. As a language model, BriLLM theoretically supports infinitely long $n$-gram models when the model size is independent of the input and predicted length of the model. The model's working signal flow provides the possibility of recall activation and innate multi-modal support similar to the cognitive patterns of the human brain. At present, we released the first BriLLM version in Chinese, with 4000 tokens, 32-dimensional node width, 16-token long sequence prediction ability, and language model prediction performance comparable to GPT-1. More computing power will help us explore the infinite possibilities depicted above.

Related papers

• Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models [4.7936447642295406]
  In Transformer language models, activation vectors transform from current token embeddings to next token predictions as they pass through the model. To isolate a minimal form of this transformation, we identify language modelworks that make bigram predictions, naive next token predictions based only on the current token. We find that bigramworks can be found in fully trained language models up to 1B parameters, and theseworks are critical for model performance even when they consist of less than 0.2% of model parameters.
  arXiv  Detail & Related papers  (2025-04-21T22:41:00Z)
• Interpretable Language Modeling via Induction-head Ngram Models [74.26720927767398]
  We propose Induction-head ngram models (Induction-Gram) to bolster modern ngram models with a hand-engineered "induction head" This induction head uses a custom neural similarity metric to efficiently search the model's input context for potential next-word completions. Experiments show that this simple method significantly improves next-word prediction over baseline interpretable models.
  arXiv  Detail & Related papers  (2024-10-31T12:33:26Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• This Probably Looks Exactly Like That: An Invertible Prototypical Network [8.957872207471311]
  Prototypical neural networks represent an exciting way forward in realizing human-comprehensible machine learning without concept annotations. We find that reliance on indirect interpretation functions for prototypical explanations imposes a severe limit on prototypes' informative power. We propose one such model, called ProtoFlow, by composing a normalizing flow with Gaussian mixture models.
  arXiv  Detail & Related papers  (2024-07-16T21:51:02Z)
• Power Failure Cascade Prediction using Graph Neural Networks [4.667031410586657]
  We propose a flow-free model that predicts grid states at every generation of a cascade process given an initial contingency and power injection values. We show that the proposed model reduces the computational time by almost two orders of magnitude.
  arXiv  Detail & Related papers  (2024-04-24T18:45:50Z)
• Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks [12.7259425362286]
  We investigate how multilingual models might leverage key-value memories. For autoregressive models trained on two or more languages, do all neurons (across layers) respond equally to all languages? Our findings reveal that the layers closest to the network's input or output tend to exhibit more language-specific behaviour compared to the layers in the middle.
  arXiv  Detail & Related papers  (2023-10-24T06:45:00Z)
• Residual Learning of Neural Text Generation with $n$-gram Language Model [41.26228768053928]
  We learn a neural LM that fits the residual between an $n$-gram LM and the real-data distribution. Our approach attains additional performance gains over popular standalone neural models consistently.
  arXiv  Detail & Related papers  (2022-10-26T02:42:53Z)
• Augmenting Interpretable Models with LLMs during Training [73.40079895413861]
  We propose Augmented Interpretable Models (Aug-imodels) to build efficient and interpretable models. Aug-imodels use LLMs during fitting but not during inference, allowing complete transparency. We explore two instantiations of Aug-imodels in natural-language processing: (i) Aug-GAM, which augments a generalized additive model with decoupled embeddings from an LLM and (ii) Aug-Tree, which augments a decision tree with LLM feature expansions.
  arXiv  Detail & Related papers  (2022-09-23T18:36:01Z)
• Hidden Schema Networks [3.4123736336071864]
  We introduce a novel neural language model that enforces, via inductive biases, explicit relational structures. The model encodes sentences into sequences of symbols, which correspond to nodes visited by biased random walkers. We show that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences.
  arXiv  Detail & Related papers  (2022-07-08T09:26:19Z)
• Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models [62.41139712595334]
  We propose a novel pre-training paradigm for Chinese -- Lattice-BERT. We construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We show that our model can bring an average increase of 1.5% under the 12-layer setting.
  arXiv  Detail & Related papers  (2021-04-15T02:36:49Z)
• Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition [80.446770909975]
  Linguistic knowledge is of great benefit to scene text recognition. How to effectively model linguistic rules in end-to-end deep networks remains a research challenge. We propose an autonomous, bidirectional and iterative ABINet for scene text recognition.
  arXiv  Detail & Related papers  (2021-03-11T06:47:45Z)
• Multi-timescale Representation Learning in LSTM Language Models [69.98840820213937]
  Language models must capture statistical dependencies between words at timescales ranging from very short to very long. We derived a theory for how the memory gating mechanism in long short-term memory language models can capture power law decay. Experiments showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution.
  arXiv  Detail & Related papers  (2020-09-27T02:13:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.