Scalable Machines with Intrinsic Higher Mental-State Dynamics

• URL: http://arxiv.org/abs/2603.13453v1
• Date: Fri, 13 Mar 2026 16:16:50 GMT
• Title: Scalable Machines with Intrinsic Higher Mental-State Dynamics
• Abstract summary: This work introduces a mathematically grounded formulation showing how models (e.g., Transformers) can implement computational principles underlying awake imaginative thought.<n>Experiments on ImageNet-1K, benchmarked against a standard Vision Transformer (ViT), demonstrate significantly faster learning with reduced computational demand.
• Score: 1.5584040836686046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Drawing on recent breakthroughs in cellular neurobiology and detailed biophysical modeling linking neocortical pyramidal neurons to distinct mental-state regimes, this work introduces a mathematically grounded formulation showing how models (e.g., Transformers) can implement computational principles underlying awake imaginative thought to pre-select relevant information before attention is applied via triadic modulation loops among queries ($Q$), keys ($K$), and values ($V$).~Scalability experiments on ImageNet-1K, benchmarked against a standard Vision Transformer (ViT), demonstrate significantly faster learning with reduced computational demand (fewer heads, layers, and tokens), consistent with our prior findings in reinforcement learning and language modeling. The approach operates at approximately $\mathcal{O}(N)$ complexity with respect to the number of input tokens $N$.

Related papers

• Learning High-Dimensional Parity Functions with Product Networks using Gradient Descent [1.6802038598427533]
  High-dimensional parity functions are critical for machine learning, cryptography, and error correction.<n>Standard neural network architectures typically require exponential sample complexity.<n>We show that compact product-based neural architectures combined with data sparsity enable efficient parity learning.<n>This work opens new possibilities for neural arithmetic, structured reasoning, binary neural networks, and machine learning applied to automated protocol discovery.
  arXiv  Detail & Related papers  (2026-05-27T15:26:16Z)
• Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons [27.855778669417592]
  Cortical neurons are complex, multi-timescale processors wired into recurrent circuits.<n>Machine learning predominantly builds models extremely simple units, a default inherited from early neural-network theory.<n>We introduce the ELM Network, whose recurrent layer is built from Expressive Memory (ELM) neurons.
  arXiv  Detail & Related papers  (2026-05-12T12:29:33Z)
• Language Model Circuits Are Sparse in the Neuron Basis [50.460651620833055]
  We show that textbfMLP neurons are as sparse a feature basis as SAEs.<n>This work advances automated interpretability of language models without additional training costs.
  arXiv  Detail & Related papers  (2026-01-30T05:41:19Z)
• Beyond Attention: Toward Machines with Intrinsic Higher Mental States [1.2147145617662436]
  This work shows how models (e.g., Transformers) can emulate high-level perceptual processing and awake thought (imagination) states.<n>Results span reinforcement learning (e.g., CarRacing in a high-dimensional visual setup), computer vision, and natural language question answering.
  arXiv  Detail & Related papers  (2025-05-02T14:31:10Z)
• A Hybrid Transformer Architecture with a Quantized Self-Attention Mechanism Applied to Molecular Generation [0.0]
  We propose a hybrid quantum-classical self-attention mechanism as part of a transformer decoder.<n>We show that the time complexity of the query-key dot product is reduced from $mathcalO(n2 d)$ in a classical model to $mathcalO(n2 d)$ in our quantum model.<n>This work provides a promising avenue for quantum-enhanced natural language processing (NLP)
  arXiv  Detail & Related papers  (2025-02-26T15:15:01Z)
• Modeling Autonomous Shifts Between Focus State and Mind-Wandering Using a Predictive-Coding-Inspired Variational RNN Model [2.163881720692685]
  The study investigates possible neural mechanisms underling autonomous shifts between focus state and mind-wandering.<n>We modeled perception processes of continuous sensory sequences using our previous proposed variational RNN model.<n>Our simulation experiments demonstrated that autonomous shifts between focused perception and mind-wandering take place when $mathbfw$ switches between low and high values.
  arXiv  Detail & Related papers  (2024-12-20T07:25:20Z)
• Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs [35.212818841550835]
  One-hidden layer neural networks and one-layer Transformers are studied.<n>One-hidden layer neural networks attain a maximum $ L_2,k+1 $-margin on a dataset.<n>We observe similar computational mechanisms in attention of one-layer Transformers.
  arXiv  Detail & Related papers  (2024-02-12T05:52:06Z)
• In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
  We study ICL through the lens of a new family of model problems we term in context language learning (ICLL) We evaluate a diverse set of neural sequence models on regular ICLL tasks.
  arXiv  Detail & Related papers  (2024-01-23T18:59:21Z)
• WaLiN-GUI: a graphical and auditory tool for neuron-based encoding [73.88751967207419]
  Neuromorphic computing relies on spike-based, energy-efficient communication. We develop a tool to identify suitable configurations for neuron-based encoding of sample-based data into spike trains. The WaLiN-GUI is provided open source and with documentation.
  arXiv  Detail & Related papers  (2023-10-25T20:34:08Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)
• Transformers Learn Shortcuts to Automata [52.015990420075944]
  We find that a low-depth Transformer can represent the computations of any finite-state automaton. We show that a Transformer with $O(log T)$ layers can exactly replicate the computation of an automaton on an input sequence of length $T$. We further investigate the brittleness of these solutions and propose potential mitigations.
  arXiv  Detail & Related papers  (2022-10-19T17:45:48Z)
• Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics [85.31710759801705]
  Current practice requires expensive computational costs in model training for performance prediction. We propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections.
  arXiv  Detail & Related papers  (2022-01-11T20:53:15Z)
• Mapping and Validating a Point Neuron Model on Intel's Neuromorphic Hardware Loihi [77.34726150561087]
  We investigate the potential of Intel's fifth generation neuromorphic chip - Loihi' Loihi is based on the novel idea of Spiking Neural Networks (SNNs) emulating the neurons in the brain. We find that Loihi replicates classical simulations very efficiently and scales notably well in terms of both time and energy performance as the networks get larger.
  arXiv  Detail & Related papers  (2021-09-22T16:52:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.