Modeling Next-Token Prediction as Left-Nested Intuitionistic Implication

• URL: http://arxiv.org/abs/2601.19915v1
• Date: Wed, 07 Jan 2026 04:10:26 GMT
• Title: Modeling Next-Token Prediction as Left-Nested Intuitionistic Implication
• Abstract summary: We introduce the emphArrow Language Model, a neural architecture derived from an intuitionistic-logic interpretation of next-token prediction.<n>We show that a neural architecture equivalent to multiplicative RNNs arises naturally from a proof-theoretic interpretation of next-token prediction as nested intuitionistic implication.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the \emph{Arrow Language Model}, a neural architecture derived from an intuitionistic-logic interpretation of next-token prediction. Instead of representing tokens as additive embeddings mixed by attention, we encode a prefix as a \emph{left-nested implication chain} whose structure preserves order through non-commutative composition. Next-token prediction corresponds to \emph{modus ponens}, and sequence processing becomes constructive proof extension under the Curry--Howard correspondence. Our Prolog-based specialized theorem provers validate fundamental properties of the neural models, among which relations between commutative vs. non-commutative sequencing and single-token vs. multi-token prediction choices. We show that a neural architecture equivalent to multiplicative RNNs arises naturally from a proof-theoretic interpretation of next-token prediction as nested intuitionistic implication, we present a practical low-rank neural realization and position the model relative to Transformers and state-space models. Keywords: logic-based derivation of neural architectures, intuitionistic implicational logic, token-as-operator neural models, state-space models, alternatives to transformer-based foundational models.

Related papers

• Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees [6.750611816045116]
  Token prediction is a central pre-training objective for modern language models.<n>We develop a statistical framework connecting token prediction with representation geometry, encoder approximation, and downstream performance.
  arXiv  Detail & Related papers  (2026-08-30T22:33:15Z)
• Perturbation is All You Need for Extrapolating Language Models [10.671612115533597]
  We introduce a simple yet powerful framework for training large language models.<n>We develop a rigorous theory of extrapolability, namely, the capacity of a model class to make reliable predictions.<n>Results show that the proposed method consistently improves out-of-support prediction while maintaining competitive in-support performance.
  arXiv  Detail & Related papers  (2026-05-05T23:03:33Z)
• A Minimal Task Reveals Emergent Path Integration and Object-Location Binding in a Predictive Sequence Model [0.0]
  We show that action-conditioned sequential prediction suffices for learning "world models"<n>We train a recurrent neural network to predict the upcoming token from current input and a saccade-like displacement.<n>Decoding analyses reveal path integration and dynamic binding of token identity to position.
  arXiv  Detail & Related papers  (2026-02-03T13:08:27Z)
• Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations [34.88156871518115]
  Next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text.<n>We demonstrate that concepts corresponding to larger singular values are learned earlier during training, yielding a natural semantic hierarchy.<n>This insight motivates orthant-based clustering, a method that combines concept signs to identify interpretable semantic categories.
  arXiv  Detail & Related papers  (2025-05-13T08:46:04Z)
• Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures [49.19753720526998]
  We derive theoretical scaling laws for neural network performance on synthetic datasets.<n>We validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance.<n>This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.
  arXiv  Detail & Related papers  (2025-05-11T17:44:14Z)
• Semformer: Transformer Language Models with Semantic Planning [18.750863564495006]
  Next-token prediction serves as the dominant component in current neural language models. We introduce Semformer, a novel method of training a Transformer language model that explicitly models the semantic planning of response.
  arXiv  Detail & Related papers  (2024-09-17T12:54:34Z)
• The Foundations of Tokenization: Statistical and Computational Concerns [51.370165245628975]
  Tokenization is a critical step in the NLP pipeline.<n>Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood.<n>The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models.
  arXiv  Detail & Related papers  (2024-07-16T11:12:28Z)
• Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation [52.270712965271656]
  We propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. We find that the graph of our model resembles transformers, with correspondences between dependencies and self-attention. Experiments show that our model performs competitively to transformers on small to medium sized datasets.
  arXiv  Detail & Related papers  (2023-11-26T06:56:02Z)
• Causal Interpretation of Self-Attention in Pre-Trained Transformers [4.419843514606336]
  We propose a causal interpretation of self-attention in the Transformer neural network architecture. We use self-attention as a mechanism that estimates a structural equation model for a given input sequence of symbols. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.
  arXiv  Detail & Related papers  (2023-10-31T09:27:12Z)
• Set-based Neural Network Encoding Without Weight Tying [91.37161634310819]
  We propose a neural network weight encoding method for network property prediction.<n>Our approach is capable of encoding neural networks in a model zoo of mixed architecture.<n>We introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture.
  arXiv  Detail & Related papers  (2023-05-26T04:34:28Z)
• Unsupervised Learning of Equivariant Structure from Sequences [30.974508897223124]
  We present an unsupervised framework to learn the symmetry from the time sequence of length at least three. We will demonstrate that, with our framework, the hidden disentangled structure of the dataset naturally emerges as a by-product.
  arXiv  Detail & Related papers  (2022-10-12T07:29:18Z)
• Instance-Based Neural Dependency Parsing [56.63500180843504]
  We develop neural models that possess an interpretable inference process for dependency parsing. Our models adopt instance-based inference, where dependency edges are extracted and labeled by comparing them to edges in a training set.
  arXiv  Detail & Related papers  (2021-09-28T05:30:52Z)
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction [86.94309120789396]
  We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks.
  arXiv  Detail & Related papers  (2021-06-06T21:53:54Z)
• Explaining and Improving Model Behavior with k Nearest Neighbor Representations [107.24850861390196]
  We propose using k nearest neighbor representations to identify training examples responsible for a model's predictions. We show that kNN representations are effective at uncovering learned spurious associations. Our results indicate that the kNN approach makes the finetuned model more robust to adversarial inputs.
  arXiv  Detail & Related papers  (2020-10-18T16:55:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.