Spiking Sequence Machines and Transformers

• URL: http://arxiv.org/abs/2605.00662v1
• Date: Fri, 01 May 2026 13:45:15 GMT
• Title: Spiking Sequence Machines and Transformers
• Abstract summary: Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space.<n>We formalise a Phase-Late Isomorphism showing that sinusoidal positional phase and spike timing are linearly related.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.

Related papers

• Real-Valued Hyperdimensional Sequence Representations with Hadamard Product Binding and Shift Equivariance [6.771283097297353]
  This paper develops real-valued position encoding motivated by Random Features.<n>It aims to retain the desirable properties of Fractional Power representations while supporting Hadamard-based operations.<n>Experiments on time-series classification datasets show that the proposed real-valued representations achieve performance comparable to standard Fractional Power kernels.
  arXiv  Detail & Related papers  (2026-08-28T13:45:37Z)
• Dynamical phase selection controls compute scaling in looped transformers [0.0]
  We show that networks with identical architecture and objective, trained to identical accuracy, nevertheless realize distinct dynamical phases.<n>The phases are distinguished by their bifurcation mechanisms, including a saddle-node fold and a Neimark-Sacker-type transition to bounded nonstationary motion.
  arXiv  Detail & Related papers  (2026-08-27T02:54:02Z)
• Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction [44.83333974000826]
  We introduce a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals.<n> Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits.<n>We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
  arXiv  Detail & Related papers  (2026-02-07T14:18:11Z)
• Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features [2.945172427769856]
  CAPE is a novel method that identifies underlying causal structure over non-sequential features as a weighted directed acyclic graph (DAG)<n>The DAG is embedded in hyperbolic space where its geometric structure is well-preserved using a hyperboloid model-based approach.<n>This step yields causality-aware positional encodings for the features, which are converted into their rotary form for integrating with transformer's self-attention mechanism.
  arXiv  Detail & Related papers  (2025-09-20T11:08:02Z)
• Wave-Based Semantic Memory with Resonance-Based Retrieval: A Phase-Aware Alternative to Vector Embedding Stores [51.56484100374058]
  We propose a novel framework that models knowledge as wave patterns $psi(x) = A(x) eiphi(x)$ and retrieves it through resonance-based interference.<n>This approach preserves both amplitude and phase information, enabling more expressive and robust semantic similarity.
  arXiv  Detail & Related papers  (2025-08-21T10:13:24Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• 'Memory States' from Almost Nothing: Representing and Computing in a Non-associative Algebra [0.0]
  This note presents a non-associative framework for the representation and computation of information items in high-dimensional space.<n>It is consistent with the principles of spatial computing and with the empirical findings in cognitive science about memory.
  arXiv  Detail & Related papers  (2025-05-13T08:43:02Z)
• EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention [88.45459681677369]
  We propose a novel transformer variant with complex vector attention, named EulerFormer. It provides a unified theoretical framework to formulate both semantic difference and positional difference. It is more robust to semantic variations and possesses moresuperior theoretical properties in principle.
  arXiv  Detail & Related papers  (2024-03-26T14:18:43Z)
• Entangled Residual Mappings [59.02488598557491]
  We introduce entangled residual mappings to generalize the structure of the residual connections. An entangled residual mapping replaces the identity skip connections with specialized entangled mappings. We show that while entangled mappings can preserve the iterative refinement of features across various deep models, they influence the representation learning process in convolutional networks.
  arXiv  Detail & Related papers  (2022-06-02T19:36:03Z)
• Combiner: Full Attention Transformer with Sparse Computation Cost [142.10203598824964]
  We propose Combiner, which provides full attention capability in each attention head while maintaining low computation complexity. We show that most sparse attention patterns used in existing sparse transformers are able to inspire the design of such factorization for full attention. An experimental evaluation on both autoregressive and bidirectional sequence tasks demonstrates the effectiveness of this approach.
  arXiv  Detail & Related papers  (2021-07-12T22:43:11Z)
• Learning 3D Dense Correspondence via Canonical Point Autoencoder [108.20735652143787]
  We propose a canonical point autoencoder (CPAE) that predicts dense correspondences between 3D shapes of the same category. The autoencoder performs two key functions: (a) encoding an arbitrarily ordered point cloud to a canonical primitive, and (b) decoding the primitive back to the original input instance shape.
  arXiv  Detail & Related papers  (2021-07-10T15:54:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.