Why SWAVE May Not Be All You Need:A Concept-Evolution Retrospective on Complex-Valued Recurrent Language Models

• URL: http://arxiv.org/abs/2606.18324v1
• Date: Tue, 16 Jun 2026 16:05:28 GMT
• Title: Why SWAVE May Not Be All You Need:A Concept-Evolution Retrospective on Complex-Valued Recurrent Language Models
• Abstract summary: SWave is a complex-valued recurrent language model (169.26M parameters, D=384, L=16, T48) trained on FineWeb-Edu using 2xH100 NVL.<n>It was designed around three founding premises: that representing language as complex waves rather than real-valued numbers enables richer information encoding.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: SWave is a complex-valued recurrent language model (169.26M parameters, D=384, L=16, T=2048) trained on FineWeb-Edu using 2xH100 NVL. It was designed around three founding premises: that representing language as complex waves rather than real-valued numbers enables richer information encoding; that a Cayley-parameterised unitary transition provides a mathematical guarantee against state decay or explosion; and that a hidden state which rotates rather than shrinks preserves signal integrity over arbitrarily long contexts. The core of SWave evolved substantially across three development phases. The Resonance Head was found to structurally admit imaginary-channel collapse as a global loss minimum (a failure mode we term cos-domination collapse) and was superseded by an untied head with independent real and imaginary embedding tables from the Phase-Associative Memory (PAM) architecture. This resolved the degenerate minimum and enabled stable 200,000-step training (best-step PPL 22.0 at step 89,861). ComplexNorm and the Wave Propagation Scan proved load-bearing throughout all three phases and were retained to the final architecture. ProtectGatedScan was reframed as a structural prior rather than a learned behaviour. The four multi-scale retention concepts showed no measurable improvement under controlled evaluation and were found non-load-bearing. The ComplexGatedUnit was superseded by a real-valued squared-ReLU channel mixer with fewer parameters. The auxiliary training objectives showed no benefit once structural constraints were resolved. The investigation yields a formal characterisation of cos-domination collapse, a parallel scan with a log-space backward pass for numerical stability, six transferable engineering principles for complex-valued recurrent training, and a plan-to-code traceability methodology for catching structural divergences that conventional test suites miss.

Related papers

• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• ProgFormer: Hierarchical Voxel Diffusion Transformer for Longitudinal Brain MRI Prediction [81.07048249760953]
  longitudinal changes are often subtle and confined to specific anatomical regions.<n>Most subject-specific brain structure remains stable over time.<n>ProgFormer is a hierarchical voxel-space Diffusion Transformer for longitudinal brain MRI prediction.
  arXiv  Detail & Related papers  (2026-07-30T00:13:42Z)
• Operator-Informed Gaussian Processes for Complex Helmholtz Wavefields: From Synthetic Benchmarks to In Vivo Brain Elastography [0.02780623255172367]
  We extend operator-informed GP regression to complex-valued Helmholtz problems.<n>The construction admits a family of priors, from a proper diagonal prior to coregionalized and multiscale variants.<n>Calibrated uncertainty emerges as the central next step for probabilistic wavefield inference in dissipative media.
  arXiv  Detail & Related papers  (2026-07-15T16:21:50Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory [0.0]
  The effect replicates, but it is not a memory improvement.<n>Training on this task is a long chance plateau followed by a sharp escape.<n>Much of the published gain needs no architecture at all.
  arXiv  Detail & Related papers  (2026-07-02T18:54:23Z)
• MambaADv2: Evolving Duality-enhanced State Space Model for Unsupervised Anomaly Detection [108.36437360254605]
  This paper proposes MambaADv2, a framework tailored for multi-class unsupervised anomaly detection.<n>MambaADv2 comprises a pre-trained encoder and a Mamba-inspired decoder, equipped with Duality-enhanced State Space (DSS) modules across multiple scales.<n>The structure of the Hybrid State Space (HSS) block is tailored by following the SSD-based Mamba lineage and incorporating Mamba3-style position-aware state-space modeling.
  arXiv  Detail & Related papers  (2026-06-22T10:14:06Z)
• When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms [43.8784307709823]
  Long-horizon autoregressive forecasting of physical signals is limited by error accumulation.<n>We ask when such rollout stays stable using synthetic three-component seismograms as a physically structured testbed.
  arXiv  Detail & Related papers  (2026-06-09T13:46:14Z)
• Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability [48.83701310501069]
  Large language models (LLMs) are increasingly used as generators in iterative neural architecture search (NAS)<n>We model iterative LLM-NAS as a parametric Cross-Entropy (CE) method over executable programs.<n>We prove six results: (1) iterative LLM fine-tuning on elite architectures is equivalent to the CE update restricted to the LLM family; (2) expected architecture quality is monotonically non-decreasing across cycles; and (3) elite-set probability converges to a fixed point at a geometric rate.
  arXiv  Detail & Related papers  (2026-05-28T15:45:19Z)
• When do complex-valued neural networks help? A study of representation, geometry, and optimization [1.4935265696074094]
  We show that complex-valued inputs alone do not determine when complex arithmetic improves learning.<n>Across synthetic RF tasks, complex representations are useful but not universally superior.<n>CVNNs are best viewed as structured inductive biases whose gains depend on representation, symmetry, and optimization.
  arXiv  Detail & Related papers  (2026-05-26T20:49:23Z)
• The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems [0.0]
  This thesis turns impossibility results from curiosities into design rules.<n>Its flagship result proves an accuracy ceiling set by architecture alone.<n>The same argument recasts across subfields.
  arXiv  Detail & Related papers  (2026-05-21T20:48:35Z)
• Beyond LLMs, Sparse Distributed Memory, and Neuromorphics <A Hyper-Dimensional SRAM-CAM "VaCoAl" for Ultra-High Speed, Ultra-Low Power, and Low Cost> [18.56157183294801]
  We propose VaCoAl on ultra-high-dimensional binary spaces on ultra-high-dimensional RAM/DRAM-CAM.<n>It embeds a cognitive bound -- the Frontier Size -- into its architecture, ranking candidates by path-integral confidence.<n>We evaluate multi-hop reasoning on about 470k mentor-student relations from Wikidata, tracing up to 57 generations.
  arXiv  Detail & Related papers  (2026-04-13T16:13:17Z)
• CircuitSynth: Reliable Synthetic Data Generation [44.80087038178069]
  Large Language Models (LLMs) frequently suffer from hallucinations, logical inconsistencies, and mode collapse when tasked with structured generation.<n>Existing approaches, such as prompting or retrieval-augmented generation, lack the mechanisms to balance linguistic expressivity with formal guarantees regarding validity and coverage.<n>We propose Circuit Synth, a novel neuro-symbolic framework that decouples semantic reasoning from surface realization.
  arXiv  Detail & Related papers  (2026-04-11T09:18:52Z)
• CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling [46.16066322190728]
  We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture.<n>Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors.<n>To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism.
  arXiv  Detail & Related papers  (2026-04-05T20:13:22Z)
• Toward Complex-Valued Neural Networks for Waveform Generation [49.128847336227636]
  We present ComVo, a complex-valued neural vocoder whose generator and discriminator use native complex arithmetic.<n>Experiments demonstrate that ComVo achieves higher quality than comparable real-valued baselines, and that its block-matrix scheme reduces training time by 25%.
  arXiv  Detail & Related papers  (2026-03-12T06:24:11Z)
• Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Prompting [44.347846669388446]
  We propose EVOL-SAM3, a zero-shot framework that reformulates reasoning segmentation as an inference-time evolutionary search process.<n>EVOL-SAM3 not only substantially outperforms static baselines but also significantly surpasses fully supervised state-of-the-art methods on the challenging ReasonSeg benchmark in a zero-shot setting.
  arXiv  Detail & Related papers  (2025-12-31T08:10:03Z)
• The Mirror Loop: Recursive Non-Convergence in Generative Reasoning Systems [0.0]
  Recursive self-evaluation without external feedback often yields reformulation rather than progress.<n>We study 144 reasoning sequences across three models (OpenAI GPT-4o-mini, Anthropic Claude 3 Haiku, and Google Gemini 2.0 Flash) and four task families (arithmetic, code, explanation, reflection)<n>We interpret this as evidence for a structural limit on self-correction in generative reasoning.
  arXiv  Detail & Related papers  (2025-10-23T07:53:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.