CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling

• URL: http://arxiv.org/abs/2604.04250v1
• Date: Sun, 05 Apr 2026 20:13:22 GMT
• Title: CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling
• Abstract summary: We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture.<n>Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors.<n>To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism.
• Score: 46.16066322190728
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Modern Large Language Models (LLMs) rely on Transformer self-attention, which scales quadratically with sequence length. Recent linear-time alternatives, like State Space Models (SSMs), often suffer from signal degradation over extended contexts. We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture. Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors, achieving sequence mixing through a causal, $O(L)$ Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism incorporating Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache to capture short-term local dependencies. We also replace standard dense linear projections with Depth-wise Harmonic Convolutions for optimal spatial frequency mixing, augmented by Block Attention Residuals for depth-wise state routing. Scaled to a 150M-parameter model, CAWN utilizes custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token corpus, the prototype is evaluated at a 5-Billion-token milestone. Empirical evaluations via a Targeted Semantic Retrieval protocol demonstrate robust vocabulary acquisition and extended explicitly learned contextual denoising. By leveraging $O(1)$ state-passing via chunked prefill, the model retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the $O(L^2)$ context memory wall.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation [63.50393485685279]
  We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
  arXiv  Detail & Related papers  (2026-07-02T12:55:11Z)
• STAMBRIDGE: Spectral-Temporal Amplitude-aware Mid-Feature Bridge for EEG Visual Decoding [5.653339889503698]
  STAMBRIDGE is a versatile two-stage framework that tackles feature conditioning and cross-modal alignment.<n>Building upon these robust neural features, we introduce a model-agnostic Mid-Feature Semantic Bridge (MFSB)<n>Experiments on the THINGS-EEG benchmark show competitive 200-way zero-shot retrieval performance, with 34.50% Top-1 and 65.95% Top-5 accuracy.
  arXiv  Detail & Related papers  (2026-05-22T01:21:45Z)
• Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion [14.692250535148196]
  Diffusion language models (DLMs) promise parallel, order-agnostic generation but have historically lagged behind autoregressive models in sample quality and diversity.<n>Recent continuous flow and diffusion approaches over token embeddings have narrowed this gap, suggesting continuous state spaces are highly effective for language.<n>In this work, we further close the autoregressive gap by modeling text as a continuous diffusion process over fixed-width binary bitstreams.
  arXiv  Detail & Related papers  (2026-05-07T22:50:51Z)
• The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle [0.09229852843814058]
  We introduce the textbfPhasor Transformer block, a phase-native alternative representing sequence states on the unit-circle manifold $S1$.<n> Stacking these blocks defines the textbfLarge Phasor Model (LPM), which learns stable global dynamics and achieves competitive forecasting behavior.<n>Our results establish an explicit efficiency-performance frontier, demonstrating that large-model scaling for time-series can emerge from geometry-constrained phase with deterministic global coupling.
  arXiv  Detail & Related papers  (2026-03-18T07:18:41Z)
• KoopmanFlow: Spectrally Decoupled Generative Control Policy via Koopman Structural Bias [31.486395926173838]
  Generative Control Policies (GCPs) show immense promise in robotic manipulation but struggle to simultaneously model stable global motions and high-frequency local corrections.<n>We introduce KoopmanFlow, a parameter-efficient generative policy guided by a Koopman-inspired structural inductive bias.
  arXiv  Detail & Related papers  (2026-03-14T06:23:07Z)
• Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixes [10.877713536966601]
  Longestahead Prefix (LSP) scheduler is a training-free and model-agnostic inference paradigm based on monolithic prefix absorption.<n>LSP evaluates token stability via a single forward pass, dynamically identifies a contiguous left-aligned block of stable predictions.<n>It snaps its boundary to natural linguistic or structural acceptances before an atomic commitment.
  arXiv  Detail & Related papers  (2026-03-05T18:25:26Z)
• Continuous Autoregressive Language Models [56.49239051750678]
  We introduce Continuous Autoregressive Language Models (CALM)<n>CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector.<n>We develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling.
  arXiv  Detail & Related papers  (2025-10-31T17:58:11Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• ECHO: Frequency-aware Hierarchical Encoding for Variable-length Signals [8.411477071838592]
  We propose a novel foundation model ECHO that integrates an advanced band-split architecture with frequency positional embeddings.<n>We evaluate our method on various kinds of machine signal datasets.
  arXiv  Detail & Related papers  (2025-08-20T13:10:44Z)
• Next Tokens Denoising for Speech Synthesis [51.320443764269726]
  Dragon-FM is a novel text-to-speech (TTS) design that unifies AR and flow-matching.<n>It processes 48 kHz audio tokens in chunks at a compact rate of 12.5 tokens per second.<n>Experiments on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts.
  arXiv  Detail & Related papers  (2025-07-30T15:03:36Z)
• CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement [5.766499647507758]
  We further develop the conformer-based metric generative adversarial network (CMGAN) model for speech enhancement (SE) in the time-frequency (TF) domain. Our findings show that CMGAN outperforms existing state-of-the-art methods in the three major speech enhancement tasks: denoising, dereverberation, and super-resolution.
  arXiv  Detail & Related papers  (2022-09-22T15:50:21Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.