Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall

• URL: http://arxiv.org/abs/2604.07716v2
• Date: Sat, 11 Apr 2026 09:51:58 GMT
• Title: Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall
• Abstract summary: Fan Duality Model resolves tension between memory efficiency and associative recall in sequence modeling.<n>FDM separates sequence processing into two components: a wave component that compresses long-range patterns into a hidden state, and a particle component that retrieves specific tokens.<n>We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimize the cache jointly with embeddings.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present FDM (Fan Duality Model), a linear sequence architecture that resolves the fundamental tension between memory efficiency and associative recall in sequence modeling. FDM separates sequence processing into two components: a wave component (recurrent scan via phase-preserving Givens rotations) that compresses long-range patterns into a fixed-size complex hidden state, and a particle component (local-global cache) that retrieves specific tokens via learned associative addressing with W+K=272 slots independent of sequence length N. This yields strictly O(1) decode memory: 867 MB fixed across all prompt lengths 128-8,192 tokens, versus Transformer's 853-4,247 MB (4.9x reduction at N=8,192). Beyond the architecture, we discover that jointly training the wave and particle components leads to suboptimal convergence. We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimizes the cache jointly with embeddings, achieving PPL=64.9 on WikiText-103 in 44K steps -- a 7.5x improvement over full fine-tuning (PPL=487). On Multi-Query Associative Recall (MQAR), FDM achieves 0.966 accuracy, surpassing Transformer (0.606) by 59.5%, while pure scan without cache scores only 0.011, confirming the necessity of the particle component. Finally, we introduce Holographic Reference Beam Decoding, interpreting the complex hidden state h_t as a holographic plate encoding the entire temporal history. Using the current input x_t as a reference beam to modulate h_t reduces PPL by up to 2.13 points (PPL=62.79) with a 4-head orthogonal reference beam using only 1.3M additional parameters, providing empirical support for the holographic interpretation. Code and pretrained weights: https://github.com/YasongFan/FDM

Related papers

• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• Hierarchical Transformer Preconditioning for Interactive Physics Simulation [28.137076331332413]
  Hierarchical Transformer Preconditioner, a neural preconditioner anchored to a weak-admissibility H-matrix partition.<n>Network models the inverse through low-rank far-field factors.<n>Uses highway connections to propagate context across depth sparse.
  arXiv  Detail & Related papers  (2026-05-13T11:02:27Z)
• DMax: Aggressive Parallel Decoding for dLLMs [77.24184219948337]
  We present DMax, a new paradigm for efficient diffusion language models (dLLMs)<n>It mitigates error accumulation in parallel decoding, enabling aggressive decoding while preserving generation quality.<n>At the core of our approach is On-Policy Uniform Training, a novel training strategy that efficiently unifies masked and uniform dLLMs.
  arXiv  Detail & Related papers  (2026-04-09T14:35:42Z)
• Decoder Dependence in Surface-Code Threshold Estimation with Native Gottesman-Kitaev-Preskill Digitization and Parallelized Sampling [4.11280059407553]
  We benchmark decoder dependence in surface-code threshold studies under two matched regimes: Pauli noise and native GKP-style Gaussian displacement digitization.<n>Using LiDMaS+ v1.1.0, we benchmark MWPM, Union-Find (UF), Belief Propagation (BP), and neural-guided MWPM with fixed seeds, identical sweep grids, and unified reporting across runs 06--14.
  arXiv  Detail & Related papers  (2026-03-25T18:07:04Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA [10.452946241750562]
  Gated DeltaNet (GDN) is a linear attention mechanism that replaces the growing KV cache with a fixed-size recurrent state.<n>We show that this bottleneck is architectural, not algorithmic, as all subquadratic sequence models exhibit arithmetic intensities below 1 FLOP/B at decode time.<n>We present an FPGA accelerator that eliminates this bottleneck by holding the full 2 MB recurrent state persistently in on-chip BRAM.
  arXiv  Detail & Related papers  (2026-03-06T06:03:38Z)
• Unsupervised Discovery of Intermediate Phase Order in the Frustrated $J_1$-$J_2$ Heisenberg Model via Prometheus Framework [0.0]
  We apply the Prometheus variational autoencoder framework to explore the $J_1$-$J$ phase diagram.<n>We identify the structure factor $S(,)$ and $S(,)$ as the dominant order parameters.<n>This work establishes a scalable pathway for applying machine learning to frustrated quantum systems.
  arXiv  Detail & Related papers  (2026-02-25T00:44:51Z)
• FlashSchNet: Fast and Accurate Coarse-Grained Neural Network Molecular Dynamics [36.3434800328807]
  Graph neural network (GNN) potentials such as SchNet improve the accuracy and transferability of molecular dynamics (MD) simulation.<n>We show that a missing principle is making GNN-MD IO-aware, carefully accounting for reads and writes between high-bandwidth memory (HBM) and on-chip.<n>We present FlashSchNet, an efficient and accurate IO-aware GNN-MD framework built on four techniques.
  arXiv  Detail & Related papers  (2026-02-13T17:49:12Z)
• CommVQ: Commutative Vector Quantization for KV Cache Compression [50.37946553931796]
  We propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference.<n>We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache.<n>Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook.
  arXiv  Detail & Related papers  (2025-06-23T17:50:11Z)
• Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling [50.994194925685434]
  LrcSSM is a $textitnon-linear$ recurrent model that processes long sequences as fast as today's linear state-space layers.<n>By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel.<n>LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 do not provide.
  arXiv  Detail & Related papers  (2025-05-27T20:02:59Z)
• Speedy MASt3R [68.47052557089631]
  MASt3R redefines image matching as a 3D task by leveraging DUSt3R and introducing a fast reciprocal matching scheme.<n>Fast MASt3R achieves a 54% reduction in inference time (198 ms to 91 ms per image pair) without sacrificing accuracy.<n>This advancement enables real-time 3D understanding, benefiting applications like mixed reality navigation and large-scale 3D scene reconstruction.
  arXiv  Detail & Related papers  (2025-03-13T03:56:22Z)
• IterL2Norm: Fast Iterative L2-Normalization [1.4602363426887837]
  Transformer-based large language models are a memory-bound model whose operation is based on a large amount of data that are marginally reused.<n> Layer normalization is one of the key workloads in the transformer model, following each of multi-head attention and feed-forward network blocks.<n>We introduce an iterative L2-normalization method for 1D input (IterL2Norm), ensuring fast convergence to the steady-state solution within five iteration steps.
  arXiv  Detail & Related papers  (2024-12-06T05:00:01Z)
• Beyond likelihood ratio bias: Nested multi-time-scale stochastic approximation for likelihood-free parameter estimation [49.78792404811239]
  We study inference in simulation-based models where the analytical form of the likelihood is unknown.<n>We use a ratio-free nested multi-time-scale approximation (SA) method that simultaneously tracks the score and drives the parameter update.<n>We show that our algorithm can eliminate the original bias $Obig(sqrtfrac1Nbig)$ and accelerate the convergence rate from $Obig(beta_k+sqrtfracalpha_kNbig)$.
  arXiv  Detail & Related papers  (2024-11-20T02:46:15Z)
• Sequence modeling of higher-order wave modes of binary black hole mergers [0.0]
  Higher-order gravitational wave modes from quasi-circular, spinning, non-precessing binary black hole mergers encode key information about these systems' nonlinear dynamics.<n>We model these waveforms using transformer architectures, targeting the evolution from late inspiral through ringdown.<n>Our results demonstrate that transformer-based models can capture the nonlinear dynamics of binary black hole mergers with high accuracy, even outside the surrogate training domain.
  arXiv  Detail & Related papers  (2024-09-05T18:00:11Z)
• Randomized Exploration for Reinforcement Learning with Multinomial Logistic Function Approximation [8.274693573069442]
  We study reinforcement learning with multinomial logistic (MNL) function approximation. We propose provably efficient algorithms with randomized exploration having frequentist regret guarantees. Numerical experiments demonstrate the superior performance of the proposed algorithms.
  arXiv  Detail & Related papers  (2024-05-30T15:39:19Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training [82.06732962485754]
  FlashAttention effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. We introduce DISTFLASHATTN, a memory-efficient attention mechanism optimized for long-context LLMs training. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses.
  arXiv  Detail & Related papers  (2023-10-05T03:47:57Z)
• Measurement-induced phase transition for free fermions above one dimension [46.176861415532095]
  Theory of the measurement-induced entanglement phase transition for free-fermion models in $d>1$ dimensions is developed. Critical point separates a gapless phase with $elld-1 ln ell$ scaling of the second cumulant of the particle number and of the entanglement entropy.
  arXiv  Detail & Related papers  (2023-09-21T18:11:04Z)
• Efficient Context-Aware Network for Abdominal Multi-organ Segmentation [8.92337236455273]
  We develop a whole-based coarse-to-fine framework for efficient and effective abdominal multi-organ segmentation. For the decoder module, anisotropic convolution with a k*k*1 intra-slice convolution and a 1*1*k inter-slice convolution is designed to reduce the burden. For the context block, we propose strip pooling module to capture anisotropic and long-range contextual information.
  arXiv  Detail & Related papers  (2021-09-22T09:05:59Z)
• Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices [65.4388266814055]
  We replace 1x1-convolutions in 1D time-channel separable convolutions with constant, sparse random ternary matrices with weights in $-1,0,+1$. For command recognition on Google Speech Commands v1, we improve the state-of-the-art accuracy from $97.21%$ to $97.41%$ at the same network size. For speech recognition on Librispeech, we half the number of weights to be trained while only sacrificing about $1%$ of the floating-point baseline's word error rate.
  arXiv  Detail & Related papers  (2021-03-31T15:09:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.