When to Think Fast and Slow? AMOR: Entropy-Based Metacognitive Gate for Dynamic SSM-Attention Switching

• URL: http://arxiv.org/abs/2602.13215v1
• Date: Thu, 22 Jan 2026 17:19:58 GMT
• Title: When to Think Fast and Slow? AMOR: Entropy-Based Metacognitive Gate for Dynamic SSM-Attention Switching
• Abstract summary: State Space Models (SSMs) offer efficient alternatives but struggle with precise information retrieval over a long horizon.<n>Inspired by dual-process theories of cognition, we propose AMOR, a hybrid architecture that dynamically engages sparse attention only when an SSM backbone is "uncertain"<n>On small-scale synthetic retrieval tasks, AMOR outperforms both SSM-only and transformer-only baselines.
• Score: 4.238040764117957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers allocate uniform computation to every position, regardless of difficulty. State Space Models (SSMs) offer efficient alternatives but struggle with precise information retrieval over a long horizon. Inspired by dual-process theories of cognition (Kahneman, 2011), we propose AMOR (Adaptive Metacognitive Output Router), a hybrid architecture that dynamically engages sparse attention only when an SSM backbone is "uncertain"--as measured by prediction entropy. Compared to standard transformers, AMOR gains efficiency by projecting keys and values from SSM hidden states (Ghost KV), reusing the SSM's O(n) computation rather than requiring O(n^2) attention at every layer. On small-scale synthetic retrieval tasks, AMOR outperforms both SSM-only and transformer-only baselines, achieving perfect retrieval accuracy while engaging attention on only 22% of positions. We validate that prediction entropy reliably signals retrieval need, with a gap of 1.09 nats (nearly half the entropy range) between retrieval and local positions. Additionally, our approach provides interpretable adaptive computation, where routing decisions can be understood in information-theoretic terms.

Related papers

• Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates [13.40796672049436]
  This paper proposes Ada2MS, an optimization algorithm that achieves a smooth transition between AdamW-like behavior and momentum-SGD-like behavior.<n>On the visual tasks evaluated in this study, Ada2MS obtains competitive results under a unified-comparison protocol.
  arXiv  Detail & Related papers  (2026-05-19T22:11:04Z)
• TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles [12.357939380869526]
  State Space Models (SSMs) have emerged as a compelling alternative to attention models for long-range vision tasks.<n>We propose Token-Conditioned Poles SSM, a structured selective SSM framework that improves efficiency while making recurrence dynamics explicit.<n>We show that TCP-SSM reduces SSM computation complexity up to 44% in Vision Mamba-style models while maintaining or surpassing baseline accuracy.
  arXiv  Detail & Related papers  (2026-05-12T05:49:46Z)
• SGA-MCTS: Decoupling Planning from Execution via Training-Free Atomic Experience Retrieval [74.1918709002557]
  We introduce textbfSGA-MCTS, a framework that casts LLM planning as non-parametric retrieval.<n>Online, a retrieval-augmented agent employs a hybrid symbolic-semantic mechanism to fetch relevant State-Goal-Action atoms.<n>By effectively amortizing the heavy computational cost of search, SGA-MCTS achieves System 2 reasoning depth at System 1 inference speeds, rendering autonomous planning both scalable and real-time feasible.
  arXiv  Detail & Related papers  (2026-04-16T07:22:36Z)
• Architectural Proprioception in State Space Models: Thermodynamic Training Induces Anticipatory Halt Detection [0.0]
  We introduce the Probability Navigation Architecture (PNA), which treats computation as navigation through a probability manifold governed by thermodynamic principles.<n>We train State Space Models (SSMs) and Transformers with a novel thermodynamic loss function that penalizes computational waste alongside standard cross-entropy.
  arXiv  Detail & Related papers  (2026-03-04T15:36:40Z)
• Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures [47.30551127397794]
  We investigate whether hybrid architectures combining Transformers and State Space Models can achieve the best of both worlds on two synthetic in-context retrieval tasks.<n>We find that hybrid models outperform SSMs and match or exceed Transformers in data efficiency and extrapolation for information-dense context retrieval.
  arXiv  Detail & Related papers  (2026-03-03T11:28:33Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• AMORE: Adaptive Multi-Output Operator Network for Stiff Chemical Kinetics [4.621457883636921]
  Time integration of stiff systems is a primary source of computational cost in combustion, hypersonics, and other reactive transport systems.<n>We develop AMORE, a framework comprising an operator capable of predicting multiple outputs and adaptive loss functions.
  arXiv  Detail & Related papers  (2025-10-15T00:43:30Z)
• HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inference [8.057006406834462]
  Large Language Models (LLMs) have driven a growing demand for efficient inference in latency-sensitive applications.<n>We present HALO, a heterogeneous memory-centric accelerator for these challenges.<n>We show that HALO achieves up to 18x geometric mean speedup over AttAcc, an attention-optimized mapping and 2.5x over CENT.
  arXiv  Detail & Related papers  (2025-10-03T02:20:17Z)
• Fast attention mechanisms: a tale of parallelism [52.7657529272906]
  We introduce an efficient attention mechanism called Approximate Nearest Neighbor Attention (ANNA) with sub-quadratic time complexity.<n>We prove that ANNA-transformers retain the expressive power previously established for standard attention in terms of matching the capabilities of MPC algorithms.
  arXiv  Detail & Related papers  (2025-09-10T20:59:44Z)
• SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot [8.080568103779893]
  State-space language models such as Mamba match Transformer quality while permitting linear complexity inference.<n>Existing one-shot pruning methods are tailored to attention blocks and fail to account for the time-shared and discretized state-transition matrix.<n>We introduce SparseSSM, the first training-free pruning framework that extends the classic optimal brain surgeon (OBS) framework to state space architectures.
  arXiv  Detail & Related papers  (2025-06-11T11:14:57Z)
• Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems [49.819436680336786]
  We propose an efficient transformed Gaussian process state-space model (ETGPSSM) for scalable and flexible modeling of high-dimensional, non-stationary dynamical systems.<n>Specifically, our ETGPSSM integrates a single shared GP with input-dependent normalizing flows, yielding an expressive implicit process prior that captures complex, non-stationary transition dynamics.<n>Our ETGPSSM outperforms existing GPSSMs and neural network-based SSMs in terms of computational efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-03-24T03:19:45Z)
• Toward Relative Positional Encoding in Spiking Transformers [76.72869420863749]
  Spiking neural networks (SNNs) are bio-inspired networks that mimic how neurons in the brain communicate through discrete spikes.<n>We introduce several strategies to approximate relative positional encoding in spiking Transformers while preserving the binary nature of spikes.
  arXiv  Detail & Related papers  (2025-01-28T06:42:37Z)
• Resource Allocation of Federated Learning for the Metaverse with Mobile Augmented Reality [13.954907748381743]
  Metaverse applications via mobile augmented reality (MAR) require rapid and accurate object detection to mix digital data with the real world. Federated learning (FL) is an intriguing distributed machine learning approach due to its privacy-preserving characteristics. We formulate an optimization problem to minimize a weighted combination of total energy consumption, completion time and model accuracy.
  arXiv  Detail & Related papers  (2022-11-16T06:37:32Z)
• Learning towards Synchronous Network Memorizability and Generalizability for Continual Segmentation across Multiple Sites [52.84959869494459]
  In clinical practice, a segmentation network is often required to continually learn on a sequential data stream from multiple sites. Existing methods are usually restricted in either network memorizability on previous sites or generalizability on unseen sites. This paper aims to tackle the problem of Synchronous Memorizability and Generalizability with a novel proposed SMG-learning framework.
  arXiv  Detail & Related papers  (2022-06-14T13:04:36Z)
• Pruning Self-attentions into Convolutional Layers in Single Path [89.55361659622305]
  Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. We propose Single-Path Vision Transformer pruning (SPViT) to efficiently and automatically compress the pre-trained ViTs. Our SPViT can trim 52.0% FLOPs for DeiT-B and get an impressive 0.6% top-1 accuracy gain simultaneously.
  arXiv  Detail & Related papers  (2021-11-23T11:35:54Z)
• Faster Depth-Adaptive Transformers [71.20237659479703]
  Depth-adaptive neural networks can dynamically adjust depths according to the hardness of input words. Previous works generally build a halting unit to decide whether the computation should continue or stop at each layer. In this paper, we get rid of the halting unit and estimate the required depths in advance, which yields a faster depth-adaptive model.
  arXiv  Detail & Related papers  (2020-04-27T15:08:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.