Emergence Transformer: Dynamical Temporal Attention Matters

• URL: http://arxiv.org/abs/2604.19816v1
• Date: Sat, 18 Apr 2026 01:10:44 GMT
• Title: Emergence Transformer: Dynamical Temporal Attention Matters
• Abstract summary: The Transformer, a breakthrough architecture in artificial intelligence, owes its success to the attention mechanism.<n>We propose an Emergence Transformer, allowing each component to interact with its own or its neighbors' past states.<n>We show how DTA reshapes social coherence, suggesting strategies to either enhance agreement or preserve plurality.
• Score: 8.268321002607737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Transformer, a breakthrough architecture in artificial intelligence, owes its success to the attention mechanism, which utilizes long-range interactions in sequential data, enabling the emergent coherence between large language models (LLMs) and data distributions. However, temporal attention, that is, different forms of long-range interactions in temporal sequences, has rarely been explored in emergence phenomenon of complex systems including oscillatory coherence in quantum, biophysical, or climate systems. Here, by designing dynamical temporal attention (DTA) with time-varying query, key, and value matrices, we propose an Emergence Transformer. This architecture allows each component to interact with its own or its neighbors' past states through dynamical attention kernels, thereby enabling the promotion and/or suppression of the emergent coherence of components. Interestingly, we uncover that neighbor-DTA consistently promotes oscillatory coherence, whereas self-DTA exhibits an optimal attention weight for coherence enhancement, owing to its non-monotonic dependence on network structure. Practically, we demonstrate how DTA reshapes social coherence, suggesting strategies to either enhance agreement or preserve plurality. We further apply DTA to the paradigmatic Hopfield neural network, achieving emergent continual learning without catastrophic forgetting. Together, these results lay a foundation and provide an immediate paradigm for modulating emergence phenomenon in networked dynamics only using DTA.

Related papers

• STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition [0.7799711162530713]
  Micro-expression recognition is challenging due to subtle and short-lived facial muscle movements.<n>Existing methods rely heavily on apex-onset frames, overlook fine-grained inter-frame dynamics, and separately model spatial and temporal information.<n>We propose STAG, a dynamic ROI-AU-coupled spatial-temporal network that jointly models motion flow and adaptive facial connectivity.
  arXiv  Detail & Related papers  (2026-06-26T13:46:48Z)
• Controlled Dynamics Attractor Transformer [58.15454173718687]
  Transformer architectures have advanced representation learning and inference in deep models through self-attention mechanisms.<n>In parallel,associative memory (AM) frameworks map representations onto energy landscapes, offering interpretable retrieval mechanisms.<n>We propose Controlled Dynamics Attractor Transformer (CCAT), which couples a mixture von Mises-Fisher (Mo-vMF) attention energy with a Hopfield refinement energy.<n>CCAT achieves state-of-the-art performance across multiple benchmarks in graph anomaly detection and graph classification.
  arXiv  Detail & Related papers  (2026-06-13T09:04:22Z)
• Forget Less, Generalize More: Unifying Temporal and Structural Adaptation for Dynamic Graphs [9.230468469629345]
  We propose a unified framework that maintains a retentive representation state encoding both temporal memory and structural context.<n>We show that Dual-Scale Retentive Dynamics consistently achieves state-of-the-art performance on both link prediction and node classification tasks.
  arXiv  Detail & Related papers  (2026-05-28T06:47:08Z)
• Dynamic Link Prediction with Temporally Enhanced Signed Graph Neural Networks [1.5259434915412646]
  Temporal signed networks (TSNs) model the time evolution of cooperative and adversarial relationships.<n>We propose a emphmodular temporal enhancement framework for signed GNNs that integrates historical context into otherwise static architectures.<n>Experiments on real-world and synthetic TSNs, including Bitcoin OTC, Bitcoin Alpha, Reddit, and small-world network models, demonstrate consistent and statistically significant improvements.
  arXiv  Detail & Related papers  (2026-05-25T19:28:01Z)
• DySCo: Dynamic Semantic Compression for Effective Long-term Time Series Forecasting [6.800568640042171]
  Time series forecasting is critical across domains such as finance, meteorology, and energy.<n>We propose a Dynamic Semantic Compression (DySCo) framework to address these challenges.<n>DySCo serves as a universal plug-and-play module, significantly enhancing the ability of mainstream models to capture long-term correlations.
  arXiv  Detail & Related papers  (2026-04-01T12:40:23Z)
• Temporal Graph Pattern Machine [17.352525018007473]
  Temporal Graph Pattern Machine (TGPM) conceptualizes each interaction as an interaction patch synthesized via temporally-biased random walks.<n>TGPM consistently achieves state-of-the-art performance in both transductive and inductive link prediction.
  arXiv  Detail & Related papers  (2026-01-30T01:46:13Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)
• PowerGrow: Feasible Co-Growth of Structures and Dynamics for Power Grid Synthesis [75.14189839277928]
  We present PowerGrow, a co-generative framework that significantly reduces computational overhead while maintaining operational validity.<n> Experiments across benchmark settings show that PowerGrow outperforms prior diffusion models in fidelity and diversity.<n>This demonstrates its ability to generate operationally valid and realistic power grid scenarios.
  arXiv  Detail & Related papers  (2025-08-29T01:47:27Z)
• Learning Locally Interacting Discrete Dynamical Systems: Towards Data-Efficient and Scalable Prediction [16.972017028598597]
  Local dynamical systems exhibit complex global dynamics from local, relatively simple, and often interactions between dynamic elements. We present Attentive Recurrent Cellular Automata (AR-NCA), to effectively discover unknown local state transition rules. AR-NCA exhibits the superior generalizability across various system configurations.
  arXiv  Detail & Related papers  (2024-04-09T17:00:43Z)
• Multi-Scale Spatial-Temporal Self-Attention Graph Convolutional Networks for Skeleton-based Action Recognition [0.0]
  In this paper, we propose self-attention GCN hybrid model, Multi-Scale Spatial-Temporal self-attention (MSST)-GCN. We utilize spatial self-attention module with adaptive topology to understand intra-frame interactions within a frame among different body parts, and temporal self-attention module to examine correlations between frames of a node.
  arXiv  Detail & Related papers  (2024-04-03T10:25:45Z)
• Relational Self-Attention: What's Missing in Attention for Video Understanding [52.38780998425556]
  We introduce a relational feature transform, dubbed the relational self-attention (RSA) Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts.
  arXiv  Detail & Related papers  (2021-11-02T15:36:11Z)
• Learn to cycle: Time-consistent feature discovery for action recognition [83.43682368129072]
  Generalizing over temporal variations is a prerequisite for effective action recognition in videos. We introduce Squeeze Re Temporal Gates (SRTG), an approach that favors temporal activations with potential variations. We show consistent improvement when using SRTPG blocks, with only a minimal increase in the number of GFLOs.
  arXiv  Detail & Related papers  (2020-06-15T09:36:28Z)
• A Spatial-Temporal Attentive Network with Spatial Continuity for Trajectory Prediction [74.00750936752418]
  We propose a novel model named spatial-temporal attentive network with spatial continuity (STAN-SC) First, spatial-temporal attention mechanism is presented to explore the most useful and important information. Second, we conduct a joint feature sequence based on the sequence and instant state information to make the generative trajectories keep spatial continuity.
  arXiv  Detail & Related papers  (2020-03-13T04:35:50Z)
• Spatial-Temporal Transformer Networks for Traffic Flow Forecasting [74.76852538940746]
  We propose a novel paradigm of Spatial-Temporal Transformer Networks (STTNs) to improve the accuracy of long-term traffic forecasting. Specifically, we present a new variant of graph neural networks, named spatial transformer, by dynamically modeling directed spatial dependencies. The proposed model enables fast and scalable training over a long range spatial-temporal dependencies.
  arXiv  Detail & Related papers  (2020-01-09T10:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.