Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix

• URL: http://arxiv.org/abs/2605.16112v1
• Date: Fri, 15 May 2026 15:58:02 GMT
• Title: Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix
• Abstract summary: We identify attention dispersion as a shared failure mode of dynamic graph Transformers under temporal distribution shift.<n>We show that prediction depends on a class of critical nodes that carry consistently more predictive signal than arbitrary neighbors.<n>We introduce DiffDyG, a reference implementation combining differential attention with standard input encodings.
• Score: 19.55465761427973
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based architectures have become the dominant paradigm for Continuous-Time Dynamic Graph (CTDG) learning, yet their performance remains limited on temporally shifted datasets. In this work, we identify attention dispersion as a shared failure mode of dynamic graph Transformers under temporal distribution shift. Through controlled ablation contrasting structurally and temporally distinguished historical neighbors against random ones, we show that prediction depends on a class of critical nodes that carry consistently more predictive signal than arbitrary neighbors. However, existing Transformers fail to focus on these nodes even when they are present in the input, as temporal shift weakens attention contrast and produces overly dispersed attention distributions. This diagnosis suggests a simple and transferable fix: replace standard attention with differential attention, which suppresses common-mode attention and amplifies distinctive token-level signals. When added to three representative CTDG Transformer baselines, differential attention consistently improves performance, with gains concentrated on high-shift datasets. Attention-level measurements further confirm the mechanism, showing reduced attention entropy and increased attention mass on critical nodes. Building on these findings, we introduce DiffDyG, a reference implementation combining differential attention with standard input encodings. Across 9 benchmarks and three negative sampling protocols, DiffDyG achieves SOTA performance, with especially large gains on the most shifted datasets.

Related papers

• Beyond Sparse Supervision: Diffusion-Guided Learning for Few-Shot Graph Fraud Detection [4.923071905608023]
  We propose ADC-GNN, a unified framework that combines diffusion-guided feature augmentation, contrastive representation learning, and multi-hop spectral attention for few-shot fraud detection.<n>We evaluate ADC-GNN primarily on three public benchmarks and report a proprietary real-world telecom transaction dataset with approximately 60,000 records as a private case study.
  arXiv  Detail & Related papers  (2026-06-26T14:34:28Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• LEFT: Learnable Fusion of Tri-view Tokens for Unsupervised Time Series Anomaly Detection [53.191369031661885]
  Unsupervised time series anomaly detection aims to build a model for identifying abnormal timestamps without assuming the availability of annotations.<n>We present Learnable Fusion of Tri-view Tokens (LEFT), a unified unsupervised TSAD framework that models anomalies as inconsistencies across complementary representations.<n>Experiments on real-world benchmarks show that LEFT yields the best detection accuracy against SOTA baselines, while achieving a 5x reduction on FLOPs and 8x speed-up for training.
  arXiv  Detail & Related papers  (2026-02-09T13:33:49Z)
• Pi-Transformer: A Physics-informed Attention Mechanism for Time Series Anomaly Detection [3.8673630752805437]
  We present Pi-Transformer, a physics-informed transformer with two attention pathways.<n>The model combines an alignment-weighted reconstruction signal with a mismatch signal that highlights timing and phase disruptions.<n>Pi-Transformer achieves state-of-the-art or highly competitive F1, with particular strength on timing and phase-breaking anomalies.
  arXiv  Detail & Related papers  (2025-09-24T10:47:48Z)
• Time-o1: Time-Series Forecasting Needs Transformed Label Alignment [50.54348432664401]
  Time-o1 is a transformation-augmented learning objective tailored for time-series forecasting.<n>The central idea is to transform the label sequence into decorrelated components with discriminated significance.<n>Time-o1 achieves state-of-the-art performance and is compatible with various forecast models.
  arXiv  Detail & Related papers  (2025-05-23T13:00:35Z)
• DAM-GT: Dual Positional Encoding-Based Attention Masking Graph Transformer for Node Classification [16.82685813961535]
  We propose DAM-GT, Dual positional encoding-based Attention Masking graph Transformer.<n>We show that DAM-GT consistently outperforms state-of-the-art methods in node classification tasks.
  arXiv  Detail & Related papers  (2025-05-23T09:23:41Z)
• Semi-supervised Anomaly Detection with Extremely Limited Labels in Dynamic Graphs [5.415950005432774]
  We propose a novel GAD framework (EL$2$-DGAD) to tackle anomaly detection problem in dynamic graphs with extremely limited labels.<n>Specifically, a transformer-based graph encoder model is designed to more effectively preserve evolving graph structures beyond the local neighborhood.
  arXiv  Detail & Related papers  (2025-01-25T02:35:48Z)
• Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
  We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture.<n>We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
  arXiv  Detail & Related papers  (2024-09-20T07:41:47Z)
• Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators [83.48423407316713]
  We present a novel diffusion transformer framework incorporating an additional set of mediator tokens to engage with queries and keys separately. Our model initiates the denoising process with a precise, non-ambiguous stage and gradually transitions to a phase enriched with detail. Our method achieves a state-of-the-art FID score of 2.01 when integrated with the recent work SiT.
  arXiv  Detail & Related papers  (2024-08-11T07:01:39Z)
• Detecting Anomalies in Dynamic Graphs via Memory enhanced Normality [39.476378833827184]
  Anomaly detection in dynamic graphs presents a significant challenge due to the temporal evolution of graph structures and attributes. We introduce a novel spatial- temporal memories-enhanced graph autoencoder (STRIPE) STRIPE significantly outperforms existing methods with 5.8% improvement in AUC scores and 4.62X faster in training time.
  arXiv  Detail & Related papers  (2024-03-14T02:26:10Z)
• Attention as Robust Representation for Time Series Forecasting [23.292260325891032]
  Time series forecasting is essential for many practical applications. Transformers' key feature, the attention mechanism, dynamically fusing embeddings to enhance data representation, often relegating attention weights to a byproduct role. Our approach elevates attention weights as the primary representation for time series, capitalizing on the temporal relationships among data points to improve forecasting accuracy.
  arXiv  Detail & Related papers  (2024-02-08T03:00:50Z)
• Accelerating Vision Transformers Based on Heterogeneous Attention Patterns [89.86293867174324]
  Vision Transformers (ViTs) have attracted a lot of attention in the field of computer vision. We propose an integrated compression pipeline based on observed heterogeneous attention patterns across layers. Experimentally, the integrated compression pipeline of DGSSA and GLAD can accelerate up to 121% run-time throughput.
  arXiv  Detail & Related papers  (2023-10-11T17:09:19Z)
• CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting [50.23240107430597]
  We design a special Transformer, i.e., Channel Aligned Robust Blend Transformer (CARD for short), that addresses key shortcomings of CI type Transformer in time series forecasting. First, CARD introduces a channel-aligned attention structure that allows it to capture both temporal correlations among signals. Second, in order to efficiently utilize the multi-scale knowledge, we design a token blend module to generate tokens with different resolutions. Third, we introduce a robust loss function for time series forecasting to alleviate the potential overfitting issue.
  arXiv  Detail & Related papers  (2023-05-20T05:16:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.