Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction

• URL: http://arxiv.org/abs/2609.08106v1
• Date: Tue, 08 Sep 2026 01:34:53 GMT
• Title: Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction
• Abstract summary: MASTER's inter-stock multi-head attention accounts for 42.5% of model parameters and 25% of predictive value.<n>We uncover a surprising structure: the learned attention is near-uniform (perplexity 278/300), yet forcing exact uniformity eliminates all cross-sectional discrimination.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: MASTER's inter-stock multi-head attention -- the module responsible for modeling cross-sectional stock relationships -- accounts for 42.5% of model parameters and 25% of predictive value. We systematically decompose this module and uncover a surprising structure: the learned attention is near-uniform (perplexity 278/300), yet forcing exact uniformity eliminates all cross-sectional discrimination. Spectral analysis resolves this paradox: the deviation from uniformity is low-rank (effective rank ~65, top-10 modes capture 96.5% of energy), explaining why sparse approximations consistently fail while Nystrom low-rank attention (m=32 landmarks) matches full O(N^2) attention at O(mN) cost -- certified equivalent via TOST at both N=300 (5 seeds, Rank IC p=0.003) and N=800 (10 seeds, Rank IC p=0.034). Additional findings include: (i) attention anti-correlates with return similarity (Spearman rho = -0.614; on the industry-labeled subset, -0.645 unconditionally and -0.627 after controlling for industry, beta, and volatility), suggesting complementarity-seeking rather than correlation mining; (ii) all graph-based alternatives degrade performance, with hard masking worse than complete module removal; and (iii) at N ~ 3,500 with adapted architectures, no cross-stock module (GCN, Nystrom, or MASTER-style pipeline) significantly outperforms a per-stock LSTM baseline (n=4 seeds), indicating that the benefits observed at smaller scales do not trivially transfer. These results establish that the inter-stock attention's value resides in a compressible, dynamic, near-global redistribution that rewards low-rank approximation but resists sparsification.

Related papers

• Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models [0.0]
  We formulate quantization as a joint optimization over the discrete codes and scales of all layers.<n>We show that error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher.<n>On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher.
  arXiv  Detail & Related papers  (2026-07-16T06:47:52Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Robust Dual-Signal Fusion: Hybrid Neuro-Symbolic Gating with Compressed Chain-of-Thought Refinement for Irony Detection in Social Media Texts [0.0]
  We introduce the Robust Dual-Signal (RDS) Fusion, a neuro-symbolic architecture that compresses Chain-of-Thought (CoT) reasoning trajectories without Supervised Fine-Tuning (SFT)<n>RDS achieves $78.1%$ accuracy and a Macro F1 of $0.777$, matching the absolute performance ceiling of a fine-tuned BERTweet.<n> RDS fusion is statistically insignificant compared to the combined RoBERTa ablation; the concurrent synergy achieves a statistically significant improvement over the standalone baseline.
  arXiv  Detail & Related papers  (2026-06-15T15:22:37Z)
• Constraint-Aware Optimization for Robust Protein Stability Prediction [0.3316543849024811]
  Multimodal $G$ integrating protein language models with inverse-folding representations achieve strong in-distribution accuracy on the Megascale dataset.<n>Existing approaches address these limitations primarily through additional architectural components.<n>We introduce a constraint-aware optimization framework combining Balanced Mean Squared Error, a Siamese anti-symmetric regularizer, and a novel OOD-margin loss on the per-position predictors feature representation.
  arXiv  Detail & Related papers  (2026-06-06T11:04:52Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization [58.03221830946145]
  We propose OmniISR, a unified framework that fuses pure CL, pure FL, and hybrid CL-FL training modes.<n>We show that OmniISR consistently improves model performance in both centralized and federated paradigms.
  arXiv  Detail & Related papers  (2026-05-19T04:13:27Z)
• A Penalty-Free Pipeline for Direct Quantum-Annealer Portfolio Optimization [0.0]
  Direct quantum-annealer portfolio optimization is commonly formulated as a penalty-encoded QUBO submitted to D-Wave hardware.<n>We show that this standard formulation fails on current devices and identify the structural reason.<n>We propose dropping the penalty entirely: build an objective-only QUBO from the expected returns and sample it on hardware.
  arXiv  Detail & Related papers  (2026-05-17T19:50:04Z)
• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction [0.0017904458681854366]
  We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1)<n>Our system achieves 94.3% accuracy with a content effect of 2.85 and a combined score of 41.88 in nested 5-fold cross-validation on the dataset (N=960)<n>Our results suggest that targeted neuro-symbolic integration, applying formal methods precisely where ensemble consensus is lowest, can improve the combined accuracy-plus-content-effect metric used by this task.
  arXiv  Detail & Related papers  (2026-04-20T14:31:00Z)
• Do Generative Metrics Predict YOLO Performance? An Evaluation Across Models, Augmentation Ratios, and Dataset Complexity [43.338311770275745]
  We present a controlled evaluation of synthetic augmentation for YOLOv11 across three single-class detection regimes.<n>We benchmark six GAN-, diffusion-, and hybrid-based generators over augmentation ratios from 10% to 150% of the real training split.<n>For each dataset-generator-augmentation configuration, we compute pre-training dataset metrics under a matched-size bootstrap protocol.
  arXiv  Detail & Related papers  (2026-02-20T03:02:36Z)
• Structure-Informed Estimation for Pilot-Limited MIMO Channels via Tensor Decomposition [51.56484100374058]
  This paper formulates pilot-limited channel estimation as low-rank tensor completion from sparse observations.<n>Experiments on synthetic channels demonstrate 10-20,dB normalized mean-square error (NMSE) improvement over least-squares (LS)<n> evaluations on DeepMIMO ray-tracing channels show 24-44% additional NMSE reduction over pure tensor-based methods.
  arXiv  Detail & Related papers  (2026-02-03T23:38:05Z)
• Predictive Coding and Information Bottleneck for Hallucination Detection in Large Language Models [0.8552050317027305]
  Hallucinations in Large Language Models (LLMs) remain a critical barrier to high-stakes deployment.<n>We introduce [Model Name], a hybrid detection framework that combines neuroscience-inspired signal design with supervised machine learning.
  arXiv  Detail & Related papers  (2026-01-22T05:00:21Z)
• SKANet: A Cognitive Dual-Stream Framework with Adaptive Modality Fusion for Robust Compound GNSS Interference Classification [47.20483076887704]
  Global Navigation Satellite Systems (GNSS) face growing threats from sophisticated jamming interference.<n>We propose a cognitive deep learning framework built upon a dual-stream architecture that integrates Time-Frequency Images (TFIs) and Power Spectral Density (PSD)<n>We show that SKANet achieves an overall accuracy of 96.99%, exhibiting superior robustness for compound jamming classification.
  arXiv  Detail & Related papers  (2026-01-19T07:42:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.