Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis

• URL: http://arxiv.org/abs/2604.08829v1
• Date: Fri, 10 Apr 2026 00:00:11 GMT
• Title: Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis
• Abstract summary: The Hierarchical Kernel Transformer (HKT) is a multi-scale attention mechanism that processes sequences at L resolution levels via trainable causal downsampling.<n>The total computational cost is bounded by 4/3 times that of standard attention, reaching 1.3125x for L = 3.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Hierarchical Kernel Transformer (HKT) is a multi-scale attention mechanism that processes sequences at L resolution levels via trainable causal downsampling, combining level-specific score matrices through learned convex weights. The total computational cost is bounded by 4/3 times that of standard attention, reaching 1.3125x for L = 3. Four theoretical results are established. (i) The hierarchical score matrix defines a positive semidefinite kernel under a sufficient condition on the symmetrised bilinear form (Proposition 3.1). (ii) The asymmetric score matrix decomposes uniquely into a symmetric part controlling reciprocal attention and an antisymmetric part controlling directional attention; HKT provides L independent such pairs across scales, one per resolution level (Propositions 3.5-3.6). (iii) The approximation error decomposes into three interpretable components with an explicit non-Gaussian correction and a geometric decay bound in L (Theorem 4.3, Proposition 4.4). (iv) HKT strictly subsumes single-head standard attention and causal convolution (Proposition 3.4). Experiments over 3 random seeds show consistent gains over retrained standard attention baselines: +4.77pp on synthetic ListOps (55.10+-0.29% vs 50.33+-0.12%, T = 512), +1.44pp on sequential CIFAR-10 (35.45+-0.09% vs 34.01+-0.19%, T = 1,024), and +7.47pp on IMDB character-level sentiment (70.19+-0.57% vs 62.72+-0.40%, T = 1,024), all at 1.31x overhead.

Related papers

• Contraction-Gauge Preconditioning for Quantized Matrix Multiplication [0.14680035572775532]
  We study low-precision computation of C=AB with both factors quantized.<n>We derive an exact finite-dimensional identity for the expected squared product error.<n>We empirically assess deterministic round-to-nearest errors.
  arXiv  Detail & Related papers  (2026-07-21T06:09:08Z)
• Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models [0.0]
  We formulate quantization as a joint optimization over the discrete codes and scales of all layers.<n>We show that error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher.<n>On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher.
  arXiv  Detail & Related papers  (2026-07-16T06:47:52Z)
• SPARC-Net: A Spectral, Causality-Aware, and Hard-Constrained Physics-Informed Architecture for Stiff and Shock-Dominated Partial Differential Equations [0.0654051537828115]
  Physics-Informed Neural Networks (PINNs) provide a meshless approach for solving partial differential equations (PDEs)<n>PINNs suffer severe degradation in stiff and shock-dominated problems, where small PDE residuals can correspond to globally inaccurate solutions.<n>We show these failures are multi-causal, arising from the concurrent interplay of (i) spectral bias against sharp features, (ii) imbalanced multi-term optimization and loss-weight collapse, (iii) violation of temporal causality, and (iv) under-resolved collocation.<n>We present-Net, a unified architecture and training
  arXiv  Detail & Related papers  (2026-07-13T09:23:17Z)
• Derivative Informed Learning of Exchange-Correlation Functionals [46.44973075720396]
  Machine-learned (ML) exchange-correlation (XC) functionals aim to replace human-designed density functional approximations.<n>We study a hybrid-distillation setting in which $mathcalO(N4)$-scaling ML-XC functionals are trained to reproduce B3LYP/def2-SVP targets.<n>We introduce Derivative Informed XC-Loss (DI-Loss), a loss that incorporates additional information from the reference hybrid functional.
  arXiv  Detail & Related papers  (2026-06-02T23:07:54Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation [45.56738584872585]
  Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$.<n>We show that a collinear design induces an inherent ill-conditioning in the Gauss-Newton least-squares problem.<n>We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation.
  arXiv  Detail & Related papers  (2026-05-08T23:00:17Z)
• THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture [0.0]
  THEIA is a 2.75M modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives.<n> Transformer baselines also reach >99% on all 39 rules, and flats match THEIA on Phase-1 accuracy within 0.04pp.
  arXiv  Detail & Related papers  (2026-04-13T10:44:15Z)
• Exhaustive Circuit Mapping of a Single-Cell Foundation Model Reveals Massive Redundancy, Heavy-Tailed Hub Architecture, and Layer-Dependent Differentiation Control [0.0]
  We present three experiments that address limitations through exhaustive circuit tracing, higher order ablation, and causal trajectory steering.<n>First, exhaustive tracing of all 4065 active autoencoder features at layer 5 yields 1393850 significant downstream edges.<n>Second, three way redundancy across 8 feature triplets shows that redundancy deepens monotonically with interaction order.<n>Third, trajectory guided feature steering establishes a causal link between layer position and differentiation directionality.
  arXiv  Detail & Related papers  (2026-03-12T13:53:20Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Dual Quaternion SE(3) Synchronization with Recovery Guarantees [18.75848171128639]
  This paper adopts a dual quaternion representation and formulates SE(3) synchronization directly over the unit dual quaternion.<n>A two-stage algorithm is developed: A spectral initializer computed via the power method on a Hermitian dual quaternion measurement matrix, followed by a dual quaternion generalized power method.<n>Experiments on synthetic benchmarks and real-world multi-scan point-set registration demonstrate that the proposed pipeline improves both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-01-30T21:24:20Z)
• Differentiable Logic Synthesis: Spectral Coefficient Selection via Sinkhorn-Constrained Composition [0.0]
  We introduce Hierarchical Spectral Composition, a differentiable architecture that selects spectral coefficients from a frozen Fourier basis.<n>We adapt this framework to logic synthesis, adding column-sign modulation to enable Boolean negation.
  arXiv  Detail & Related papers  (2026-01-20T13:26:52Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Relative Pose from SIFT Features [50.81749304115036]
  We derive a new linear constraint relating the unknown elements of the fundamental matrix and the orientation and scale. The proposed constraint is tested on a number of problems in a synthetic environment and on publicly available real-world datasets on more than 80000 image pairs.
  arXiv  Detail & Related papers  (2022-03-15T14:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.