Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

• URL: http://arxiv.org/abs/2607.00479v1
• Date: Wed, 01 Jul 2026 06:03:14 GMT
• Title: Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
• Abstract summary: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks.<n>With richer context, transformers adapt more effectively to the current use case without any parameter updates.<n>However, the quadratic computational and memory complexity with respect to context length significantly slows data processing in softmax transformers.
• Score: 17.68725202024536
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning. With richer context, transformers adapt more effectively to the current use case without any parameter updates. However, the quadratic computational and memory complexity with respect to context length significantly slows data processing in softmax transformers. Linear transformers were proposed to address this issue by reducing the complexity to linear dependence on context length, but the design and understanding of the feature mapping in linear attention, from a theoretical viewpoint, remain unclear. In this paper, we investigate the approximation and generalization abilities of linear transformers under a two-staged sampling process from domain generalization. We show that linear transformers perform in-context learning as learning a mapping from context distributions to response functions. A dimension-independent convergence rate is obtained for our generalization analysis, which also exhibits the tradeoff between the regularities of data distributions and latent features. Guided by our theoretical framework, we propose a new perspective on activation and loss design for linearizing pretrained softmax large language models.

Related papers

• Adaptive inference and function vectors in deep transformers [6.568378556428861]
  We introduce a theory of a deep transformer as a mean-field interacting system.<n>We show that such a system can exploit internal state representations to infer a latent context variable.<n>In an in-context regression task, the theory predicts a non-trivial relationship between non-Gaussian, hierarchical structure in the latent context variable, and transformer depth.
  arXiv  Detail & Related papers  (2026-06-15T13:30:49Z)
• When Does Context Help? Error Dynamics of Contextual Information in Large Language Models [64.88201012057822]
  We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in large language models.<n>Our analysis characterizes contextual influence through output error dynamics.<n> Experiments across ICL, retrieval-augmented generation, and memory evolution validate our theory and motivate a principled context selection strategy.
  arXiv  Detail & Related papers  (2026-02-09T05:58:41Z)
• Softmax $\geq$ Linear: Transformers may learn to classify in-context by kernel gradient descent [17.629377639287775]
  We focus on understanding the learning algorithm transformers use to learn from context.<n>We find that transformers still learn to do gradient descent in-context, though on functionals in the kernel feature space.<n>These theoretical findings suggest a greater adaptability to context for softmax attention.
  arXiv  Detail & Related papers  (2025-10-12T03:20:27Z)
• Transformer learns the cross-task prior and regularization for in-context learning [11.570071580371964]
  We introduce a linear transformer to learn the inverse mapping from contextual examples to the underlying weight vector.<n>We numerically verify that the error of the transformer estimator scales linearly with the noise level, the ratio of task dimension to context length, and the condition number of the input data.
  arXiv  Detail & Related papers  (2025-05-17T20:42:23Z)
• On the Robustness of Transformers against Context Hijacking for Linear Classification [26.1838836907147]
  Transformer-based Large Language Models (LLMs) have demonstrated powerful in-context learning capabilities.<n>They can be disrupted by factually correct context, a phenomenon known as context hijacking.<n>We show that a well-trained deeper transformer can achieve higher robustness, which aligns with empirical observations.
  arXiv  Detail & Related papers  (2025-02-21T17:31:00Z)
• Learning Linear Attention in Polynomial Time [127.14106124669645]
  We provide the first results on learnability of single-layer Transformers with linear attention.<n>We show that linear attention may be viewed as a linear predictor in a suitably defined RKHS.<n>We show how to efficiently identify training datasets for which every empirical riskr is equivalent to the linear Transformer.
  arXiv  Detail & Related papers  (2024-10-14T02:41:01Z)
• In-Context Learning with Representations: Contextual Generalization of Trained Transformers [66.78052387054593]
  In-context learning (ICL) refers to a capability of pretrained large language models, which can learn a new task given a few examples during inference. This paper investigates the training dynamics of transformers by gradient descent through the lens of non-linear regression tasks.
  arXiv  Detail & Related papers  (2024-08-19T16:47:46Z)
• Linear Transformers are Versatile In-Context Learners [19.988368693379087]
  We prove that each layer of a linear transformer maintains a weight vector for an implicit linear regression problem. We also investigate the use of linear transformers in a challenging scenario where the training data is corrupted with different levels of noise. Remarkably, we demonstrate that for this problem linear transformers discover an intricate and highly effective optimization algorithm.
  arXiv  Detail & Related papers  (2024-02-21T23:45:57Z)
• On the Convergence of Encoder-only Shallow Transformers [62.639819460956176]
  We build the global convergence theory of encoder-only shallow Transformers under a realistic setting. Our results can pave the way for a better understanding of modern Transformers, particularly on training dynamics.
  arXiv  Detail & Related papers  (2023-11-02T20:03:05Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Transformers learn in-context by gradient descent [58.24152335931036]
  Training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass.
  arXiv  Detail & Related papers  (2022-12-15T09:21:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.