Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling

• URL: http://arxiv.org/abs/2609.08981v2
• Date: Fri, 11 Sep 2026 00:07:59 GMT
• Title: Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
• Abstract summary: We show that frozen transformers can simulate iterative samplers from in-context samples.<n>We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers.<n>We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
• Score: 20.11728693585654
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to \emph{data generation}: frozen transformers can simulate iterative generative samplers from in-context samples. We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers. The construction identifies a concrete generative role for softmax attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained language models, we study \emph{semantic-topic sampling}: prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.

Related papers

• Inverting Data Transformations via Diffusion Sampling [33.6473603169954]
  We study the problem of transformation inversion on general Lie groups.<n>The goal is to recover an inverse transformation that maps it back to the original data distribution.<n>We take a probabilistic view and model the posterior over transformations as a Boltzmann distribution.
  arXiv  Detail & Related papers  (2026-02-09T04:58:34Z)
• QLENS: Towards A Quantum Perspective of Language Transformers [14.34282709638159]
  We present QLENS, a novel attempt to develop a physics-based perspective on the Transformer generation process.<n>Under QLENS, a Transformer is studied by converting its latent activations into a state vector in a Hilbert space derived from the model's output units.<n>The model's final probability distribution is obtained by applying the Born rule to the end state using a specific measurement operator.
  arXiv  Detail & Related papers  (2025-10-13T21:53:05Z)
• ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior [39.590138981646696]
  Post-hoc interpretability methods typically attribute a model's behavior to its components, data, or training trajectory in isolation.<n>We present ExPLAIND, a unified framework that integrates all these perspectives.
  arXiv  Detail & Related papers  (2025-05-26T14:53:11Z)
• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• In-Context Learning with Representations: Contextual Generalization of Trained Transformers [66.78052387054593]
  In-context learning (ICL) refers to a capability of pretrained large language models, which can learn a new task given a few examples during inference. This paper investigates the training dynamics of transformers by gradient descent through the lens of non-linear regression tasks.
  arXiv  Detail & Related papers  (2024-08-19T16:47:46Z)
• What Secrets Do Your Manifolds Hold? Understanding the Local Geometry of Generative Models [17.273596999339077]
  We study the local geometry of the learned manifold and its relationship to generation outcomes for a wide range of generative models.<n>We provide quantitative and qualitative evidence showing that for a given latent-image pair, the local descriptors are indicative of generation aesthetics, diversity, and memorization by the generative model.
  arXiv  Detail & Related papers  (2024-08-15T17:59:06Z)
• Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation [52.270712965271656]
  We propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. We find that the graph of our model resembles transformers, with correspondences between dependencies and self-attention. Experiments show that our model performs competitively to transformers on small to medium sized datasets.
  arXiv  Detail & Related papers  (2023-11-26T06:56:02Z)
• Grokking of Hierarchical Structure in Vanilla Transformers [72.45375959893218]
  We show that transformer language models can learn to generalize hierarchically after training for extremely long periods. intermediate-depth models generalize better than both very deep and very shallow transformers.
  arXiv  Detail & Related papers  (2023-05-30T04:34:13Z)
• Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation [66.86987509942607]
  We evaluate how such a paradigm should be done in imitation learning. We consider a setting where the pretraining corpus consists of multitask demonstrations. We argue that inverse dynamics modeling is well-suited to this setting.
  arXiv  Detail & Related papers  (2023-05-26T14:40:46Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.