Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses

• URL: http://arxiv.org/abs/2609.09902v1
• Date: Wed, 09 Sep 2026 09:01:09 GMT
• Title: Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses
• Abstract summary: We use a training-free tracer that reads at every position, sub-layer, and head average.<n>It traces a compound- noun->attention chain in Phi-2, confirmed there by activation patching.<n>It reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a set of domain-to-domain mappings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reading a transformer's internal states in token space is easy to do and hard to trust: a logit lens on a single hidden state is dominated, at intermediate layers, by the generic tokens the model would predict for almost any input. We read the difference instead. Subtracting two closely matched prompts' hidden states and projecting through the unembedding cancels the shared component and surfaces what separates them, an operation equivalent to reading a RepE/ActAdd steering vector through a logit lens. Built into a training-free tracer that reads at every position, sub-layer, and head and averages over designed baselines, it traces a compound- noun MLP->attention chain in Phi-2, confirmed there by activation patching, with the same distinction recovered across three architectures by readout and probe rather than by patching; it reads what retrieval surfaces for real versus fictional entities, and reads metaphor as a set of domain-to-domain mappings rather than a single figurativity feature. A cross-seed control marks the boundary: across five networks differing only in initialization, the same distinction surfaces as almost entirely different tokens (top-10 overlap 0.08). What a computation looks like in token space is network-specific; the distinction it draws is not

Related papers

• Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens [11.948805377284694]
  A lens reading reflects both the hidden state and the readout used to decode it.<n>We introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights.<n>SRP reveals readout features as a new unit of analysis for lens readings.
  arXiv  Detail & Related papers  (2026-09-01T23:01:56Z)
• Metaphor Tracer: A Theory-Informed Analysis of Hidden States [0.0]
  From one forward pass, without training, we score every token position on two properties.<n>The *aggregator* measures whether the position consolidates the whole text into a stable configuration.<n>The *differentiator*, whether other tokens are transiently carried into its subspace as the model reads.
  arXiv  Detail & Related papers  (2026-07-30T16:12:13Z)
• Controlling Embedding Spaces with Text-Conditioned Transformations [54.94594915955756]
  We propose a text-conditioned transformation of visual embeddings that makes attributes explicitly accessible.<n> Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface.
  arXiv  Detail & Related papers  (2026-07-24T21:14:29Z)
• TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection [62.95726973851089]
  TokenCLIP is a token-wise adaptation framework for anomaly learning.<n>It enables dynamic alignment between visual and learnable textual spaces for fine-grained anomaly learning.
  arXiv  Detail & Related papers  (2025-10-24T05:51:31Z)
• TCFormer: Visual Recognition via Token Clustering Transformer [79.24723479088097]
  We propose the Token Clustering Transformer (TCFormer), which generates dynamic vision tokens based on semantic meaning. Our dynamic tokens possess two crucial characteristics: (1) Representing image regions with similar semantic meanings using the same vision token, even if those regions are not adjacent, and (2) concentrating on regions with valuable details and represent them using fine tokens.
  arXiv  Detail & Related papers  (2024-07-16T02:26:18Z)
• SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding [56.079013202051094]
  We present SegVG, a novel method transfers the box-level annotation as signals to provide an additional pixel-level supervision for Visual Grounding. This approach allows us to iteratively exploit the annotation as signals for both box-level regression and pixel-level segmentation.
  arXiv  Detail & Related papers  (2024-07-03T15:30:45Z)
• Cross-domain Detection Transformer based on Spatial-aware and Semantic-aware Token Alignment [31.759205815348658]
  We propose a new method called Spatial-aware and Semantic-aware Token Alignment (SSTA) for cross-domain detection transformers. For spatial-aware token alignment, we can extract the information from the cross-attention map (CAM) to align the distribution of tokens according to their attention to object queries. For semantic-aware token alignment, we inject the category information into the cross-attention map and construct domain embedding to guide the learning of a multi-class discriminator.
  arXiv  Detail & Related papers  (2022-06-01T04:13:22Z)
• Augmenting Convolutional networks with attention-based aggregation [55.97184767391253]
  We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth) It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption.
  arXiv  Detail & Related papers  (2021-12-27T14:05:41Z)
• SWAT: Spatial Structure Within and Among Tokens [53.525469741515884]
  We argue that models can have significant gains when spatial structure is preserved during tokenization. We propose two key contributions: (1) Structure-aware Tokenization and, (2) Structure-aware Mixing.
  arXiv  Detail & Related papers  (2021-11-26T18:59:38Z)
• AAformer: Auto-Aligned Transformer for Person Re-Identification [82.45385078624301]
  We introduce an alignment scheme in transformer architecture for the first time. We propose the auto-aligned transformer (AAformer) to automatically locate both the human parts and nonhuman ones at patch level. AAformer integrates the part alignment into the self-attention and the output [PART]s can be directly used as part features for retrieval.
  arXiv  Detail & Related papers  (2021-04-02T08:00:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.