Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

• URL: http://arxiv.org/abs/2609.05333v1
• Date: Fri, 04 Sep 2026 16:38:39 GMT
• Title: Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models
• Abstract summary: A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers.<n>Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way.<n>This manual documents an open toolkit built around such a construct, which we call a bridge form.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way. This manual documents an open toolkit built around such a construct, which we call a bridge form: a single written word that recurs, unchanged, across two or more subject domains with a different sense in each. We describe, and justify, every stage of the pipeline: the declarative specification of bridge forms and their source domains, corpus acquisition from Wikipedia, occurrence localization, layer-wise representation extraction, a domain-pairwise silhouette measurement of separation in the model's representation space, and a paired visualization protocol. Each design choice is presented together with the methodological failure mode it is meant to avoid (sense contamination from overly broad category labels, the multi-group bias of the silhouette coefficient, subword-tokenization misalignment, and axis-comparability artifacts in dimensionality-reduced plots, among others). This manuscript is a methodological and implementation reference: it does not report or interpret empirical outcomes of running the toolkit on any particular model or bridge-form set. The toolkit, its full source, and the corpora used to exercise it are archived separately (Section 9) under a persistent identifier, and are intended to be cited as an instrument by studies that use it to produce and interpret empirical results.

Related papers

• An Ontology-Guided, Deduplication-Aware Extraction Layer for Knowledge Graph Construction from Heterogeneous Documents [0.0]
  This paper presents the design, implementation, and empirical refinement of a production extraction layer that converts a live document stream into a validated knowledge graph aligned to a formal ontology.<n>The relevant slice of a curated ontology is retrieved live from a graph database by embedding similarity and injected into the extraction prompt.<n>Extracted results then pass through a refinement pipeline of five stages: deterministic cleaning, merging across chunks, a second pass for relationships, six deduplication algorithms that require no model inference, and an embedding resolution subsystem whose conflict guard no similarity score can override.
  arXiv  Detail & Related papers  (2026-07-22T09:45:26Z)
• Text-as-Signal: Quantitative Semantic Scoring with Embeddings, Logprobs, and Noise Reduction [0.0]
  This paper presents a practical pipeline for turning text corpora into quantitative semantic signals.<n>We show how Qwen embeddings, UMAP, semantic indicators derived directly from the model output space, and a three-stage anomaly-detection procedure combine into an operational text-as-signal workflow.
  arXiv  Detail & Related papers  (2026-03-17T19:56:29Z)
• From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding [22.390804161191635]
  Current state-of-the-art models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. This process known as tokenization relies on a pre-built vocabulary of words or sub-word morphemes. We introduce a novel open-vocabulary language model that adopts a hierarchical two-level approach.
  arXiv  Detail & Related papers  (2023-05-23T23:22:20Z)
• Author Name Disambiguation via Heterogeneous Network Embedding from Structural and Semantic Perspectives [13.266320447769564]
  Name ambiguity is common in academic digital libraries, such as multiple authors having the same name. The proposed method is mainly based on representation learning for heterogeneous networks and clustering. The semantic representation is generated using NLP tools.
  arXiv  Detail & Related papers  (2022-12-24T11:22:34Z)
• LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution [76.615287796753]
  We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models. This is achieved by combining contextual information with knowledge from structured lexical resources. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-11T21:25:56Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)
• Grounded and Controllable Image Completion by Incorporating Lexical Semantics [111.47374576372813]
  Lexical Semantic Image Completion (LSIC) may have potential applications in art, design, and heritage conservation. We advocate generating results faithful to both visual and lexical semantic context. One major challenge for LSIC comes from modeling and aligning the structure of visual-semantic context.
  arXiv  Detail & Related papers  (2020-02-29T16:54:21Z)
• Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation [50.01708049531156]
  We focus on a new practical task, document-scale text content manipulation, which is the opposite of text style transfer. In detail, the input is a set of structured records and a reference text for describing another recordset. The output is a summary that accurately describes the partial content in the source recordset with the same writing style of the reference.
  arXiv  Detail & Related papers  (2020-02-24T12:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.