When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification

• URL: http://arxiv.org/abs/2605.07120v1
• Date: Fri, 08 May 2026 01:50:52 GMT
• Title: When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification
• Abstract summary: We study the fixed-label classification version of this problem, where train and test examples share latent templates but may use disjoint vocabularies.<n>Unlike next-token prediction, the model need not emit unseen symbols; it must learn a decision rule invariant to symbol renaming.<n>We encode accidental token overlaps by a colored collision graph and prove high-probability margin-transfer guarantees for fresh-symbol classification.
• Score: 0.08594140167290097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Template tasks have emerged as a clean testbed for asking whether transformers reason with abstract symbols rather than concrete token names. We study the fixed-label classification version of this problem, where train and test examples share latent templates but may use disjoint vocabularies. Unlike next-token prediction, the model need not emit unseen symbols; it must learn a decision rule invariant to symbol renaming. We analyze regularized kernel logistic classification in the transformer-kernel regime. Our main result decomposes the learned predictor into an ideal template-level classifier and a finite-sample perturbation caused by accidental token overlaps in the training data. We encode these overlaps by a colored collision graph and prove high-probability margin-transfer guarantees for fresh-symbol classification. This perspective extends template-based analyses to logistic classification and refines scalar diversity conditions: vocabulary size controls the average rate of collisions, but collision geometry controls whether the ideal classification margin is preserved. More broadly, the same perturbation framework applies to abstraction-augmented inputs, yielding a general margin-versus-collision criterion for identifying when prompting strategies improve fresh-symbol generalization. Synthetic template experiments illustrate the predicted roles of regularization, sample size, and transformer-kernel structure.

Related papers

• Causal dictionary learning reveals and validates transcription-factor binding features in genomic language models [5.237172334460829]
  We introduce a framework that combines sparse dictionary learning with causal intervention to extract, validate, and causally test interpretable features in genomic foundation models.<n>We show that the naive validation of such features against position weight matrices is severely confounded by GC composition and repetitive elements.<n>We establish that specific features are emphcausally used to represent cell-type-specific TF binding, not merely motif presence.
  arXiv  Detail & Related papers  (2026-07-21T22:54:46Z)
• Emergent retokenization symmetry in large language models: phenomenology and applications [3.055499449876386]
  We replace a prompt's canonical tokenization with an alternative segmentation while preserving its bytes exactly.<n>We use retokenization to study sensitivity and robustness to semantically identical input representations across pretraining and post-training.<n>We find that while this retokenization sampling strategy can hurt performance on easy problems, it can also recover solutions that conventional sampling does not find.
  arXiv  Detail & Related papers  (2026-06-14T00:38:50Z)
• Weakly Supervised Segmentation as Semantic-Based Regularization [5.8837215684098645]
  Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations.<n>We take a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models.<n>We show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation.
  arXiv  Detail & Related papers  (2026-05-13T15:33:27Z)
• Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens [4.0704009036918025]
  Reasoning in Large Language Models incurs significant inference-time compute.<n>We show that reasoning tokens split into two functional types: low-entropy textitstructural tokens (recurring phrases that scaffold the reasoning process) and higher-entropy textitorganic tokens (problem-specific content that drives toward a solution)<n>We derive textitsupertokens that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-04-29T07:06:43Z)
• Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification [49.09505771145326]
  We propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels. Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-04-26T06:00:27Z)
• Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection [24.11900895337062]
  Unsupervised image Anomaly Detection (UAD) aims to learn robust and discriminative representations of normal samples. This paper focuses on building a unified framework for multiple classes.
  arXiv  Detail & Related papers  (2023-10-22T08:20:33Z)
• Variational Classification [51.2541371924591]
  We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency. We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
  arXiv  Detail & Related papers  (2023-05-17T17:47:19Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• Parametric Classification for Generalized Category Discovery: A Baseline Study [70.73212959385387]
  Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. We investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers.
  arXiv  Detail & Related papers  (2022-11-21T18:47:11Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.