Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting

• URL: http://arxiv.org/abs/2608.29930v1
• Date: Sun, 30 Aug 2026 17:58:49 GMT
• Title: Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting
• Abstract summary: Black-box model attribution is increasingly relevant when large language models are served through relay and reseller APIs.<n>A tempting low-cost signal is the prompt-token count returned by an OpenAI-compatible endpoint.<n>Yet the validity of this signal for broader emphmodel-family attribution has received little direct holdout testing.<n>We conduct a frozen-threshold study over 24 labeled endpoint pairs, split evenly into a development set and an untouched holdout set.
• Score: 4.093009863847175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Black-box model attribution is increasingly relevant when large language models (LLMs) are served through relay and reseller APIs. A tempting low-cost signal is the prompt-token count returned by an OpenAI-compatible endpoint: two models that share a tokenizer and chat template may produce the same count sequence up to a fixed offset. Yet the validity of this signal for broader \emph{model-family} attribution has received little direct holdout testing. We conduct a frozen-threshold study over 24 labeled endpoint pairs, split evenly into a development set and an untouched holdout set, with three temporal repeats and 30 controlled texts per pair. We introduce a validity-gated result contract that distinguishes an observed dissimilarity from an uninformative measurement caused by missing usage data, rate limits, or endpoint policy. The resulting shift-invariant exact-match score perfectly separates the 12 development pairs, yielding a frozen threshold of 0.725. On holdout, however, only 6 of 12 pairs are eligible under the pre-specified three-repeat rule. Among eligible pairs, balanced accuracy is 0.75, sensitivity is 0.50 (95\% Wilson interval 0.15--0.85), and specificity is 1.00 (0.342--1.00). Two same-family pairs---Qwen 3.8 and DeepSeek V4 variants---fall below the frozen threshold. Across 4,320 formal API calls, every log is replayable, while holdout contains 189 non-200 responses and 157 successful responses without prompt-token usage. The study therefore validates token-count consistency as a fingerprint of a shared \emph{tokenization stack}, but rejects its use as a standalone necessary test for model-family lineage.

Related papers

• Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format [0.0]
  We compare probabilities obtained by scoring answer tokens with predictions generated after a written rationale.<n>Across 13 model-domain cells covering four retail tasks, the scored readout ranks outcomes more accurately in 12 of 13 cells.
  arXiv  Detail & Related papers  (2026-09-09T08:37:46Z)
• Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation [41.01488608247856]
  Outcome verifiers score completed reasoning traces but do not assign credit to intermediate tokens.<n> Privileged self-distillation attempts to fill this gap by rescoring a model's own rollout with training-only information.<n>We separate three questions: whether the score tracks better actions, whether feedback construction changes what is compared, and what behavior the training loss reinforces.
  arXiv  Detail & Related papers  (2026-08-10T08:18:48Z)
• TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance [8.657685524975246]
  We introduce a training-free fingerprint based on the top-$k$ vocabulary projections of late hidden states elicited by 250 fixed knowledge probes.<n>We evaluate the method on 32 open-weight models from nine families (0.6B--32B) with documented relationships.
  arXiv  Detail & Related papers  (2026-08-08T13:56:39Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions [0.0]
  API aggregators, resellers, and inference providers consume large language models (LLMs)<n>Existing identification techniques require long generated texts, token-level log-probabilities, adversarially crafted prompts, or the model owner's cooperation.<n>We define a behavioral fingerprint of an LLM as the empirical distribution of its answers to trivial one-word prompts.
  arXiv  Detail & Related papers  (2026-07-11T10:34:17Z)
• Metric-Dependent Annotation Saturation for Learning from Label Distributions [0.0]
  We fine-tune NLI models on label distributions subsampled from ChaosNLI.<n>In our 3-class NLI setting, entropy correlation requires N 20-50 annotators to converge.<n>Soft labels carry item-specific signal that label smoothing cannot replicate.
  arXiv  Detail & Related papers  (2026-05-28T11:46:51Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference [0.0]
  VERDI (VERification-Decomposed Inference Inference) is a method that extracts confidence from a structured judge.<n> VERDI decomposes each evaluation into sub-checks and derives three structural signals.<n>On three public benchmarks, VERDI achieves AUROC 0.72-0.91 on GPT-4.1-mini and 0.66-0.80 on GPT-5.4-mini.
  arXiv  Detail & Related papers  (2026-05-11T23:39:19Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive [0.0]
  CPU-only scoring primitive runs at 2.6 microseconds per token.<n> token rank-frequency distributions converge to the same two- parameter Mandelbrot ranking distribution.<n>We derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs.
  arXiv  Detail & Related papers  (2026-04-28T13:35:31Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Model Equality Testing: Which Model Is This API Serving? [59.005869726179455]
  API providers may quantize, watermark, or finetune the underlying model, changing the output distribution.<n>We formalize detecting such distortions by Model Equality Testing, a two-sample testing problem.<n>A test built on a simple string kernel achieves a median of 77.4% power against a range of distortions.
  arXiv  Detail & Related papers  (2024-10-26T18:34:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.