Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models

• URL: http://arxiv.org/abs/2605.29459v1
• Date: Thu, 28 May 2026 06:53:18 GMT
• Title: Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
• Abstract summary: Large language models route every input through a learned embedding table of shape.<n>We introduce Kronecker Embeddings, a deterministic byte-level character-position factorization.<n>It eliminates 91--94% of input-side trainable parameters at frontier scale.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models route every input through a learned embedding table of shape |V| x d_model, consuming hundreds of millions to billions of trainable parameters at frontier scale. We introduce Kronecker Embeddings, a deterministic byte-level character-position factorization that replaces this table with a fixed encoder and a single learned projection, compatible with standard BPE tokenizers, eliminating 91--94% of input-side trainable parameters at frontier scale. We provide five contributions. First, a cross-model probe across six LMs (135M-671B parameters) shows trained input embeddings cluster typographic variants of the probe word far more than morphological relatives; Kronecker escapes this clustering at the embedding layer. Second, a controlled three-seed comparison on nanoGPT GPT-2 124M over 2.5B tokens of FineWeb-Edu shows Kronecker reaching 2.5 +- 0.2% lower validation loss than the BPE-tied baseline (gap 0.083 +- 0.007 nats, ~9% lower perplexity), needing ~1.43x fewer steps to reach BPE's converged loss. Third, a spelling-robustness probe over 110 clean/typo pairs shows Kronecker preserves the top-1 prediction on 55.5% of pairs vs. 47.3% for BPE (+8.2 pp) and lowers KL by 7.6%, winning or tying in 10 of 11 categories; a generation probe shows Kronecker echoes byte-novel strings and typos through generation where BPE forgets them. Fourth, BPE embedding norm drifts during training while Kronecker projection norm stays near 1.0, consistent with a stable representational target. Fifth, an on-the-fly runtime variant reconstructs embeddings from a 4.5 MB byte buffer rather than a 2.15 GB table at vocabulary 131,072, with 0.01--0.24% step-time overhead. Byte-level locality has a tradeoff: byte-similar but semantically distant pairs (compute/commute, nation/notion) cluster together, shifting disambiguation to early attention layers.

Related papers

• Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models [67.72077328700088]
  We present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously.<n>We find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau.<n>By operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-kruncation during logit dumping.
  arXiv  Detail & Related papers  (2026-09-11T00:17:22Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result [0.0]
  We show how one language model can operate at several vocabulary sizes.<n>A control token indicates the active size, and be deployed at any trained size by slicing its embedding and output head.<n>A control with neither cap token nor output restriction is equally robust, attributing this benefit to multi-granularity training rather than conditioning.
  arXiv  Detail & Related papers  (2026-08-28T10:13:32Z)
• Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation [0.3525984766628294]
  On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy.<n>We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points.
  arXiv  Detail & Related papers  (2026-08-27T21:40:19Z)
• CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield [1.2203679037551938]
  We study three techniques for training compute-efficient language models.<n>SGT concentrates supervision on the 15% of output tokens that carry semantic payload.<n> depth compression with recurrent recovery.<n>We validate these techniques on CHERRY-1.8B, a Korean foundation model.
  arXiv  Detail & Related papers  (2026-06-30T15:14:38Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data [0.0]
  We evaluate seven configurations combining three inference methods with two training pipelines on the PAMAP2 dataset.<n>We find that the modality gap is a training-time phenomenon governed by the objective.<n>We also demonstrate that overall accuracy is a misleading primary metric when test-set class distributions are imbalanced.
  arXiv  Detail & Related papers  (2026-06-09T12:39:41Z)
• How Much Capacity Does EEG Denoising Need? Ultra-Compact Networks reveal Benchmark Saturation and Metric-Utility Gap [0.0]
  Deep learning EEG denoising architectures have scaled from tens of thousands to tens of millions of parameters, yet no prior study has isolated model capacity as the experimental variable.<n>We address both gaps by fixing architecture, loss, data split, and training recipe while sweeping only channel width from 1.05K to 40.26K parameters in a minimal depthwiseseparable convolutional U-Net.
  arXiv  Detail & Related papers  (2026-06-07T12:17:25Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• A Modelling and Evaluation Framework for EuroCrops-Driven Sentinel-2 Crop Segmentation [78.66324246922831]
  This work presents a pipeline for generating semantic-segmentation-ready agricultural datasets from Sentinel-2 imagery and EuroCrops parcel-level annotations.<n>The dataset contains 67,337 patches from five European countries and uses a reduced taxonomy of ten crop classes plus background.<n>A four-level U-Net with Group Normalization was trained using 10 Sentinel-2 spectral bands and a composite loss combining class-weighted cross-entropy and Dice loss.
  arXiv  Detail & Related papers  (2026-05-30T11:20:29Z)
• ATLAS: All-round Testing of Long-context Abilities across Scales [42.83200405769639]
  ATLAS is a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling.<n> ATLAS contributes three methodological principles: (i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles.
  arXiv  Detail & Related papers  (2026-05-27T07:33:47Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification [4.738949927143789]
  We show that homogeneous labels--even semantically valid ones--collapse accuracy to =12% across six models.<n>The trigger is label-slot content: the model treats tokens occupying the label position as an exhaustive answer vocabulary.
  arXiv  Detail & Related papers  (2026-05-08T10:20:39Z)
• Compute Optimal Tokenization [79.3815358070537]
  We investigate how the information granularity of tokens, controlled by the compression rate, affects scaling trends.<n>We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate.<n>Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived.
  arXiv  Detail & Related papers  (2026-05-02T01:53:22Z)
• Lightweight Distillation of SAM 3 and DINOv3 for Edge-Deployable Individual-Level Livestock Monitoring and Longitudinal Visual Analytics [0.3609538870261841]
  Foundation-model pipelines for individual-level livestock monitoring have raised the accuracy ceiling of precision livestock farming.<n>But their GPU memory budgets exceed the envelope of commodity edge accelerators.<n>To close this gap, the backbone of SAM 3 is distilled into a 40.66M-scale multi-scale student.<n>The pipeline achieves 97.34% top-1 accuracy with 91.67% macro-F1 on nine-class pig behaviour classification.
  arXiv  Detail & Related papers  (2026-04-29T19:25:22Z)
• Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT [0.0]
  Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive.<n>We study Matrix Product Operator decomposition as a principled compression method for transformers.<n>MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi.
  arXiv  Detail & Related papers  (2026-03-30T14:57:47Z)
• PLUME: Building a Network-Native Foundation Model for Wireless Traces via Protocol-Aware Tokenization [0.9558392439655014]
  Plume is a compact foundation model for 802.11 traces that learns from structured PDML dissections.<n>A protocol-aware tokenizer splits along the dissector field tree, emits gap tokens for timing, and normalizes identifiers, yielding 6.2x shorter sequences than BPE.
  arXiv  Detail & Related papers  (2026-03-13T23:14:43Z)
• Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging [0.0]
  Significance-Gain BPE is a drop-in alternative merge criterion that measures cohesion via a z-statistic under an independence null model.<n>It reduces validation and test perplexity by 13% and 12%, respectively, and improves validation and test BPC by about 0.9 to 1.0%.
  arXiv  Detail & Related papers  (2026-02-26T20:29:18Z)
• Near, far: Patch-ordering enhances vision foundation models' scene understanding [35.768260232640756]
  NeCo: Patch Neighbor Consistency enforces patch-level nearest neighbor consistency across a student and teacher model.<n>Our method leverages differentiable sorting applied on top of pretrained representations, such as DINOv2-registers to bootstrap the learning signal.<n>This dense post-pretraining leads to superior performance across various models and datasets, despite requiring only 19 hours on a single GPU.
  arXiv  Detail & Related papers  (2024-08-20T17:58:59Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.