MultiHashFormer: Hash-based Generative Language Models

• URL: http://arxiv.org/abs/2606.28057v1
• Date: Fri, 26 Jun 2026 13:03:29 GMT
• Title: MultiHashFormer: Hash-based Generative Language Models
• Abstract summary: We propose MultiHashFormer, a new framework that allows hash-based autoregression.<n>A Hash Decoder generates the hash signature of the next token, which is then mapped back to text.<n>We show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.
• Score: 32.20759503542573
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.

Related papers

• Beyond Post-Quantization: Native Hash Learning with a Dedicated HASH Token [20.81211439061764]
  HashViT is a Vision Transformer framework for native hash token learning.<n>It is optimized with a unified objective that combines learnable semantic center supervision, class-token similarity distillation, and quantization regularization.<n>Experiments on three widely used benchmarks demonstrate that HashViT achieves state-of-the-art or highly competitive retrieval performance.
  arXiv  Detail & Related papers  (2026-07-03T13:46:18Z)
• SpikeHash: Learning Binary Codes with Spiking Neural Networks for Cross-Modal Hashing Retrieval [10.799183419959055]
  Cross-modal hashing retrieval encodes heterogeneous data into compact binary codes for efficient Hamming-space search.<n>We propose SpikeHash, a unified spiking framework that formulates cross-modal hashing as spike-state evolution, directional spike interaction, and competitive spike readout.
  arXiv  Detail & Related papers  (2026-05-30T14:14:59Z)
• Sampling from Your Language Model One Byte at a Time [82.71473348639489]
  Tokenization can introduce distortion into the model's generations, known as the Prompt Boundary Problem (PBP)<n>We present an inference-time method to convert any autore LM with a BPE tokenizer into a character-level or byte-level LM.<n>Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers.
  arXiv  Detail & Related papers  (2025-06-17T02:37:04Z)
• Nested Hash Layer: A Plug-and-play Module for Multiple-length Hash Code Learning [61.095479786194836]
  Nested Hash Layer (NHL) is a plug-and-play module for deep supervised hashing models.<n>NHL generates hash codes of multiple lengths simultaneously in a nested structure.<n>NHL achieves an overall training speed improvement of approximately 5 to 8 times across various deep supervised hashing models.
  arXiv  Detail & Related papers  (2024-12-12T04:13:09Z)
• ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery [128.30514851911218]
  ConceptHash is a novel method that achieves sub-code level interpretability. In ConceptHash, each sub-code corresponds to a human-understandable concept, such as an object part. We incorporate language guidance to ensure that the learned hash codes are distinguishable within fine-grained object classes.
  arXiv  Detail & Related papers  (2024-06-12T17:49:26Z)
• Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration [54.897493351694195]
  We propose a novel parallel decoding approach, namely textithidden transfer, which decodes multiple successive tokens simultaneously in a single forward pass. In terms of acceleration metrics, we outperform all the single-model acceleration techniques, including Medusa and Self-Speculative decoding.
  arXiv  Detail & Related papers  (2024-04-18T09:17:06Z)
• Object Recognition as Next Token Prediction [99.40793702627396]
  We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels.
  arXiv  Detail & Related papers  (2023-12-04T18:58:40Z)
• HashFormers: Towards Vocabulary-independent Pre-trained Transformers [30.699644290131044]
  Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. We propose HashFormers, a new family of vocabulary-independent pre-trained transformers.
  arXiv  Detail & Related papers  (2022-10-14T15:39:34Z)
• Additive Feature Hashing [0.0]
  We show that additive feature hashing can be performed directly by adding the hash values and converting them into high-dimensional numerical vectors. We show that the performance of additive feature hashing is similar to the hashing trick, and we illustrate the results numerically using synthetic, language recognition, and SMS spam detection data.
  arXiv  Detail & Related papers  (2021-02-07T23:15:04Z)
• Bio-Inspired Hashing for Unsupervised Similarity Search [12.78093617645299]
  We propose a novel hashing algorithm BioHash that produces sparse high dimensional hash codes in a data-driven manner. Our work provides evidence for the proposal that LSH might be a computational reason for the abundance of sparse expansive motifs in a variety of biological systems.
  arXiv  Detail & Related papers  (2020-01-14T17:04:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.