LLMs Need Encoders for Semantic IDs Too

• URL: http://arxiv.org/abs/2606.00324v1
• Date: Fri, 29 May 2026 20:01:06 GMT
• Title: LLMs Need Encoders for Semantic IDs Too
• Abstract summary: PrefixMem is a lightweight SID encoder based on prefix n-gram memory tables.<n>We show that PrefixMem improves deepest-level SID accuracy by up to 46% relative and full-SID retrieval recall by up to 22% relative at matched training compute.
• Score: 6.952078655385385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal LLMs use dedicated encoders to bridge non-language modalities (vision encoders for images, depth models for audio codec tokens) because raw token embeddings alone cannot capture modality-specific structure. We argue that Semantic IDs (SIDs), the hierarchical codes used in generative recommendation, constitute another such modality: a SID level token's meaning depends on its prefix context, yet current systems simply add SID tokens to the vocabulary and rely on training to learn these context-dependent meanings from scratch. We propose PrefixMem, a lightweight SID encoder based on prefix n-gram memory tables that provides the LLM with structured, prefix-conditioned representations at SID token positions. Like vision encoders in multimodal LLMs, PrefixMem can be pre-trained independently and then attached to any LLM for joint training. We evaluate on large-scale data from Pinterest across multiple LLM families and show that PrefixMem improves deepest-level SID accuracy by up to 46% relative and full-SID retrieval recall by up to 22% relative at matched training compute. The encoder's benefit concentrates on hard examples where greedy decoding fails, with up to 77% relative accuracy gains, confirming that SID tokens benefit from a dedicated encoder just as other non-language modalities do.

Related papers

• HF-SID: High-Fidelity Semantic IDs for Generative Retrieval in Location-Based Services [6.767079940213956]
  Generative retrieval has attracted increasing attention in Location-Based Services (LBS), where each Point-of-Interest (POI) is represented as a Semantic ID (SID)<n>We propose HF-SID, which restores geographic, numerical, and structural fidelity at the representation stage, before any information is committed to a discrete code.<n>Because these mechanisms enrich the representation rather than lengthen the representation, HF-SID uses a 3-token identifier SID at no extra decoding cost.
  arXiv  Detail & Related papers  (2026-08-31T09:05:09Z)
• Interpretable Representation via LLM-Driven Generative Disentanglement for Local-Life Service Recommendation [28.83516611687355]
  We propose Interpretable Representation via LLM-Driven Generative Disentanglement for Local-Life Service Recommendation (LGRID)<n>LGRID introduces a generative disentanglement paradigm through an Encode -> Disentangle -> Align -> Quantize pipeline.<n> Experiments on Kuaishou and Foursquare show that LGRID consistently outperforms strong SID baselines.
  arXiv  Detail & Related papers  (2026-07-30T09:52:18Z)
• VaLiDRec: Variable-Length LLM-Aligned Semantic IDs for Generative Recommendation [39.59701442467294]
  VaLiDRec is a generative recommendation framework based on variable-length, LLM-aligned semantic identifiers.<n>To model user preferences, VaLiDRec incorporates graph-aware soft prompts and reformulates recommendation as token-set prediction.<n>Experiments on four real-world datasets show that VaLiDRec consistently outperforms strong and sequential generative recommendation baselines.
  arXiv  Detail & Related papers  (2026-07-28T02:29:10Z)
• Diffusion Large Language Models for Visual Speech Recognition [56.81307584718608]
  We propose the first Diffusion Large Language Model (DLLM)-based Visual Speech Recognition (VSR) framework.<n>DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones.<n>We develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses.<n>The proposed method achieves a state-of-the-art WER of 19.5% on LRS3 using only its labeled training data.
  arXiv  Detail & Related papers  (2026-05-27T13:22:08Z)
• Fine-grained Semantics Integration for Large Language Model-based Recommendation [35.75224379727093]
  We propose TS-Rec, which can integrate Token-level Semantics into LLM-based Recommenders.<n>Extensive experiments on two real-world benchmarks demonstrate that TS-Rec consistently outperforms traditional and generative baselines.
  arXiv  Detail & Related papers  (2026-02-26T05:17:24Z)
• Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders [37.16464474575651]
  We show that pointwise generative ranking with multi-token docIDs is superior to that of dual encoders.<n>We propose SToICaL - a Simple Token-Item Calibrated Loss - which can incorporate rank-aware supervision at both the item and token levels.
  arXiv  Detail & Related papers  (2026-01-09T07:16:28Z)
• Decoding in Latent Spaces for Efficient Inference in LLM-based Recommendation [75.72196852363116]
  Light Latent-space Decoding (L2D) is an effective and efficient latent-space decoding method.<n>L2D is more than 10x faster than language-space decoding while maintaining or enhancing performance.
  arXiv  Detail & Related papers  (2025-09-15T02:30:35Z)
• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models [92.37117312251755]
  We propose a progressive pruning framework, namely Multi-Encoder collaboraTivE tOken pRuning (METEOR)<n>For multi-vision encoding, we discard redundant tokens within each encoder via a rank guided collaborative token assignment strategy.<n>For multi-vision fusion, we combine the visual features from different encoders while reducing cross-encoder redundancy with cooperative pruning.
  arXiv  Detail & Related papers  (2025-07-28T13:50:53Z)
• Self-Explained Keywords Empower Large Language Models for Code Generation [5.236633572296712]
  Large language models (LLMs) have achieved impressive performance in code generation. Sek(textbfSelf-textbfExplained textbfKeywords) extracts and explains the key terms in the problem description with the LLM itself.
  arXiv  Detail & Related papers  (2024-10-21T12:52:03Z)
• CodecLM: Aligning Language Models with Tailored Synthetic Data [51.59223474427153]
  We introduce CodecLM, a framework for adaptively generating high-quality synthetic data for instruction-following abilities. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples.
  arXiv  Detail & Related papers  (2024-04-08T21:15:36Z)
• In-context Autoencoder for Context Compression in a Large Language Model [70.7621953091318]
  We propose the In-context Autoencoder (ICAE) to compress a long context into short compact memory slots. ICAE is first pretrained using both autoencoding and language modeling objectives on massive text data.
  arXiv  Detail & Related papers  (2023-07-13T17:59:21Z)
• Inference with Reference: Lossless Acceleration of Large Language Models [97.04200102556551]
  LLMA is an accelerator to speed up Large Language Model (LLM) inference with references. It is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios.
  arXiv  Detail & Related papers  (2023-04-10T09:55:14Z)
• Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model [57.92200214957124]
  External language models (LMs) are used to improve the recognition performance of end-to-end (E2E) automatic speech recognition (ASR) systems. We propose a novel decoding algorithm where a word-level lattice is constructed on-the-fly to consider all possible word sequences. Our method consistently outperforms subword-level LMs, including N-gram LM and neural network LM.
  arXiv  Detail & Related papers  (2022-01-06T10:04:56Z)
• Exploring the Use of an Unsupervised Autoregressive Model as a Shared Encoder for Text-Dependent Speaker Verification [22.894402178709136]
  We propose a novel way of addressing text-dependent automatic speaker verification (TD-ASV) by using a shared-encoder with task-specific decoders. We show that the proposed approach can leverage from large, unlabeled, data-rich domains, and learn speech patterns independent of downstream tasks.
  arXiv  Detail & Related papers  (2020-08-08T22:47:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.