Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation

• URL: http://arxiv.org/abs/2608.07816v1
• Date: Fri, 07 Aug 2026 23:33:08 GMT
• Title: Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation
• Abstract summary: In generative recommendation systems, items are often represented through semantic IDs (SIDs)<n>Standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry.<n>We propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space.
• Score: 33.094341397930776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories. In these systems, items are often represented through semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally, SIDs imbue item token representations with semantic priors, thereby improving model generalization. However, standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry and forcing the LLM to relearn token relationships from interaction data. To demonstrate the consequences of this design, we first show that training from this initialization tends to organize SID embeddings around item popularity rather than semantics. We further show that, despite partially reducing the reliance on popularity and improving cold item performance, the computationally expensive process of continual pretraining (CPT) fails to reliably recover the original semantic geometry. To address these findings, we propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space. Requiring only a few lines of code and no additional training or inference overhead, this drop-in approach improves pure-SFT Recall@5 by up to 16%, reaches peak performance with up to 40% fewer SFT steps, and improves cold-item Recall@5 by up to 60%. Moreover, on datasets that benefit from additional CPT, centroid initialization reaches comparable performance while requiring half as many CPT epochs. Together, our findings show that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.

Related papers

• Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation [39.5807784132569]
  Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic tokens and predict the next item by autoregressively generating this token sequence.<n>This paradigm enables a unified generation interface for item IDs, histories, and item text, but it also creates a structured optimization bottleneck during reward-based post-training.<n>We propose HCGRec, a semantic-ID generative recommendation framework that recovers learning signal for such hard training instances.<n>HCGRec substantially improves over supervised fine-tuning and vanilla reward-based post-training, while reducing zero-advantage training samples from over 70% to below 20%.
  arXiv  Detail & Related papers  (2026-08-12T12:13:08Z)
• VaLiDRec: Variable-Length LLM-Aligned Semantic IDs for Generative Recommendation [39.59701442467294]
  VaLiDRec is a generative recommendation framework based on variable-length, LLM-aligned semantic identifiers.<n>To model user preferences, VaLiDRec incorporates graph-aware soft prompts and reformulates recommendation as token-set prediction.<n>Experiments on four real-world datasets show that VaLiDRec consistently outperforms strong and sequential generative recommendation baselines.
  arXiv  Detail & Related papers  (2026-07-28T02:29:10Z)
• DREAM: Dynamic Refinement of Early Assignment Mappings [7.781421530363403]
  Current methods assign each item a single static identifier through offline tokenization before sufficient user feedback is observed.<n>We identify this one-shot commitment as the fundamental cold-start bottleneck in SID-based generative recommendation.<n>We propose DREAM, a three-stage framework that resolves this flaw through progressive refinement.
  arXiv  Detail & Related papers  (2026-06-05T06:21:01Z)
• Fine-grained Semantics Integration for Large Language Model-based Recommendation [35.75224379727093]
  We propose TS-Rec, which can integrate Token-level Semantics into LLM-based Recommenders.<n>Extensive experiments on two real-world benchmarks demonstrate that TS-Rec consistently outperforms traditional and generative baselines.
  arXiv  Detail & Related papers  (2026-02-26T05:17:24Z)
• GLASS: A Generative Recommender for Long-sequence Modeling via SID-Tier and Semantic Search [51.44490997013772]
  GLASS is a novel framework that integrates long-term user interests into the generative process via SID-Tier and Semantic Search.<n>We show that GLASS outperforms state-of-the-art baselines in experiments on two large-scale real-world datasets.
  arXiv  Detail & Related papers  (2026-02-05T13:48:33Z)
• Rethinking Generative Recommender Tokenizer: Recsys-Native Encoding and Semantic Quantization Beyond LLMs [17.944727019161878]
  ReSID is a principled, SID framework that recommend learning from the perspective of information preservation and sequential predictability.<n>It consistently outperforms strong sequential and SID-based generative baselines by an average of over 10%, while reducing tokenization cost by up to 122x.
  arXiv  Detail & Related papers  (2026-02-02T17:00:04Z)
• Differentiable Semantic ID for Generative Recommendation [65.83703273297492]
  Generative recommendation provides a novel paradigm in which each item is represented by a discrete semantic ID (SID) learned from rich content.<n>In practice, SIDs are typically optimized only for content reconstruction rather than recommendation accuracy.<n>A natural approach is to make semantic indexing differentiable so that recommendation gradients can directly influence SID learning.<n>We propose DIGER, a first step toward effective differentiable semantic IDs for generative recommendation.
  arXiv  Detail & Related papers  (2026-01-27T15:34:11Z)
• Unleashing the Native Recommendation Potential: LLM-Based Generative Recommendation via Structured Term Identifiers [51.64398574262054]
  This paper introduces Term IDs (TIDs), defined as a set of semantically rich and standardized textual keywords, to serve as robust item identifiers.<n>We propose GRLM, a novel framework centered on TIDs, to convert item's metadata into standardized TIDs and utilize Integrative Instruction Fine-tuning to collaboratively optimize term internalization and sequential recommendation.
  arXiv  Detail & Related papers  (2026-01-11T07:53:20Z)
• RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation [78.01030342481246]
  RecBase is a domain-agnostic foundational model pretrained with a recommendation-oriented objective.<n>We introduce a unified item tokenizer that encodes items into hierarchical concept identifiers.<n>Our model matches or surpasses the performance of LLM baselines up to 7B parameters in zero-shot and cross-domain recommendation tasks.
  arXiv  Detail & Related papers  (2025-09-03T08:33:43Z)
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization [50.27950279695363]
  Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages.<n>A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.<n>We propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding.
  arXiv  Detail & Related papers  (2025-04-21T19:40:32Z)
• Enhancing Item Tokenization for Generative Recommendation through Self-Improvement [67.94240423434944]
  Generative recommendation systems are driven by large language models (LLMs)<n>Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens.<n>We propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process.
  arXiv  Detail & Related papers  (2024-12-22T21:56:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.