C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search

• URL: http://arxiv.org/abs/2510.19221v1
• Date: Wed, 22 Oct 2025 04:05:38 GMT
• Title: C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search
• Abstract summary: We propose C2T-ID, which builds semantic numerical docid via hierarchical clustering.<n>C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines.
• Score: 73.61009656398384
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing document identifiers (docids) that carry rich semantic information while maintaining tractable search spaces is a important challenge in generative retrieval (GR). Popular codebook methods address this by building a hierarchical semantic tree and constraining generation to its child nodes, yet their numeric identifiers cannot leverage the large language model's pretrained natural language understanding. Conversely, using text as docid provides more semantic expressivity but inflates the decoding space, making the system brittle to early-step errors. To resolve this trade-off, we propose C2T-ID: (i) first construct semantic numerical docid via hierarchical clustering; (ii) then extract high-frequency metadata keywords and iteratively replace each numeric label with its cluster's top-K keywords; and (iii) an optional two-level semantic smoothing step further enhances the fluency of C2T-ID. Experiments on Natural Questions and Taobao's product search demonstrate that C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines, demonstrating its effectiveness in balancing semantic expressiveness with search space constraints.

Related papers

• Exploring Bottom-Up Clustering for Creating Semantic IDs [6.910490819628779]
  We present an algorithm for generating Semantic IDs that ensure the identifiers are both unique and preserve the structure of the original embedding.<n>Key to our work is the use of bottom-up clustering to preserve local structure in the embedding space, improving the clustering quality of the resulting Semantic IDs.
  arXiv  Detail & Related papers  (2026-09-08T06:32:57Z)
• CodeHID: Learning an Addressable Hierarchical Code Index for Generative Code Retrieval [16.418964135916433]
  We propose CodeHID, a generative code retrieval framework that reformulates the code retrieval task from flat candidate matching to coarse-to-fine semantic address generation.<n>CodeHID relies on two core components. First, Pseudo-Neighbor Guided DocID Learning constructs a globally static hierarchical index by applying multi-level residual quantization and $k$-nearest-neighbor pseudo-supervision.<n>Second, Dual-Phase DocID Generation Guidance reliably navigates this fixed index by combining training-side ranking enhancements, using hard negatives and rank distillation, with inference-side candidate constraints and prefix-aware decoding.
  arXiv  Detail & Related papers  (2026-08-25T05:35:17Z)
• Rethinking Item Tokenization in Generative Recommenders: From Fixed Atoms to Semantic Subwords [50.621722853036005]
  In generative recommender systems, items are typically tokenized into fixed-length semantic ID sequences for autoregressive next-item prediction.<n>For user-context modeling, this fine-grained representation triggers Intra-item Attention Overload.<n>We propose Semantic Subword Tokenization (SST), which represents historical items as variable-length semantic subwords.
  arXiv  Detail & Related papers  (2026-08-24T02:42:51Z)
• TopoGR: Revealing and Preserving Latent Structure of Semantic ID in Generative Recommendation [25.345503799250476]
  TopoGR is a topology-preserving generative recommendation framework based on Bit-decomposable Semantic ID( Binary SID)<n>We show that TopoGR consistently outperforms existing state-of-the-art baselines in recommendation performance.
  arXiv  Detail & Related papers  (2026-07-28T02:45:33Z)
• Beyond Fixed Depths and Widths: Optimizing Textual Decoding Tries in LLM-based Generative Recommendation [59.20704286413859]
  We propose a novel framework that co-designs textual term IDs and their underlying decoding trie.<n> BONSAI achieves up to a 21.6% relative improvement over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-07-18T04:31:02Z)
• Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval [28.366331215978445]
  We propose purely semantic indexing to generate unique, semantic-preserving IDs without appending non-semantic tokens.<n>We enable unique ID assignment by relaxing the strict nearest-centroid selection and introduce two model-agnostic algorithms.
  arXiv  Detail & Related papers  (2025-09-19T21:59:55Z)
• Order-agnostic Identifier for Large Language Model-based Generative Recommendation [94.37662915542603]
  Items are assigned identifiers for Large Language Models (LLMs) to encode user history and generate the next item.<n>Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings.<n>We propose SETRec, which leverages semantic tokenizers to obtain order-agnostic multi-dimensional tokens.
  arXiv  Detail & Related papers  (2025-02-15T15:25:38Z)
• LIST: Learning to Index Spatio-Textual Data for Embedding based Spatial Keyword Queries [53.843367588870585]
  List K-kNN spatial keyword queries (TkQs) return a list of objects based on a ranking function that considers both spatial and textual relevance. There are two key challenges in building an effective and efficient index, i.e., the absence of high-quality labels and the unbalanced results. We develop a novel pseudolabel generation technique to address the two challenges.
  arXiv  Detail & Related papers  (2024-03-12T05:32:33Z)
• Summarization-Based Document IDs for Generative Retrieval with Language Models [65.11811787587403]
  We introduce summarization-based document IDs, in which each document's ID is composed of an extractive summary or abstractive keyphrases. We show that using ACID improves top-10 and top-20 recall by 15.6% and 14.4% (relative) respectively. We also observed that extractive IDs outperformed abstractive IDs on Wikipedia articles in NQ but not the snippets in MSMARCO.
  arXiv  Detail & Related papers  (2023-11-14T23:28:36Z)
• Language Models As Semantic Indexers [78.83425357657026]
  We introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval.
  arXiv  Detail & Related papers  (2023-10-11T18:56:15Z)
• Multiview Identifiers Enhanced Generative Retrieval [78.38443356800848]
  generative retrieval generates identifier strings of passages as the retrieval target. We propose a new type of identifier, synthetic identifiers, that are generated based on the content of a passage. Our proposed approach performs the best in generative retrieval, demonstrating its effectiveness and robustness.
  arXiv  Detail & Related papers  (2023-05-26T06:50:21Z)
• CSSAM:Code Search via Attention Matching of Code Semantics and Structures [8.547332796736107]
  This paper introduces a code search model named CSSAM (Code Semantics and Structures Attention Matching) By introducing semantic and structural matching mechanisms, CSSAM effectively extracts and fuses multidimensional code features. By leveraging the residual interaction, a matching module is designed to preserve more code semantics and descriptive features.
  arXiv  Detail & Related papers  (2022-08-08T05:45:40Z)
• Constrained Sequence-to-Tree Generation for Hierarchical Text Classification [10.143177923523407]
  Hierarchical Text Classification (HTC) is a challenging task where a document can be assigned to multiple hierarchically structured categories within a taxonomy. In this paper, we formulate HTC as a sequence generation task and introduce a sequence-to-tree framework (Seq2Tree) for modeling the hierarchical label structure.
  arXiv  Detail & Related papers  (2022-04-02T08:35:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.