Beyond Semantic IDs: Encoding Business-Value Ranking into Document Identifiers for Generative Retrieval

• URL: http://arxiv.org/abs/2607.11392v1
• Date: Mon, 13 Jul 2026 10:57:20 GMT
• Title: Beyond Semantic IDs: Encoding Business-Value Ranking into Document Identifiers for Generative Retrieval
• Abstract summary: Generative Retrieval (GR) formulates retrieval as a sequence-to-sequence generation task, assigning each document a document identifier (DocID) and retrieving it through autoregressive decoding.<n>Existing schemes based on discrete representation learning suffer from inherent collision issues.<n>We propose Cluster-Ranked Identifier (CRID), which decouples DocID into semantic clustering and business-value ranking, yielding collision-free identifiers.
• Score: 28.485766948414206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative Retrieval (GR) formulates retrieval as a sequence-to-sequence generation task, assigning each document a document identifier (DocID) and retrieving it through autoregressive decoding, making DocID design a critical factor in retrieval quality. However, existing schemes based on discrete representation learning suffer from inherent collision issues and create a mismatch between the DocID's encoding objective and the system's business optimization target. To address these limitations, we propose Cluster-Ranked Identifier (CRID), which decouples DocID into semantic clustering and business-value ranking, yielding collision-free identifiers that support incremental updates via intra-cluster reranking. We further introduce an analytical framework that decomposes retrieval gains into personalized preference and statistical prior generalization, revealing how semantic cluster size governs the balance between the two components. Experiments on a 300M-item Taobao e-commerce corpus show that CRID surpasses the strongest embedding-based retrieval baseline on top-K Hitrate, and delivers +1.06% GMV in full-traffic deployment.

Related papers

• Token-Level Credit Assignment Optimization for Generative Document Retrieval [48.04815798182699]
  Generative retrieval models perform document retrieval by autoregressively generating document identifiers (DocIDs)<n>Retrieval effectiveness is typically evaluated only after the full DocID is generated, creating a mismatch between token-level generation and document-level relevance supervision.<n>We propose a fine-grained reinforcement learning framework for generative retrieval with token-level relevance rewards.
  arXiv  Detail & Related papers  (2026-08-12T13:28:56Z)
• CAMI: Cost-Aware Agent-Guided Multi-Indexing for Semantic Retrieval [7.07006462273528]
  CAMI (Cost-Aware Multi-Indexing) is a framework that formalizes multi-index construction as a budgeted, multi-objective portfolio selection problem.<n>Our findings reveal that the framework systematically isolates high-recall portfolios under strict budget constraints.<n> CAMI is able to systematically identify these high-recall portfolios using up to 5x less budget compared to random search baselines.
  arXiv  Detail & Related papers  (2026-06-14T16:59:18Z)
• CHOP: Chunkwise Context-Preserving Framework for RAG on Multi Documents [6.134883925217888]
  We propose CHOP, a framework that iteratively evaluates chunk relevance with Large Language Models (LLMs)<n>CHOP integrates two key components: the CNM-Extractor, which generates compact per-chunk signatures capturing categories, key nouns, and model names, and the Continuity Decision Module, which preserves contextual coherence.<n> Experiments on benchmark datasets show that CHOP alleviates retrieval confusion and provides a scalable approach for building high-quality knowledge bases.
  arXiv  Detail & Related papers  (2026-04-17T08:05:17Z)
• Order-agnostic Identifier for Large Language Model-based Generative Recommendation [94.37662915542603]
  Items are assigned identifiers for Large Language Models (LLMs) to encode user history and generate the next item.<n>Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings.<n>We propose SETRec, which leverages semantic tokenizers to obtain order-agnostic multi-dimensional tokens.
  arXiv  Detail & Related papers  (2025-02-15T15:25:38Z)
• Generative Retrieval Meets Multi-Graded Relevance [104.75244721442756]
  We introduce a framework called GRaded Generative Retrieval (GR$2$) GR$2$ focuses on two key components: ensuring relevant and distinct identifiers, and implementing multi-graded constrained contrastive training. Experiments on datasets with both multi-graded and binary relevance demonstrate the effectiveness of GR$2$.
  arXiv  Detail & Related papers  (2024-09-27T02:55:53Z)
• CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling [53.97609687516371]
  Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data.<n>Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates.<n>We propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling.
  arXiv  Detail & Related papers  (2024-06-25T12:47:04Z)
• ASI++: Towards Distributionally Balanced End-to-End Generative Retrieval [29.65717446547002]
  ASI++ is a novel fully end-to-end generative retrieval method. It aims to simultaneously learn balanced ID assignments and improve retrieval performance.
  arXiv  Detail & Related papers  (2024-05-23T07:54:57Z)
• Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding [23.061797784952855]
  This paper introduces PAG, a novel optimization and decoding approach that guides autoregressive generation of document identifiers. Experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin.
  arXiv  Detail & Related papers  (2024-04-22T21:50:01Z)
• Language Models As Semantic Indexers [78.83425357657026]
  We introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval.
  arXiv  Detail & Related papers  (2023-10-11T18:56:15Z)
• How Does Generative Retrieval Scale to Millions of Passages? [68.98628807288972]
  We conduct the first empirical study of generative retrieval techniques across various corpus scales. We scale generative retrieval to millions of passages with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. While generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge.
  arXiv  Detail & Related papers  (2023-05-19T17:33:38Z)
• Recommender Systems with Generative Retrieval [58.454606442670034]
  We propose a novel generative retrieval approach, where the retrieval model autoregressively decodes the identifiers of the target candidates. To that end, we create semantically meaningful of codewords to serve as a Semantic ID for each item. We show that recommender systems trained with the proposed paradigm significantly outperform the current SOTA models on various datasets.
  arXiv  Detail & Related papers  (2023-05-08T21:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.