Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval

• URL: http://arxiv.org/abs/2407.21488v2
• Date: Thu, 31 Oct 2024 11:45:00 GMT
• Title: Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval
• Authors: Zhirui Kuai, Zuxu Chen, Huimu Wang, Mingming Li, Dadong Miao, Binbin Wang, Xusong Chen, Li Kuang, Yuxing Han, Jiaxing Wang, Guoyu Tang, Lin Liu, Songlin Wang, Jingwei Zhuo,
• Abstract summary: Generative retrieval (GR) has emerged as a transformative paradigm in search and recommender systems. "Hourglass" phenomenon substantially impacts the performance of RQ-SID in generative retrieval. We propose effective solutions to mitigate this issue, thereby enhancing the effectiveness of generative retrieval in real-world E-commerce applications.
• Score: 16.953923822238455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative retrieval (GR) has emerged as a transformative paradigm in search and recommender systems, leveraging numeric-based identifier representations to enhance efficiency and generalization. Notably, methods like TIGER employing Residual Quantization-based Semantic Identifiers (RQ-SID), have shown significant promise in e-commerce scenarios by effectively managing item IDs. However, a critical issue termed the "\textbf{Hourglass}" phenomenon, occurs in RQ-SID, where intermediate codebook tokens become overly concentrated, hindering the full utilization of generative retrieval methods. This paper analyses and addresses this problem by identifying data sparsity and long-tailed distribution as the primary causes. Through comprehensive experiments and detailed ablation studies, we analyze the impact of these factors on codebook utilization and data distribution. Our findings reveal that the "Hourglass" phenomenon substantially impacts the performance of RQ-SID in generative retrieval. We propose effective solutions to mitigate this issue, thereby significantly enhancing the effectiveness of generative retrieval in real-world E-commerce applications.

Related papers

• HeteRAG: A Heterogeneous Retrieval-augmented Generation Framework with Decoupled Knowledge Representations [36.61614799098233]
  Retrieval-augmented generation (RAG) methods can enhance the performance of LLMs. Existing RAG methods typically employ identical representations of knowledge chunks for both retrieval and generation. We propose a heterogeneous RAG framework (myname) that decouples the representations of knowledge chunks for retrieval and generation.
  arXiv  Detail & Related papers  (2025-04-12T13:12:54Z)
• Exploring Training and Inference Scaling Laws in Generative Retrieval [50.82554729023865]
  We investigate how model size, training data scale, and inference-time compute jointly influence generative retrieval performance. Our experiments show that n-gram-based methods demonstrate strong alignment with both training and inference scaling laws. We find that LLaMA models consistently outperform T5 models, suggesting a particular advantage for larger decoder-only models in generative retrieval.
  arXiv  Detail & Related papers  (2025-03-24T17:59:03Z)
• SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction [20.6787276745193]
  We introduce an automatic evaluation method that measures retrieval quality through the lens of information gain within the RAG framework. We quantify the utility of retrieval by the extent to which it reduces semantic perplexity post-retrieval.
  arXiv  Detail & Related papers  (2025-03-03T12:37:34Z)
• ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents [27.90338725230132]
  ViDoSeek is a dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. We propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark.
  arXiv  Detail & Related papers  (2025-02-25T09:26:12Z)
• DeepRAG: Thinking to Retrieval Step by Step for Large Language Models [92.87532210660456]
  We propose DeepRAG, a framework that models retrieval-augmented reasoning as a Markov Decision Process (MDP) By iteratively decomposing queries, DeepRAG dynamically determines whether to retrieve external knowledge or rely on parametric reasoning at each step. Experiments show that DeepRAG improves retrieval efficiency while improving answer accuracy by 21.99%, demonstrating its effectiveness in optimizing retrieval-augmented reasoning.
  arXiv  Detail & Related papers  (2025-02-03T08:22:45Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• Context Awareness Gate For Retrieval Augmented Generation [2.749898166276854]
  Retrieval Augmented Generation (RAG) has emerged as a widely adopted approach to mitigate the limitations of large language models (LLMs) in answering domain-specific questions. Previous research has predominantly focused on improving the accuracy and quality of retrieved data chunks to enhance the overall performance of the generation pipeline. We investigate the impact of retrieving irrelevant information in open-domain question answering, highlighting its significant detrimental effect on the quality of LLM outputs.
  arXiv  Detail & Related papers  (2024-11-25T06:48:38Z)
• Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation [72.70046559930555]
  We propose a generic RAG approach called Adaptive Note-Enhanced RAG (Adaptive-Note) for complex QA tasks. Specifically, Adaptive-Note introduces an overarching view of knowledge growth, iteratively gathering new information in the form of notes. In addition, we employ an adaptive, note-based stop-exploration strategy to decide "what to retrieve and when to stop" to encourage sufficient knowledge exploration.
  arXiv  Detail & Related papers  (2024-10-11T14:03:29Z)
• WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs [10.380692079063467]
  We propose WeKnow-RAG, which integrates Web search and Knowledge Graphs into a "Retrieval-Augmented Generation (RAG)" system. First, the accuracy and reliability of LLM responses are improved by combining the structured representation of Knowledge Graphs with the flexibility of dense vector retrieval. Our approach effectively balances the efficiency and accuracy of information retrieval, thus improving the overall retrieval process.
  arXiv  Detail & Related papers  (2024-08-14T15:19:16Z)
• Generative Retrieval with Preference Optimization for E-commerce Search [16.78829577915103]
  We develop an innovative framework for E-commerce search, called generative retrieval with preference optimization. We employ multi-span identifiers to represent raw item titles and transform the task of generating titles from queries into the task of generating multi-span identifiers from queries. Our experiments show that this framework achieves competitive performance on a real-world dataset, and online A/B tests demonstrate the superiority and effectiveness in improving conversion gains.
  arXiv  Detail & Related papers  (2024-07-29T09:31:19Z)
• Dynamic Data Pruning for Automatic Speech Recognition [58.95758272440217]
  We introduce Dynamic Data Pruning for ASR (DDP-ASR), which offers fine-grained pruning granularities specifically tailored for speech-related datasets. Our experiments show that DDP-ASR can save up to 1.6x training time with negligible performance loss.
  arXiv  Detail & Related papers  (2024-06-26T14:17:36Z)
• A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems [67.52782366565658]
  State-of-the-art recommender systems (RSs) depend on categorical features, which ecoded by embedding vectors, resulting in excessively large embedding tables. Despite the prosperity of lightweight embedding-based RSs, a wide diversity is seen in evaluation protocols. This study investigates various LERS' performance, efficiency, and cross-task transferability via a thorough benchmarking process.
  arXiv  Detail & Related papers  (2024-06-25T07:45:00Z)
• ASI++: Towards Distributionally Balanced End-to-End Generative Retrieval [29.65717446547002]
  ASI++ is a novel fully end-to-end generative retrieval method. It aims to simultaneously learn balanced ID assignments and improve retrieval performance.
  arXiv  Detail & Related papers  (2024-05-23T07:54:57Z)
• A Survey on Retrieval-Augmented Text Generation for Large Language Models [1.4579344926652844]
  Retrieval-Augmented Generation (RAG) merges retrieval methods with deep learning advancements. This paper organizes the RAG paradigm into four categories: pre-retrieval, retrieval, post-retrieval, and generation. It outlines RAG's evolution and discusses the field's progression through the analysis of significant studies.
  arXiv  Detail & Related papers  (2024-04-17T01:27:42Z)
• Transformer for Object Re-Identification: A Survey [69.61542572894263]
  Vision Transformers have spurred a growing number of studies delving deeper into Transformer-based Re-ID. This paper provides a comprehensive review and in-depth analysis of the Transformer-based Re-ID. Considering the trending unsupervised Re-ID, we propose a new Transformer baseline, UntransReID, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-01-13T03:17:57Z)
• Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition [94.56304526014875]
  We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER) Our method exploits self-supervised pretraining to learn good feature representations from the target data. We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
  arXiv  Detail & Related papers  (2022-10-11T08:24:50Z)
• Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval [152.3504607706575]
  This research aims to conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks. We exploit to train a more effective cross-modal model which is adaptively capable of incorporating key concept information from the multi-modal data.
  arXiv  Detail & Related papers  (2022-06-17T15:40:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.