Comparative Analysis of Retrieval Systems in the Real World

• URL: http://arxiv.org/abs/2405.02048v1
• Date: Fri, 3 May 2024 12:30:01 GMT
• Title: Comparative Analysis of Retrieval Systems in the Real World
• Authors: Dmytro Mozolevskyi, Waseem AlShikh,
• Abstract summary: The objective is to evaluate and compare various state-of-the-art methods based on their performance in terms of accuracy and efficiency. The analysis explores different combinations of technologies, including Azure Cognitive Search Retriever with GPT-4, Pinecone's Canopy framework, Langchain with Pinecone and different language models. The motivation for this analysis arises from the increasing demand for robust and responsive question-answering systems in various domains.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This research paper presents a comprehensive analysis of integrating advanced language models with search and retrieval systems in the fields of information retrieval and natural language processing. The objective is to evaluate and compare various state-of-the-art methods based on their performance in terms of accuracy and efficiency. The analysis explores different combinations of technologies, including Azure Cognitive Search Retriever with GPT-4, Pinecone's Canopy framework, Langchain with Pinecone and different language models (OpenAI, Cohere), LlamaIndex with Weaviate Vector Store's hybrid search, Google's RAG implementation on Cloud VertexAI-Search, Amazon SageMaker's RAG, and a novel approach called KG-FID Retrieval. The motivation for this analysis arises from the increasing demand for robust and responsive question-answering systems in various domains. The RobustQA metric is used to evaluate the performance of these systems under diverse paraphrasing of questions. The report aims to provide insights into the strengths and weaknesses of each method, facilitating informed decisions in the deployment and development of AI-driven search and retrieval systems.

Related papers

• HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks [50.871243190126826]
  HawkBench is a human-labeled, multi-domain benchmark designed to rigorously assess RAG performance. By stratifying tasks based on information-seeking behaviors, HawkBench provides a systematic evaluation of how well RAG systems adapt to diverse user needs.
  arXiv  Detail & Related papers  (2025-02-19T06:33:39Z)
• Enhancing Retrieval-Augmented Generation: A Study of Best Practices [16.246719783032436]
  We develop advanced RAG system designs that incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG. Our study systematically investigates key factors, including language model size, prompt design, document chunk size, knowledge base size, retrieval stride, query expansion techniques, and Focus Mode retrieving relevant context at sentence-level. Our findings offer actionable insights for developing RAG systems, striking a balance between contextual richness and retrieval-generation efficiency.
  arXiv  Detail & Related papers  (2025-01-13T15:07:55Z)
• A Proposed Large Language Model-Based Smart Search for Archive System [0.0]
  This study presents a novel framework for smart search in digital archival systems. By employing a Retrieval-Augmented Generation (RAG) approach, the framework enables the processing of natural language queries. We present the architecture and implementation of the system and evaluate its performance in four experiments.
  arXiv  Detail & Related papers  (2025-01-13T02:53:07Z)
• GeAR: Generation Augmented Retrieval [82.20696567697016]
  Document retrieval techniques form the foundation for the development of large-scale information systems. The prevailing methodology is to construct a bi-encoder and compute the semantic similarity. We propose a new method called $textbfGe$neration that incorporates well-designed fusion and decoding modules.
  arXiv  Detail & Related papers  (2025-01-06T05:29:00Z)
• RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems [7.418034397164883]
  RAG Playground is an open-source framework for systematic evaluation of Retrieval-Augmented Generation (RAG) systems. We introduce a comprehensive evaluation framework with novel metrics and provide empirical results comparing different language models.
  arXiv  Detail & Related papers  (2024-12-16T19:40:26Z)
• Enhancing LLM Reasoning with Reward-guided Tree Search [95.06503095273395]
  o1-like reasoning approach is challenging, and researchers have been making various attempts to advance this open area of research. We present a preliminary exploration into enhancing the reasoning abilities of LLMs through reward-guided tree search algorithms.
  arXiv  Detail & Related papers  (2024-11-18T16:15:17Z)
• Optimizing Retrieval-Augmented Generation with Elasticsearch for Enhanced Question-Answering Systems [2.4299671488193497]
  This study aims to improve the accuracy and quality of large-scale language models (LLMs) in answering questions by integrating into the Retrieval Augmented Generation (RAG) framework. The experiment uses the Stanford Question Answering dataset (SQuAD) version 2.0 as the test dataset.
  arXiv  Detail & Related papers  (2024-10-18T04:17:49Z)
• Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
  We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books. Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
  arXiv  Detail & Related papers  (2024-05-31T20:15:10Z)
• STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases [93.96463520716759]
  We develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties.
  arXiv  Detail & Related papers  (2024-04-19T22:54:54Z)
• End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations [7.780766187171571]
  We propose a neural ASR-free keyword search model which achieves competitive performance. We extend this work with multilingual pretraining and detailed analysis of the model. Our experiments show that the proposed multilingual training significantly improves the model performance.
  arXiv  Detail & Related papers  (2023-08-15T20:33:25Z)
• Large Language Models for Information Retrieval: A Survey [58.30439850203101]
  Information retrieval has evolved from term-based methods to its integration with advanced neural models. Recent research has sought to leverage large language models (LLMs) to improve IR systems. We delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers.
  arXiv  Detail & Related papers  (2023-08-14T12:47:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.