SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative
AI Tool
- URL: http://arxiv.org/abs/2308.03983v1
- Date: Tue, 8 Aug 2023 02:00:43 GMT
- Title: SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative
AI Tool
- Authors: Youyang Ng, Daisuke Miyashita, Yasuto Hoshi, Yasuhiro Morioka, Osamu
Torii, Tomoya Kodama, Jun Deguchi
- Abstract summary: Large Language Model (LLM) based Generative AI systems have seen significant progress in recent years.
Integrating a knowledge retrieval architecture allows for seamless integration of private data into publicly available Generative AI systems.
Retrieval-Centric Generation (RCG) approach separates roles of LLMs and retrievers in context interpretation and knowledge memorization.
SimplyRetrieve is an open-source tool with the goal of providing a localized, lightweight, and user-friendly interface to these sophisticated advancements.
- Score: 0.14777718769290524
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large Language Model (LLM) based Generative AI systems have seen significant
progress in recent years. Integrating a knowledge retrieval architecture allows
for seamless integration of private data into publicly available Generative AI
systems using pre-trained LLM without requiring additional model fine-tuning.
Moreover, Retrieval-Centric Generation (RCG) approach, a promising future
research direction that explicitly separates roles of LLMs and retrievers in
context interpretation and knowledge memorization, potentially leads to more
efficient implementation. SimplyRetrieve is an open-source tool with the goal
of providing a localized, lightweight, and user-friendly interface to these
sophisticated advancements to the machine learning community. SimplyRetrieve
features a GUI and API based RCG platform, assisted by a Private Knowledge Base
Constructor and a Retrieval Tuning Module. By leveraging these capabilities,
users can explore the potential of RCG for improving generative AI performance
while maintaining privacy standards. The tool is available at
https://github.com/RCGAI/SimplyRetrieve with an MIT license.
Related papers
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models [71.25225058845324]
Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation.
Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge.
RA-LLMs have emerged to harness external and authoritative knowledge bases, rather than relying on the model's internal knowledge.
arXiv Detail & Related papers (2024-05-10T02:48:45Z) - Self-Retrieval: Building an Information Retrieval System with One Large
Language Model [102.78988790457004]
Self-Retrieval is an end-to-end, LLM-driven information retrieval architecture.
We show that Self-Retrieval significantly outperforms previous retrieval approaches by a large margin.
arXiv Detail & Related papers (2024-02-23T18:45:35Z) - How to Build an AI Tutor that Can Adapt to Any Course and Provide Accurate Answers Using Large Language Model and Retrieval-Augmented Generation [0.0]
The OpenAI Assistants API allows AI Tutor to easily embed, store, retrieve, and manage files and chat history.
The AI Tutor prototype demonstrates its ability to generate relevant, accurate answers with source citations.
arXiv Detail & Related papers (2023-11-29T15:02:46Z) - GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training
Data Exploration [97.68234051078997]
We discuss how Pyserini can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts.
We include a Jupyter Notebook-based walk through the core interoperability features, available on GitHub.
We present GAIA Search - a search engine built following previously laid out principles, giving access to four popular large-scale text collections.
arXiv Detail & Related papers (2023-06-02T12:09:59Z) - Synergistic Interplay between Search and Large Language Models for
Information Retrieval [141.18083677333848]
InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections.
InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
arXiv Detail & Related papers (2023-05-12T11:58:15Z) - Retrieval-Enhanced Machine Learning [110.5237983180089]
We describe a generic retrieval-enhanced machine learning framework, which includes a number of existing models as special cases.
REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization.
REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.
arXiv Detail & Related papers (2022-05-02T21:42:45Z) - Enabling Un-/Semi-Supervised Machine Learning for MDSE of the Real-World
CPS/IoT Applications [0.5156484100374059]
We propose a novel approach to support domain-specific Model-Driven Software Engineering (MDSE) for the real-world use-case scenarios of smart Cyber-Physical Systems (CPS) and the Internet of Things (IoT)
We argue that the majority of available data in the nature for Artificial Intelligence (AI) are unlabeled. Hence, unsupervised and/or semi-supervised ML approaches are the practical choices.
Our proposed approach is fully implemented and integrated with an existing state-of-the-art MDSE tool to serve the CPS/IoT domain.
arXiv Detail & Related papers (2021-07-06T15:51:39Z) - A Privacy-Preserving Distributed Architecture for
Deep-Learning-as-a-Service [68.84245063902908]
This paper introduces a novel distributed architecture for deep-learning-as-a-service.
It is able to preserve the user sensitive data while providing Cloud-based machine and deep learning services.
arXiv Detail & Related papers (2020-03-30T15:12:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.