MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

• URL: http://arxiv.org/abs/2407.20183v1
• Date: Mon, 29 Jul 2024 17:12:40 GMT
• Title: MindSearch: Mimicking Human Minds Elicits Deep AI Searcher
• Authors: Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao,
• Abstract summary: We introduce MindSearch to mimic the human minds in web information seeking and integration. The framework can be instantiated by a simple yet effective LLM-based multi-agent framework. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth.
• Score: 20.729251584466983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and Perplexity.ai applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.

Related papers

• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent [102.31558123570437]
  Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs) We propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch.
  arXiv  Detail & Related papers  (2024-11-05T09:27:21Z)
• MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines [91.08394877954322]
  Large Multimodal Models (LMMs) have made impressive strides in AI search engines. But, whether they can function as AI search engines remains under-explored. We first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities.
  arXiv  Detail & Related papers  (2024-09-19T17:59:45Z)
• Tree Search for Language Model Agents [69.43007235771383]
  We propose an inference-time search algorithm for LM agents to perform exploration and multi-step planning in interactive web environments. Our approach is a form of best-first tree search that operates within the actual environment space. It is the first tree search algorithm for LM agents that shows effectiveness on realistic web tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:07:55Z)
• When Search Engine Services meet Large Language Models: Visions and Challenges [53.32948540004658]
  This paper conducts an in-depth examination of how integrating Large Language Models with search engines can mutually benefit both technologies. We focus on two main areas: using search engines to improve LLMs (Search4LLM) and enhancing search engine functions using LLMs (LLM4Search)
  arXiv  Detail & Related papers  (2024-06-28T03:52:13Z)
• CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models [13.108014924612114]
  We propose CoSearchAgent, a lightweight collaborative search agent powered by large language models (LLMs) CoSearchAgent is designed as a Slack plugin that can support collaborative search during multi-party conversations on this platform. It can respond to user queries with answers grounded on the relevant search results.
  arXiv  Detail & Related papers  (2024-02-09T12:10:00Z)
• Advancing the Search Frontier with AI Agents [6.839870353268828]
  Complex search tasks require more than support for rudimentary fact finding or re-finding. The recent emergence of generative artificial intelligence (AI) has the potential to offer further assistance to searchers. This article explores these issues and how AI agents are advancing the frontier of search system capabilities.
  arXiv  Detail & Related papers  (2023-11-02T13:43:22Z)
• Large Search Model: Redefining Search Stack in the Era of LLMs [63.503320030117145]
  We introduce a novel conceptual framework called large search model, which redefines the conventional search stack by unifying search tasks with one large language model (LLM) All tasks are formulated as autoregressive text generation problems, allowing for the customization of tasks through the use of natural language prompts. This proposed framework capitalizes on the strong language understanding and reasoning capabilities of LLMs, offering the potential to enhance search result quality while simultaneously simplifying the existing cumbersome search stack.
  arXiv  Detail & Related papers  (2023-10-23T05:52:09Z)
• WebCPM: Interactive Web Search for Chinese Long-form Question Answering [104.676752359777]
  Long-form question answering (LFQA) aims at answering complex, open-ended questions with detailed, paragraph-length responses. We introduce WebCPM, the first Chinese LFQA dataset. We collect 5,500 high-quality question-answer pairs, together with 14,315 supporting facts and 121,330 web search actions.
  arXiv  Detail & Related papers  (2023-05-11T14:47:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.