Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
- URL: http://arxiv.org/abs/2405.16546v2
- Date: Tue, 2 Jul 2024 12:23:37 GMT
- Title: Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
- Authors: Sunhao Dai, Weihao Liu, Yuqi Zhou, Liang Pang, Rongju Ruan, Gang Wang, Zhenhua Dong, Jun Xu, Ji-Rong Wen,
- Abstract summary: The proliferation of Large Language Models (LLMs) has led to an influx of AI-generated content (AIGC) on the internet.
The impact of this surge in AIGC on Information Retrieval systems remains an open question.
We introduce Cocktail, a benchmark tailored for evaluating IR models in this mixed-sourced data landscape.
- Score: 60.535793237063885
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: The proliferation of Large Language Models (LLMs) has led to an influx of AI-generated content (AIGC) on the internet, transforming the corpus of Information Retrieval (IR) systems from solely human-written to a coexistence with LLM-generated content. The impact of this surge in AIGC on IR systems remains an open question, with the primary challenge being the lack of a dedicated benchmark for researchers. In this paper, we introduce Cocktail, a comprehensive benchmark tailored for evaluating IR models in this mixed-sourced data landscape of the LLM era. Cocktail consists of 16 diverse datasets with mixed human-written and LLM-generated corpora across various text retrieval tasks and domains. Additionally, to avoid the potential bias from previously included dataset information in LLMs, we also introduce an up-to-date dataset, named NQ-UTD, with queries derived from recent events. Through conducting over 1,000 experiments to assess state-of-the-art retrieval models against the benchmarked datasets in Cocktail, we uncover a clear trade-off between ranking performance and source bias in neural retrieval models, highlighting the necessity for a balanced approach in designing future IR systems. We hope Cocktail can serve as a foundational resource for IR research in the LLM era, with all data and code publicly available at \url{https://github.com/KID-22/Cocktail}.
Related papers
- Data-Centric AI in the Age of Large Language Models [51.20451986068925]
This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs)
We make the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs.
We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
arXiv Detail & Related papers (2024-06-20T16:34:07Z) - DCA-Bench: A Benchmark for Dataset Curation Agents [9.60250892491588]
We propose a dataset curation agent benchmark, DCA-Bench, to measure large language models' capability of detecting hidden dataset quality issues.
Specifically, we collect diverse real-world dataset quality issues from eight open dataset platforms as a testbed.
The proposed benchmark can also serve as a testbed for measuring the capability of LLMs in problem discovery rather than just problem-solving.
arXiv Detail & Related papers (2024-06-11T14:02:23Z) - Self-Retrieval: Building an Information Retrieval System with One Large
Language Model [102.78988790457004]
Self-Retrieval is an end-to-end, LLM-driven information retrieval architecture.
We show that Self-Retrieval significantly outperforms previous retrieval approaches by a large margin.
arXiv Detail & Related papers (2024-02-23T18:45:35Z) - LLMs may Dominate Information Access: Neural Retrievers are Biased
Towards LLM-Generated Texts [36.73455759259717]
Large language models (LLMs) have revolutionized the paradigm of information retrieval (IR) applications.
Surprisingly, our findings indicate that neural retrieval models tend to rank LLM-generated documents higher.
To mitigate the source bias, we also propose a plug-and-play debiased constraint for the optimization objective.
arXiv Detail & Related papers (2023-10-31T14:42:23Z) - Data-Juicer: A One-Stop Data Processing System for Large Language Models [73.27731037450995]
A data recipe is a mixture of data from different sources for training Large Language Models (LLMs)
We build a new system named Data-Juicer, with which we can efficiently generate diverse data recipes.
The data recipes derived with Data-Juicer gain notable improvements on state-of-the-art LLMs.
arXiv Detail & Related papers (2023-09-05T08:22:07Z) - HAGRID: A Human-LLM Collaborative Dataset for Generative
Information-Seeking with Attribution [46.41448772928026]
We introduce HAGRID (Human-in-the-loop Attributable Generative Retrieval for Information-seeking dataset) for building end-to-end generative information-seeking models.
Unlike recent efforts that focus on black-box proprietary search engines, we built our dataset atop the English subset of MIRACL, a publicly available information retrieval dataset.
arXiv Detail & Related papers (2023-07-31T17:49:18Z) - Synergistic Interplay between Search and Large Language Models for
Information Retrieval [141.18083677333848]
InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections.
InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
arXiv Detail & Related papers (2023-05-12T11:58:15Z) - DataPerf: Benchmarks for Data-Centric AI Development [81.03754002516862]
DataPerf is a community-led benchmark suite for evaluating ML datasets and data-centric algorithms.
We provide an open, online platform with multiple rounds of challenges to support this iterative development.
The benchmarks, online evaluation platform, and baseline implementations are open source.
arXiv Detail & Related papers (2022-07-20T17:47:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.