Open-source Large Language Models are Strong Zero-shot Query Likelihood
Models for Document Ranking
- URL: http://arxiv.org/abs/2310.13243v1
- Date: Fri, 20 Oct 2023 02:54:42 GMT
- Title: Open-source Large Language Models are Strong Zero-shot Query Likelihood
Models for Document Ranking
- Authors: Shengyao Zhuang and Bing Liu and Bevan Koopman and Guido Zuccon
- Abstract summary: Large language models (LLMs) have emerged as effective Query Likelihood Models (QLMs)
This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs.
We introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever.
- Score: 36.90911173089409
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: In the field of information retrieval, Query Likelihood Models (QLMs) rank
documents based on the probability of generating the query given the content of
a document. Recently, advanced large language models (LLMs) have emerged as
effective QLMs, showcasing promising ranking capabilities. This paper focuses
on investigating the genuine zero-shot ranking effectiveness of recent LLMs,
which are solely pre-trained on unstructured text data without supervised
instruction fine-tuning. Our findings reveal the robust zero-shot ranking
ability of such LLMs, highlighting that additional instruction fine-tuning may
hinder effectiveness unless a question generation task is present in the
fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking
system that integrates LLM-based QLMs with a hybrid zero-shot retriever,
demonstrating exceptional effectiveness in both zero-shot and few-shot
scenarios. We make our codebase publicly available at
https://github.com/ielab/llm-qlm.
Related papers
- PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval [76.50690734636477]
We propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus.
The retrieval system harnesses both dense text embedding and sparse bag-of-words representations.
arXiv Detail & Related papers (2024-04-29T04:51:30Z) - Contextual Biasing of Named-Entities with Large Language Models [12.396054621526643]
This paper studies contextual biasing with Large Language Models (LLMs)
During second-pass rescoring additional contextual information is provided to a LLM to boost Automatic Speech Recognition (ASR) performance.
We propose to leverage prompts for a LLM without fine tuning during rescoring which incorporate a biasing list and few-shot examples.
arXiv Detail & Related papers (2023-09-01T20:15:48Z) - Element-aware Summarization with Large Language Models: Expert-aligned
Evaluation and Chain-of-Thought Method [35.181659789684545]
Automatic summarization generates concise summaries that contain key ideas of source documents.
References from CNN/DailyMail and BBC XSum are noisy, mainly in terms of factual hallucination and information redundancy.
We propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step.
Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L.
arXiv Detail & Related papers (2023-05-22T18:54:35Z) - Large Language Models are Zero-Shot Rankers for Recommender Systems [76.02500186203929]
This work aims to investigate the capacity of large language models (LLMs) to act as the ranking model for recommender systems.
We show that LLMs have promising zero-shot ranking abilities but struggle to perceive the order of historical interactions.
We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies.
arXiv Detail & Related papers (2023-05-15T17:57:39Z) - Zero-Shot Listwise Document Reranking with a Large Language Model [58.64141622176841]
We propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data.
Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker.
arXiv Detail & Related papers (2023-05-03T14:45:34Z) - Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios.
Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
arXiv Detail & Related papers (2023-04-27T14:45:55Z) - Self-Prompting Large Language Models for Zero-Shot Open-Domain QA [67.08732962244301]
Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing background documents.
This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models.
We propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of Large Language Models.
arXiv Detail & Related papers (2022-12-16T18:23:43Z) - Beyond prompting: Making Pre-trained Language Models Better Zero-shot
Learners by Clustering Representations [24.3378487252621]
We show that zero-shot text classification can be improved simply by clustering texts in the embedding spaces of pre-trained language models.
Our approach achieves an average of 20% absolute improvement over prompt-based zero-shot learning.
arXiv Detail & Related papers (2022-10-29T16:01:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.