Match Your Words! A Study of Lexical Matching in Neural Information Retrieval

• URL: http://arxiv.org/abs/2112.05662v2
• Date: Mon, 13 Dec 2021 08:28:05 GMT
• Title: Match Your Words! A Study of Lexical Matching in Neural Information Retrieval
• Authors: Thibault Formal, Benjamin Piwowarski, St\'ephane Clinchant
• Abstract summary: We study the behavior of different state-of-the-art neural IR models, focusing on whether they are able to perform lexical matching when it's actually useful. We show that neural IR models fail to properly generalize term importance on out-of-domain collections or terms almost unseen during training.
• Score: 11.930815087240479
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Neural Information Retrieval models hold the promise to replace lexical matching models, e.g. BM25, in modern search engines. While their capabilities have fully shone on in-domain datasets like MS MARCO, they have recently been challenged on out-of-domain zero-shot settings (BEIR benchmark), questioning their actual generalization capabilities compared to bag-of-words approaches. Particularly, we wonder if these shortcomings could (partly) be the consequence of the inability of neural IR models to perform lexical matching off-the-shelf. In this work, we propose a measure of discrepancy between the lexical matching performed by any (neural) model and an 'ideal' one. Based on this, we study the behavior of different state-of-the-art neural IR models, focusing on whether they are able to perform lexical matching when it's actually useful, i.e. for important terms. Overall, we show that neural IR models fail to properly generalize term importance on out-of-domain collections or terms almost unseen during training

Related papers

• Predicting the Performance of Black-box LLMs through Self-Queries [60.87193950962585]
  Large language models (LLMs) are increasingly relied on in AI systems, predicting when they make mistakes is crucial. In this paper, we extract features of LLMs in a black-box manner by using follow-up prompts and taking the probabilities of different responses as representations. We demonstrate that training a linear model on these low-dimensional representations produces reliable predictors of model performance at the instance level.
  arXiv  Detail & Related papers  (2025-01-02T22:26:54Z)
• Discovering Biases in Information Retrieval Models Using Relevance Thesaurus as Global Explanation [23.50629779375759]
  We propose a novel method to globally explain neural relevance models by constructing a "relevance thesaurus" This thesaurus is used to augment lexical matching models such as BM25 to approximate the neural model's predictions.
  arXiv  Detail & Related papers  (2024-10-04T16:42:13Z)
• SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection [13.811326866261888]
  Existing ADD models suffer from generalization issues. Black-box nature of existing models limits their use in real-world scenarios. We introduce a new ADD model that explicitly uses the StyleLInguistics Mismatch (SLIM) in fake speech to separate them from real speech.
  arXiv  Detail & Related papers  (2024-07-26T05:23:41Z)
• Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR [99.13855300096925]
  We run a study on the Touch'e 2020 data to explore the potential limits of neural retrieval models. Our black-box evaluation reveals an inherent bias of neural models towards retrieving short passages. As many of the short Touch'e passages are not argumentative and thus non-relevant per se, we denoise the Touch'e 2020 data by excluding very short passages.
  arXiv  Detail & Related papers  (2024-07-10T16:07:51Z)
• Plot Retrieval as an Assessment of Abstract Semantic Association [131.58819293115124]
  Text pairs in Plot Retrieval have less word overlap and more abstract semantic association. Plot Retrieval can be the benchmark for further research on the semantic association modeling ability of IR models.
  arXiv  Detail & Related papers  (2023-11-03T02:02:43Z)
• Distilling BlackBox to Interpretable models for Efficient Transfer Learning [19.40897632956169]
  Building generalizable AI models is one of the primary challenges in the healthcare domain. Fine-tuning a model to transfer knowledge from one domain to another requires a significant amount of labeled data in the target domain. We develop an interpretable model that can be efficiently fine-tuned to an unseen target domain with minimal computational cost.
  arXiv  Detail & Related papers  (2023-05-26T23:23:48Z)
• NevIR: Negation in Neural Information Retrieval [45.9442701147499]
  Negation is a common everyday phenomenon and has been a consistent area of weakness for language models (LMs) We construct a benchmark asking IR models to rank two documents that differ only by negation. We show that the results vary widely according to the type of IR architecture: cross-encoders perform best, followed by late-interaction models, and in last place are bi-encoder and sparse neural architectures.
  arXiv  Detail & Related papers  (2023-05-12T17:05:54Z)
• Reveal to Revise: An Explainable AI Life Cycle for Iterative Bias Correction of Deep Models [11.879170124003252]
  State-of-the-art machine learning models often learn spurious correlations embedded in the training data. This poses risks when deploying these models for high-stake decision-making. We propose Reveal to Revise (R2R) to identify, mitigate, and (re-)evaluate spurious model behavior.
  arXiv  Detail & Related papers  (2023-03-22T15:23:09Z)
• Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding [82.46024259137823]
  We propose a cross-model comparative loss for a broad range of tasks. We demonstrate the universal effectiveness of comparative loss through extensive experiments on 14 datasets from 3 distinct NLU tasks.
  arXiv  Detail & Related papers  (2023-01-10T03:04:27Z)
• Learning from Context or Names? An Empirical Study on Neural Relation Extraction [112.06614505580501]
  We study the effect of two main information sources in text: textual context and entity mentions (names) We propose an entity-masked contrastive pre-training framework for relation extraction (RE) Our framework can improve the effectiveness and robustness of neural models in different RE scenarios.
  arXiv  Detail & Related papers  (2020-10-05T11:21:59Z)
• Neural Additive Models: Interpretable Machine Learning with Neural Nets [77.66871378302774]
  Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature.
  arXiv  Detail & Related papers  (2020-04-29T01:28:32Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.