iQPP: A Benchmark for Image Query Performance Prediction

• URL: http://arxiv.org/abs/2302.10126v3
• Date: Mon, 10 Apr 2023 06:41:46 GMT
• Title: iQPP: A Benchmark for Image Query Performance Prediction
• Authors: Eduard Poesina, Radu Tudor Ionescu, Josiane Mothe
• Abstract summary: We propose the first benchmark for image query performance prediction (iQPP) We estimate the ground-truth difficulty of each query as the average precision or the precision@k, using two state-of-the-art image retrieval models. Next, we propose and evaluate novel pre-retrieval and post-retrieval query performance predictors, comparing them with existing or adapted (from text to image) predictors. Our comprehensive experiments indicate that iQPP is a challenging benchmark, revealing an important research gap that needs to be addressed in future work.
• Score: 24.573869540845124
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To date, query performance prediction (QPP) in the context of content-based image retrieval remains a largely unexplored task, especially in the query-by-example scenario, where the query is an image. To boost the exploration of the QPP task in image retrieval, we propose the first benchmark for image query performance prediction (iQPP). First, we establish a set of four data sets (PASCAL VOC 2012, Caltech-101, ROxford5k and RParis6k) and estimate the ground-truth difficulty of each query as the average precision or the precision@k, using two state-of-the-art image retrieval models. Next, we propose and evaluate novel pre-retrieval and post-retrieval query performance predictors, comparing them with existing or adapted (from text to image) predictors. The empirical results show that most predictors do not generalize across evaluation scenarios. Our comprehensive experiments indicate that iQPP is a challenging benchmark, revealing an important research gap that needs to be addressed in future work. We release our code and data as open source at https://github.com/Eduard6421/iQPP, to foster future research.

Related papers

• PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction [21.5792639312506]
  We propose the first benchmark for joint text-to-image prompt and query performance prediction, comprising 10K queries. We present results with several pre-generation/retrieval and post-generation/retrieval performance predictors, thus providing competitive baselines for future research.
  arXiv  Detail & Related papers  (2024-06-07T08:46:19Z)
• Beyond MOS: Subjective Image Quality Score Preprocessing Method Based on Perceptual Similarity [2.290956583394892]
  ITU-R BT.500, ITU-T P.910, and ITU-T P.913 have been standardized to clean up the original opinion scores. PSP exploit the perceptual similarity between images to alleviate subjective bias in less annotated scenarios.
  arXiv  Detail & Related papers  (2024-04-30T16:01:14Z)
• Query Performance Prediction using Relevance Judgments Generated by Large Language Models [53.97064615557883]
  We propose a QPP framework using automatically generated relevance judgments (QPP-GenRE) QPP-GenRE decomposes QPP into independent subtasks of predicting relevance of each item in a ranked list to a given query. This allows us to predict any IR evaluation measure using the generated relevance judgments as pseudo-labels.
  arXiv  Detail & Related papers  (2024-04-01T09:33:05Z)
• On the Estimation of Image-matching Uncertainty in Visual Place Recognition [7.769607568805291]
  In Visual Place Recognition (VPR) the pose of a query image is estimated by comparing the image to a map of reference images with known reference poses. This work compares for the first time the main approaches for estimating the image-matching uncertainty. We formulate a simple baseline method, SUE'', which unlike the other methods considers the freely-available poses of the reference images in the map.
  arXiv  Detail & Related papers  (2024-03-31T03:24:48Z)
• Less is More: One-shot Subgraph Reasoning on Large-scale Knowledge Graphs [49.547988001231424]
  We propose the one-shot-subgraph link prediction to achieve efficient and adaptive prediction. Design principle is that, instead of directly acting on the whole KG, the prediction procedure is decoupled into two steps. We achieve promoted efficiency and leading performances on five large-scale benchmarks.
  arXiv  Detail & Related papers  (2024-03-15T12:00:12Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)
• Query Performance Prediction: From Ad-hoc to Conversational Search [55.37199498369387]
  Query performance prediction (QPP) is a core task in information retrieval. Research has shown the effectiveness and usefulness of QPP for ad-hoc search. Despite its potential, QPP for conversational search has been little studied.
  arXiv  Detail & Related papers  (2023-05-18T12:37:01Z)
• Progressive End-to-End Object Detection in Crowded Scenes [96.92416613336096]
  Previous query-based detectors suffer from two drawbacks: first, multiple predictions will be inferred for a single object, typically in crowded scenes; second, the performance saturates as the depth of the decoding stage increases. We propose a progressive predicting method to address the above issues. Specifically, we first select accepted queries to generate true positive predictions, then refine the rest noisy queries according to the previously accepted predictions. Experiments show that our method can significantly boost the performance of query-based detectors in crowded scenes.
  arXiv  Detail & Related papers  (2022-03-15T06:12:00Z)
• On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances and Million-AID [57.71601467271486]
  This article discusses the problem of how to efficiently prepare a suitable benchmark dataset for RS image interpretation. We first analyze the current challenges of developing intelligent algorithms for RS image interpretation with bibliometric investigations. Following the presented guidances, we also provide an example on building RS image dataset, i.e., Million-AID, a new large-scale benchmark dataset.
  arXiv  Detail & Related papers  (2020-06-22T17:59:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.