QueryGym: A Toolkit for Reproducible LLM-Based Query Reformulation

• URL: http://arxiv.org/abs/2511.15996v1
• Date: Thu, 20 Nov 2025 02:45:50 GMT
• Title: QueryGym: A Toolkit for Reproducible LLM-Based Query Reformulation
• Authors: Amin Bigdeli, Radin Hamidi Rad, Mert Incesu, Negar Arabzadeh, Charles L. A. Clarke, Ebrahim Bagheri,
• Abstract summary: We present QueryGym, a Python toolkit that supports large language model (LLM)-based query reformulation.<n>The toolkit provides a unified framework for implementing, executing, and comparing llm-based reformulation methods.
• Score: 21.804685308876326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present QueryGym, a lightweight, extensible Python toolkit that supports large language model (LLM)-based query reformulation. This is an important tool development since recent work on llm-based query reformulation has shown notable increase in retrieval effectiveness. However, while different authors have sporadically shared the implementation of their methods, there is no unified toolkit that provides a consistent implementation of such methods, which hinders fair comparison, rapid experimentation, consistent benchmarking and reliable deployment. QueryGym addresses this gap by providing a unified framework for implementing, executing, and comparing llm-based reformulation methods. The toolkit offers: (1) a Python API for applying diverse LLM-based methods, (2) a retrieval-agnostic interface supporting integration with backends such as Pyserini and PyTerrier, (3) a centralized prompt management system with versioning and metadata tracking, (4) built-in support for benchmarks like BEIR and MS MARCO, and (5) a completely open-source extensible implementation available to all researchers. QueryGym is publicly available at https://github.com/radinhamidi/QueryGym.

Related papers

• DiffuRank: Effective Document Reranking with Diffusion Language Models [71.16830004674513]
  We propose DiffuRank, a reranking framework built upon diffusion language models (dLLMs)<n>dLLMs support more flexible decoding and generation processes that are not constrained to a left-to-right order.<n>We show dLLMs achieve performance comparable to, and in some cases exceeding, that of autoregressive LLMs with similar model sizes.
  arXiv  Detail & Related papers  (2026-02-13T02:18:14Z)
• ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers [33.08308979741825]
  Existing retrieval models rank tools based on the similarity between a user query and a tool description (TD)<n>This leads to suboptimal retrieval as user requests are often poorly aligned with the language of TD.<n>We propose ToolDreamer, a framework to condition retriever models to fetch tools based on hypothetical (synthetic) TD.
  arXiv  Detail & Related papers  (2025-10-22T17:26:05Z)
• A Framework for Testing and Adapting REST APIs as LLM Tools [11.757827071584737]
  Large Language Models (LLMs) are increasingly used to build autonomous agents that perform complex tasks with external tools.<n>Current benchmarks overlook these challenges, leaving a gap in assessing API readiness for agent-driven automation.<n>We present a testing framework that systematically evaluates enterprise APIs when wrapped as Python tools for LLM-based agents.
  arXiv  Detail & Related papers  (2025-04-22T02:52:08Z)
• LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking [15.060195612587805]
  We introduce a unified framework, textbfLLM4Ranking, which enables users to adopt different ranking methods using open-source or closed-source API-based LLMs.<n>Our framework provides a simple and interface for document reranking with LLMs, as well as easy-to-use evaluation and fine-tuning scripts for this task.
  arXiv  Detail & Related papers  (2025-04-10T04:08:38Z)
• SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution [56.9361004704428]
  Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks.<n>SWE-Fixer is a novel open-source framework designed to effectively and efficiently resolve GitHub issues.<n>We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving competitive performance among open-source models.
  arXiv  Detail & Related papers  (2025-01-09T07:54:24Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research [70.6584488911715]
  retrieval-augmented generation (RAG) has attracted considerable research attention.<n>Existing RAG toolkits are often heavy and inflexibly, failing to meet the customization needs of researchers.<n>Our toolkit has implemented 16 advanced RAG methods and gathered and organized 38 benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-22T12:12:40Z)
• AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents [19.439775106707344]
  AgentQuest is a framework where benchmarks and metrics are modular and easily through well documented and easy-to-use APIs. We offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase.
  arXiv  Detail & Related papers  (2024-04-09T16:01:24Z)
• Optimizing LLM Queries in Relational Data Analytics Workloads [50.95919232839785]
  Batch data analytics is a growing application for Large Language Models (LLMs)<n>LLMs enable users to perform a wide range of natural language tasks, such as classification, entity extraction, and translation, over large datasets.<n>We propose novel techniques that can significantly reduce the cost of LLM calls for relational data analytics workloads.
  arXiv  Detail & Related papers  (2024-03-09T07:01:44Z)
• SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval [92.27387459751309]
  We provide SPRINT, a unified Python toolkit for evaluating neural sparse retrieval. We establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document.
  arXiv  Detail & Related papers  (2023-07-19T22:48:02Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• PyRelationAL: a python library for active learning research and development [1.0061110876649197]
  Active learning (AL) is a sub-field of ML focused on the development of methods to iteratively and economically acquire data. Here, we introduce PyRelationAL, an open source library for AL research. We describe a modular toolkit based around a two step design methodology for composing pool-based active learning strategies.
  arXiv  Detail & Related papers  (2022-05-23T08:21:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.