Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval

• URL: http://arxiv.org/abs/2411.04677v1
• Date: Thu, 07 Nov 2024 13:03:21 GMT
• Title: Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval
• Authors: Ferdinand Schlatt, Maik Fröbe, Matthias Hagen,
• Abstract summary: This paper introduces Lightning IR, a PyTorch-based framework for fine-tuning and inference of transformer-based language models for information retrieval. Lightning IR supports all stages of an information retrieval pipeline: from fine-tuning and indexing to searching and re-ranking.
• Score: 43.40675637622642
• License:
• Abstract: A wide range of transformer-based language models have been proposed for information retrieval tasks. However, fine-tuning and inference of these models is often complex and requires substantial engineering effort. This paper introduces Lightning IR, a PyTorch Lightning-based framework for fine-tuning and inference of transformer-based language models for information retrieval. Lightning IR provides a modular and extensible architecture that supports all stages of an information retrieval pipeline: from fine-tuning and indexing to searching and re-ranking. It is designed to be straightforward to use, scalable, and reproducible. Lightning IR is available as open-source: https://github.com/webis-de/lightning-ir.

Related papers

• Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective [111.58315434849047]
  robustness of neural information retrieval models (IR) models has garnered significant attention. We view the robustness of IR to be a multifaceted concept, emphasizing its necessity against adversarial attacks, out-of-distribution (OOD) scenarios and performance variance. We provide an in-depth discussion of existing methods, datasets, and evaluation metrics, shedding light on challenges and future directions in the era of large language models.
  arXiv  Detail & Related papers  (2024-07-09T16:07:01Z)
• ir_explain: a Python Library of Explainable IR Methods [2.6746131626710725]
  irexplain is a Python library that implements a variety of techniques for Explainable IR (ExIR) within a common framework. irexplain supports the three standard categories of post-hoc explanations, namely pointwise, pairwise, and listwise explanations. The library is designed to make it easy to reproduce state-of-the-art ExIR baselines on standard test collections.
  arXiv  Detail & Related papers  (2024-04-29T09:37:24Z)
• Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge [11.905387325966311]
  This paper focuses on adapting a large object detection model trained on RGB images to new data extracted from IR images. We propose Modality Translator (ModTr) as an alternative to the common approach of fine-tuning a large model to the new modality.
  arXiv  Detail & Related papers  (2024-04-01T21:28:50Z)
• Large Language Models for Information Retrieval: A Survey [58.30439850203101]
  Information retrieval has evolved from term-based methods to its integration with advanced neural models. Recent research has sought to leverage large language models (LLMs) to improve IR systems. We delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers.
  arXiv  Detail & Related papers  (2023-08-14T12:47:22Z)
• Zero-shot Composed Text-Image Retrieval [72.43790281036584]
  We consider the problem of composed image retrieval (CIR) It aims to train a model that can fuse multi-modal information, e.g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
  arXiv  Detail & Related papers  (2023-06-12T17:56:01Z)
• SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization [59.732036564862796]
  We propose the Structure Information Modeling Transformer (SIM-Trans) to incorporate object structure information into transformer for enhancing discriminative representation learning. The proposed two modules are light-weighted and can be plugged into any transformer network and trained end-to-end easily. Experiments and analyses demonstrate that the proposed SIM-Trans achieves state-of-the-art performance on fine-grained visual categorization benchmarks.
  arXiv  Detail & Related papers  (2022-08-31T03:00:07Z)
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration [130.89746032163106]
  We propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data. We show that the energy function and sampler can be trained efficiently via a new variational form of power iteration. We present an energy model guided fuzzer for software testing that achieves comparable performance to well engineered fuzzing engines like libfuzzer.
  arXiv  Detail & Related papers  (2020-11-10T19:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.