SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

• URL: http://arxiv.org/abs/2609.03047v1
• Date: Wed, 02 Sep 2026 18:17:41 GMT
• Title: SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking
• Abstract summary: SHELF is a Python system that turns labelled budget, writing specifications, and a generation of data into benchmark data and evaluation tasks.<n>This first release contains 62,899 model-written documents based on Library of Congress vocabularies.<n>We compare TFBench, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Libraries and archives manage large collections with limited staff and computing budgets, yet common benchmarks do not systematically test their bibliographic work. They need to know which methods work for their tasks and what those methods require to run. SHELF, the Synthetic Harness for Evaluating LLM Fitness, addresses this gap. It is a Python system that turns labelled taxonomies, writing specifications, and a generation budget into controlled benchmark data and evaluation tasks. This first release contains 62,899 model-written documents based on Library of Congress vocabularies, with tasks for classification, clustering, retrieval, pair classification, and instruction retrieval. We compare TF, TF-IDF, BM25, popular encoders, and, on subject classification only, zero-shot decoders; each method appears only on tasks that support it. Subject classification reaches 0.8887, while genre-form classification reaches only 0.2605, and several pair and clustering tasks remain near chance. Sparse methods remain competitive on classification, while TF-IDF is the fastest measured arm in the subject timing experiment. SHELF also varies bibliographic facets independently and can generate new, verifiably unseen documents after a model's training cutoff. Comparisons with LCSHBench and Project Gutenberg show that model rankings transfer more reliably than absolute scores, but SHELF scores do not estimate accuracy on production catalogue data. We release all source code and data under permissive licenses on GitHub and Hugging Face.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• ReLeVAnT: Relevance Lexical Vectors for Accurate Legal Text Classification [5.530730417910086]
  ReLeVAnT is a framework for legal document binary classification.<n>It uses n-gram processing, contrastive score matching, and a shallow neural network as the primary drivers for discriminative classification.<n>It has 99.3% accuracy and 98.7% F1 score on the LexGLUE dataset.
  arXiv  Detail & Related papers  (2026-04-24T07:13:24Z)
• Delete and Retain: Efficient Unlearning for Document Classification [1.0026496861838448]
  Hessian Reassignment is a two-step, model-agnostic solution for class unlearning in document classification.<n>On standard text benchmarks, Hessian Reassignment retained-class accuracy close to full-without-class while running orders of magnitude faster.<n>Results demonstrate a practical, principled path to efficient class unlearning in document classification.
  arXiv  Detail & Related papers  (2025-12-06T18:57:06Z)
• Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes [44.51779041553597]
  FusionSent is an efficient and prompt-free approach for few-shot classification of scientific documents with many classes. Experiments show that FusionSent significantly outperforms strong baselines by an average of $6.0$ $F_1$label points.
  arXiv  Detail & Related papers  (2024-10-08T07:52:35Z)
• Blueprinting the Future: Automatic Item Categorization using Hierarchical Zero-Shot and Few-Shot Classifiers [6.907552533477328]
  This study unveils a novel approach employing the zero-shot and few-shot Generative Pretrained Transformer (GPT) for hierarchical item categorization. The hierarchical nature of examination blueprints is navigated seamlessly, allowing for a tiered classification of items across multiple levels. An initial simulation with artificial data demonstrates the efficacy of this method, achieving an average accuracy of 92.91% measured by the F1 score.
  arXiv  Detail & Related papers  (2023-12-06T15:51:49Z)
• Zero-Shot Listwise Document Reranking with a Large Language Model [58.64141622176841]
  We propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data. Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker.
  arXiv  Detail & Related papers  (2023-05-03T14:45:34Z)
• Precise Zero-Shot Dense Retrieval without Relevance Labels [60.457378374671656]
  Hypothetical Document Embeddings(HyDE) is a zero-shot dense retrieval system. We show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever.
  arXiv  Detail & Related papers  (2022-12-20T18:09:52Z)
• Evaluating the Factual Consistency of Large Language Models Through News Summarization [97.04685401448499]
  We propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. For factually inconsistent summaries, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.
  arXiv  Detail & Related papers  (2022-11-15T18:50:34Z)
• Self-Adaptive Label Augmentation for Semi-supervised Few-shot Classification [121.63992191386502]
  Few-shot classification aims to learn a model that can generalize well to new tasks when only a few labeled samples are available. We propose a semi-supervised few-shot classification method that assigns an appropriate label to each unlabeled sample by a manually defined metric. A major novelty of SALA is the task-adaptive metric, which can learn the metric adaptively for different tasks in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-06-16T13:14:03Z)
• LeQua@CLEF2022: Learning to Quantify [76.22817970624875]
  LeQua 2022 is a new lab for the evaluation of methods for learning to quantify'' in textual datasets. The goal of this lab is to provide a setting for the comparative evaluation of methods for learning to quantify, both in the binary setting and in the single-label multiclass setting.
  arXiv  Detail & Related papers  (2021-11-22T14:54:20Z)
• Procode: the Swiss Multilingual Solution for Automatic Coding and Recoding of Occupations and Economic Activities [0.0]
  Epidemiological studies require data that is in alignment with the classifications established for occupations or economic activities. The goal was to develop and test a web-tool, named Procode, for coding of free-texts against classifications and recoding between different classifications.
  arXiv  Detail & Related papers  (2020-11-30T07:46:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.