SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics

• URL: http://arxiv.org/abs/2605.31575v1
• Date: Fri, 29 May 2026 17:44:15 GMT
• Title: SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics
• Abstract summary: This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections.<n>A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth.<n>Results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.
• Score: 0.8557392136621891
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scalable information retrieval testing needs corpora that are large enough to stress index construction, ranking latency, query routing, and evaluation tooling, yet human-judged test collections remain expensive and may be unavailable when documents are private or still under design. This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections through a separation of latent topical structure, surface text realization, metadata controls, query intent generation, and deterministic relevance oracles. The framework is intended as a diagnostic complement to Cranfield-style and TREC-style evaluation, not as a replacement for human assessment. A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth and producing graded relevance labels for 96 queries. In the local simulation study, generation remained close to linear at roughly 12K to 14K documents per second, estimated Zipf slopes stayed near 0.86 in absolute value, and increasing cross-topic distractor text reduced BM25 nDCG@10 from 1.00 at 2% distractors to 0.43 at 36% distractors. These results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.

Related papers

• Sketch2Inspire: Structure-Sensitive Evaluation for Product Retrieval [2.7074235008521246]
  We present Sketch2Inspire, built from a curated subset of Amazon Berkeley Objects with aligned text queries, edge-based sketch-proxy queries, and fused text--sketch queries.<n>The resource separates broad category-level retrieval from structure-sensitive within-category retrieval and includes a human-graded reference protocol for calibration.
  arXiv  Detail & Related papers  (2026-08-29T16:52:25Z)
• Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill [8.917367808177904]
  We present Spark-to-Paper, an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant.<n>It separates experiment planning from reporting, so that required evidence is specified before results are observed and manuscript claims are revised according to measured outcomes.<n>Across eight controlled research topics, Spark-to-Paper achieves 99.5% citation validity and 96.4% figure editability.
  arXiv  Detail & Related papers  (2026-08-12T11:11:07Z)
• Guardian Crawler: Retrieval-First Knowledge Discovery with Bounded LLM Augmentation for Noisy Web Intelligence [0.688204255655161]
  We present Guardian, a reproducible retrieval-first testbed for controlled experiments on knowledge discovery and evidence-grounded summarization over synthetic web-like corpora.<n>The architecture combines BM25 retrieval with risk-aware, embedding-augmented, and hybrid reranking, followed by constrained document retrieval-augmented generation with explicit citations.
  arXiv  Detail & Related papers  (2026-08-10T01:28:11Z)
• Automated sign detection across the Electronic Babylonian Library: A large-scale dataset and end-to-end cuneiform OCR pipeline [0.8481798330936975]
  The largest annotated cuneiform sign dataset to date is used, and a Deformable Detection Transformer (DETR)-based object detection model is evaluated.<n>The proposed system integrates automatic tablet-side extraction, line grouping, and n-gram-based textual similarity evaluation to bridge visual sign detection and textual structure.<n>At inference, the method is applied to 87,668 tablet fragments from the Electronic Babylonian Library (eBL) corpus, producing nearly 2.9 million sign detections.
  arXiv  Detail & Related papers  (2026-06-21T17:31:05Z)
• STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval [0.0]
  We propose STCALIR, a framework for generating test collections directly from raw legal documents.<n>The pipeline follows the Cranfield paradigm, maintaining its core components of topics, corpus, and relevance judgments.<n>We validate STCALIR using the Mr. TyDi benchmark, demonstrating that the resulting semi-synthetic relevance judgments yield retrieval effectiveness comparable to human-annotated evaluations.
  arXiv  Detail & Related papers  (2026-04-01T10:50:28Z)
• Graph-Aware Late Chunking for Retrieval-Augmented Generation in Biomedical Literature [0.0]
  GraLC-RAG is a framework that unifies late chunking with graph-aware structural intelligence.<n>We evaluate six strategies on 2,359 IMRaD-filtered Central articles using 2,033 cross-section questions.
  arXiv  Detail & Related papers  (2026-03-23T23:22:58Z)
• Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
  Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
  arXiv  Detail & Related papers  (2026-03-09T18:20:20Z)
• CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning [48.56088080889236]
  We introduce CorpusQA, a new benchmark scaling up to 10 million tokens, generated via a novel data synthesis framework.<n>We show that finetuning on our synthesized data effectively enhances an LLM's general long-context reasoning capabilities.<n>Our findings indicate that memory-augmented agentic architectures offer a more robust alternative.
  arXiv  Detail & Related papers  (2026-01-21T12:52:30Z)
• Fixturize: Bridging the Fixture Gap in Test Generation [31.82935387488973]
  Fixturize is a diagnostic framework that proactively identifies fixture-dependent functions.<n>It synthesizes test fixtures accordingly through an iterative, feedback-driven process.
  arXiv  Detail & Related papers  (2026-01-10T16:47:32Z)
• DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing [53.85037373860246]
  We introduce Deep Synth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities.<n>We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization)<n>Our results demonstrate that agentic plan-and-write significantly outperform single-turn generation.
  arXiv  Detail & Related papers  (2026-01-07T03:07:52Z)
• HySemRAG: A Hybrid Semantic Retrieval-Augmented Generation Framework for Automated Literature Synthesis and Methodological Gap Analysis [55.2480439325792]
  HySemRAG is a framework that combines Extract, Transform, Load (ETL) pipelines with Retrieval-Augmented Generation (RAG)<n>System addresses limitations in existing RAG architectures through a multi-layered approach.
  arXiv  Detail & Related papers  (2025-08-01T20:30:42Z)
• CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checks [46.89839054706183]
  We propose CROC: a framework for automated Contrastive Robustness Checks.<n>We generate a pseudo-labeled dataset of over one million contrastive prompt-image pairs.<n>We also use the dataset to train CROCScore, a new metric that achieves state-of-the-art performance among open-source methods.
  arXiv  Detail & Related papers  (2025-05-16T14:39:44Z)
• A document processing pipeline for the construction of a dataset for topic modeling based on the judgments of the Italian Supreme Court [5.612141846711729]
  We develop a document processing pipeline that produces an anonymized dataset optimized for topic modeling.<n>The pipeline integrates document layout analysis (YOLOv8x), optical character recognition, and text anonymization.<n>Compared to OCR-only methods, our dataset improved topic modeling with a diversity score of 0.6198 and a coherence score of 0.6638.
  arXiv  Detail & Related papers  (2025-05-13T11:06:24Z)
• INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback [80.57617091714448]
  We present InstructScore, an explainable evaluation metric for text generation. We fine-tune a text evaluation metric based on LLaMA, producing a score for generated text and a human readable diagnostic report.
  arXiv  Detail & Related papers  (2023-05-23T17:27:22Z)
• How Does Generative Retrieval Scale to Millions of Passages? [68.98628807288972]
  We conduct the first empirical study of generative retrieval techniques across various corpus scales. We scale generative retrieval to millions of passages with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. While generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge.
  arXiv  Detail & Related papers  (2023-05-19T17:33:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.