The Science Data Lake: A Unified Open Infrastructure Integrating 293 Million Papers Across Eight Scholarly Sources with Embedding-Based Ontology Alignment

• URL: http://arxiv.org/abs/2603.03126v1
• Date: Tue, 03 Mar 2026 15:58:18 GMT
• Title: The Science Data Lake: A Unified Open Infrastructure Integrating 293 Million Papers Across Eight Scholarly Sources with Embedding-Based Ontology Alignment
• Abstract summary: We present the Science Data Lake, a locally-deployable infrastructure built on DuckDB and Parquet files.<n>The resource comprises approximately 960GB of Parquet files spanning 293 million uniquely identifiable papers.<n>The resource is open source, deployable remotely via HuggingFace, and includes structured documentation suitable for large language model (LLM) based research agents.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scholarly data are largely fragmented across siloed databases with divergent metadata and missing linkages among them. We present the Science Data Lake, a locally-deployable infrastructure built on DuckDB and simple Parquet files that unifies eight open sources - Semantic Scholar, OpenAlex, SciSciNet, Papers with Code, Retraction Watch, Reliance on Science, a preprint-to-published mapping, and Crossref - via DOI normalization while preserving source-level schemas. The resource comprises approximately 960GB of Parquet files spanning ~293 million uniquely identifiable papers across ~22 schemas and ~153 SQL views. An embedding-based ontology alignment using BGE-large sentence embeddings maps 4,516 OpenAlex topics to 13 scientific ontologies (~1.3 million terms), yielding 16,150 mappings covering 99.8% of topics ($\geq 0.65$ threshold) with $F1 = 0.77$ at the recommended $\geq 0.85$ operating point, outperforming TF-IDF, BM25, and Jaro-Winkler baselines on a 300-pair gold-standard evaluation. We validate through 10 automated checks, cross-source citation agreement analysis (pairwise Pearson $r = 0.76$ - $0.87$), and stratified manual annotation. Four vignettes demonstrate cross-source analyses infeasible with any single database. The resource is open source, deployable on a single drive or queryable remotely via HuggingFace, and includes structured documentation suitable for large language model (LLM) based research agents.

Related papers

• Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction [4.062546658033461]
  We introduce Doc2DB-Bench, a benchmark for Document-to-Database construction.<n>It contains 203 long-document instances across 42 schemas and seven domain groups, with 117 entity tables, 132 relationship tables, 7,341 rows, and 41,935 cells.
  arXiv  Detail & Related papers  (2026-08-09T03:58:39Z)
• GPTKB 2.0: Browsing, Querying, and Auditing a Disambiguated LLM-Derived Knowledge Base [11.119171720690238]
  We present a web demo for exploring a large-scale disambiguated knowledge base (KB) materialized from a large language model (LLM)<n>GPTKB 2.0 contains 38.4M triples over 1.6M canonical entities, together with 207.6K consolidated relations and 66K consolidated classes.<n>The interface further supports structured SPARQL queries, natural-language questions translated to SPARQL, and entity linking from user-provided text to canonical GPTKB 2.0 entries.
  arXiv  Detail & Related papers  (2026-08-07T09:10:30Z)
• Lacuna: A Research Map for Machine Learning [49.4551062599854]
  Lacuna is a research map for machine learning that uses LLMs to turn papers and scholarly metadata into markdown summaries.<n>Across LitSearch, Multi-XScience-CS/ML, and ScholarQA-CS-ML, Lacuna outperforms OpenScholar with the strongest gains on LitSearch retrieval.<n>We also evaluate Lacuna Deep Research, a multi-stage report agent over the map, on 25 ReportBench-ML survey tasks.
  arXiv  Detail & Related papers  (2026-06-24T18:00:12Z)
• DocAtlas: Multilingual Document Understanding Across 80+ Languages [58.715440331861295]
  We introduce DocAtlas, a framework that constructs high-fidelity OCR datasets and benchmarks covering 82 languages.<n>Our dual pipelines, differential rendering of native DOCX documents and synthetic-based generation for right-to-left scripts produce precise structural annotations.
  arXiv  Detail & Related papers  (2026-05-12T18:09:38Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• ClaimFlow: Tracing the Evolution of Scientific Claims in NLP [67.23189226608389]
  We introduce $texttClaimFlow$, a claim-centric view of the NLP literature.<n>We define a new task $-$ $textitClaim Relation Classification$ $-$ which requires models to infer the scientific stance toward a cited claim.<n>Our analysis reveals that $63.5$% claims are never reused; only $11.1$% are ever challenged.
  arXiv  Detail & Related papers  (2026-03-17T02:43:36Z)
• Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database [0.0]
  We present three open-source biomedical knowledge graphs -- Pathways KG, Clinical Trials KG, and Drug Interactions KG.<n>Our contributions are threefold. First, we describe a high-performance graph for constructing large-scale KGs from heterogeneous public data sources.<n>Second, we demonstrate cross-KG federation: loading all three snapshots into a single graph tenant enables property-based joins.
  arXiv  Detail & Related papers  (2026-03-16T10:36:13Z)
• MoDora: Tree-Based Semi-Structured Document Analysis System [62.01015188258797]
  Semi-structured documents integrate diverse interleaved data elements arranged in various and often irregular layouts.<n>MoDora is an LLM-powered system for semi-structured document analysis.<n> Experiments show MoDora outperforms baselines by 5.97%-61.07% in accuracy.
  arXiv  Detail & Related papers  (2026-02-26T14:48:49Z)
• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction [11.272127170490059]
  No end-to-end benchmark evaluates PDF-to-JSON extraction under enterprise-scale schema breadth.<n>No principled methodology captures the semantics of nested extraction.<n> ExtractBench is an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction.
  arXiv  Detail & Related papers  (2026-02-12T18:31:37Z)
• Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNs [7.158582263989492]
  We build paired citation graphs, ground truth and GPT-4o-generated (from parametric knowledge) for 10,000 papers from SciSciNet.<n>We show that GPT barely separates from ground truth (RF accuracy $approx$ 0.60) despite cleanly rejecting the random baseline.<n> embeddings sharply increase separability: RF on aggregated embeddings reaches $approx$ 0.83, and GNNs with embedding node features achieve 93% test accuracy on GPT vs. ground truth.
  arXiv  Detail & Related papers  (2026-01-28T15:37:31Z)
• LLM-Guided Probabilistic Fusion for Label-Efficient Document Layout Analysis [6.908972852063454]
  Document layout understanding remains data-intensive despite advances in semi-supervised learning.<n>We present a framework that enhances semi-supervised detection by fusing visual predictions with structural priors.<n>Our method demonstrates consistent gains across model scales.
  arXiv  Detail & Related papers  (2025-11-12T02:25:58Z)
• HySemRAG: A Hybrid Semantic Retrieval-Augmented Generation Framework for Automated Literature Synthesis and Methodological Gap Analysis [55.2480439325792]
  HySemRAG is a framework that combines Extract, Transform, Load (ETL) pipelines with Retrieval-Augmented Generation (RAG)<n>System addresses limitations in existing RAG architectures through a multi-layered approach.
  arXiv  Detail & Related papers  (2025-08-01T20:30:42Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Text-Driven Neural Collaborative Filtering Model for Paper Source Tracing [1.124958340749622]
  The Paper Source Tracing (PST) task seeks to automate the identification of pivotal references for given scholarly articles. This framework employs the Neural Collaborative Filtering (NCF) model to generate final predictions. Our method achieved a score of 0.37814 on the Mean Average Precision (MAP) metric, outperforming baseline models and ranking 11th among all participating teams.
  arXiv  Detail & Related papers  (2024-07-25T02:48:56Z)
• $\texttt{MixGR}$: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularity [88.78750571970232]
  This paper introduces $texttMixGR$, which improves dense retrievers' awareness of query-document matching. $texttMixGR$ fuses various metrics based on granularities to a united score that reflects a comprehensive query-document similarity.
  arXiv  Detail & Related papers  (2024-07-15T13:04:09Z)
• DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models [63.466265039007816]
  We present DocGenome, a structured document benchmark constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community. We conduct extensive experiments to demonstrate the advantages of DocGenome and objectively evaluate the performance of large models on our benchmark.
  arXiv  Detail & Related papers  (2024-06-17T15:13:52Z)
• SemOpenAlex: The Scientific Landscape in 26 Billion RDF Triples [0.0]
  SemOpenAlex is an extensive RDF knowledge graph that contains over 26 billion triples about scientific publications and their associated entities. We offer the data through multiple channels, including RDF dump files, a SPARQL endpoint, and as a data source in the Linked Open Data cloud.
  arXiv  Detail & Related papers  (2023-08-07T15:46:39Z)
• Document Layout Annotation: Database and Benchmark in the Domain of Public Affairs [62.38140271294419]
  We propose a procedure to semi-automatically annotate digital documents with different layout labels. We collect a novel database for DLA in the public affairs domain using a set of 24 data sources from the Spanish Administration. The results of our experiments validate the proposed text labeling procedure with accuracy up to 99%.
  arXiv  Detail & Related papers  (2023-06-12T08:21:50Z)
• Multimodal Tree Decoder for Table of Contents Extraction in Document Images [32.46909366312659]
  Table of contents (ToC) extraction aims to extract headings of different levels in documents to better understand the outline of the contents. We first introduce a standard dataset, HierDoc, including image samples from 650 documents of scientific papers with their content labels. We propose a novel end-to-end model by using the multimodal tree decoder (MTD) for ToC as a benchmark for HierDoc.
  arXiv  Detail & Related papers  (2022-12-06T11:38:31Z)
• Exploring and Evaluating Attributes, Values, and Structures for Entity Alignment [100.19568734815732]
  Entity alignment (EA) aims at building a unified Knowledge Graph (KG) of rich content by linking the equivalent entities from various KGs. attribute triples can also provide crucial alignment signal but have not been well explored yet. We propose to utilize an attributed value encoder and partition the KG into subgraphs to model the various types of attribute triples efficiently.
  arXiv  Detail & Related papers  (2020-10-07T08:03:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.