ACL-rlg: A Dataset for Reading List Generation

• URL: http://arxiv.org/abs/2502.15692v1
• Date: Mon, 30 Dec 2024 07:48:32 GMT
• Title: ACL-rlg: A Dataset for Reading List Generation
• Authors: Julien Aubert-Béduchaud, Florian Boudin, Béatrice Daille, Richard Dufour,
• Abstract summary: We introduce ACL-rlg, the largest open expert-annotated reading list dataset.<n>Traditional scholarly search engines and indexing methods perform poorly on this task.
• Score: 8.526112833986183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Familiarizing oneself with a new scientific field and its existing literature can be daunting due to the large amount of available articles. Curated lists of academic references, or reading lists, compiled by experts, offer a structured way to gain a comprehensive overview of a domain or a specific scientific challenge. In this work, we introduce ACL-rlg, the largest open expert-annotated reading list dataset. We also provide multiple baselines for evaluating reading list generation and formally define it as a retrieval task. Our qualitative study highlights the fact that traditional scholarly search engines and indexing methods perform poorly on this task, and GPT-4o, despite showing better results, exhibits signs of potential data contamination.

Related papers

• Can LLMs Generate Tabular Summaries of Science Papers? Rethinking the Evaluation Protocol [83.90769864167301]
  Literature review tables are essential for summarizing and comparing collections of scientific papers. We explore the task of generating tables that best fulfill a user's informational needs given a collection of scientific papers. Our contributions focus on three key challenges encountered in real-world use: (i) User prompts are often under-specified; (ii) Retrieved candidate papers frequently contain irrelevant content; and (iii) Task evaluation should move beyond shallow text similarity techniques.
  arXiv  Detail & Related papers  (2025-04-14T14:52:28Z)
• SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers [20.273439120429025]
  SciDQA is a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles. Unlike other scientific QA datasets, SciDQA sources questions from peer reviews by domain experts and answers by paper authors. Questions in SciDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials.
  arXiv  Detail & Related papers  (2024-11-08T05:28:22Z)
• SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents [49.54155332262579]
  We release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations.
  arXiv  Detail & Related papers  (2024-10-28T15:56:49Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Knowledge Navigator: LLM-guided Browsing Framework for Exploratory Search in Scientific Literature [48.572336666741194]
  We present Knowledge Navigator, a system designed to enhance exploratory search abilities. It organizes retrieved documents into a navigable, two-level hierarchy of named and descriptive scientific topics and subtopics.
  arXiv  Detail & Related papers  (2024-08-28T14:48:37Z)
• SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers [43.18330795060871]
  SPIQA is a dataset specifically designed to interpret complex figures and tables within the context of scientific research articles.<n>We employ automatic and manual curation to create the dataset.<n> SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits.
  arXiv  Detail & Related papers  (2024-07-12T16:37:59Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• Enriched BERT Embeddings for Scholarly Publication Classification [0.13654846342364302]
  The NSLP 2024 FoRC Task I addresses this challenge organized as a competition. The goal is to develop a classifier capable of predicting one of 123 predefined classes from the Open Research Knowledge Graph (ORKG) taxonomy of research fields for a given article.
  arXiv  Detail & Related papers  (2024-05-07T09:05:20Z)
• LitLLM: A Toolkit for Scientific Literature Review [15.785989492351684]
  We propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles. Our system first initiates a web search to retrieve relevant papers. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Third, the related work section is generated based on the re-ranked results and the abstract.
  arXiv  Detail & Related papers  (2024-02-02T02:41:28Z)
• ACL-Fig: A Dataset for Scientific Figure Classification [15.241086410108512]
  We develop a pipeline that extracts figures and tables from the scientific literature and a deep-learning-based framework that classifies scientific figures using visual features. We build the first large-scale automatically annotated corpus, ACL-Fig, consisting of 112,052 scientific figures extracted from 56K research papers in the ACL Anthology. The ACL-Fig-Pilot dataset contains 1,671 manually labeled scientific figures belonging to 19 categories.
  arXiv  Detail & Related papers  (2023-01-28T20:27:35Z)
• Tell Me How to Survey: Literature Review Made Simple with Automatic Reading Path Generation [16.07200776251764]
  How to glean papers worth reading from the massive literature to do a quick survey or keep up with the latest advancement about a specific research topic has become a challenging task. Existing academic search engines such as Google Scholar return relevant papers by individually calculating the relevance between each paper and query. We introduce Reading Path Generation (RPG) which aims at automatically producing a path of papers to read for a given query.
  arXiv  Detail & Related papers  (2021-10-12T20:58:46Z)
• CitationIE: Leveraging the Citation Graph for Scientific Information Extraction [89.33938657493765]
  We use the citation graph of referential links between citing and cited papers. We observe a sizable improvement in end-to-end information extraction over the state-of-the-art.
  arXiv  Detail & Related papers  (2021-06-03T03:00:12Z)
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension [53.037401638264235]
  We present an evaluation server, ORB, that reports performance on seven diverse reading comprehension datasets. The evaluation server places no restrictions on how models are trained, so it is a suitable test bed for exploring training paradigms and representation learning.
  arXiv  Detail & Related papers  (2019-12-29T07:27:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.