Self-Supervised Learning for Visual Summary Identification in Scientific Publications

• URL: http://arxiv.org/abs/2012.11213v2
• Date: Thu, 14 Jan 2021 09:00:18 GMT
• Title: Self-Supervised Learning for Visual Summary Identification in Scientific Publications
• Authors: Shintaro Yamamoto, Anne Lauscher, Simone Paolo Ponzetto, Goran Glava\v{s}, Shigeo Morishima
• Abstract summary: We create a new benchmark dataset for selecting figures to serve as visual summaries of publications based on their abstracts. We develop a self-supervised learning approach, based on matching of inline references to figures with figure captions. Experiments in both biomedical and computer science domains show that our model is able to outperform the state of the art.
• Score: 21.26121265868308
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Providing visual summaries of scientific publications can increase information access for readers and thereby help deal with the exponential growth in the number of scientific publications. Nonetheless, efforts in providing visual publication summaries have been few and far apart, primarily focusing on the biomedical domain. This is primarily because of the limited availability of annotated gold standards, which hampers the application of robust and high-performing supervised learning techniques. To address these problems we create a new benchmark dataset for selecting figures to serve as visual summaries of publications based on their abstracts, covering several domains in computer science. Moreover, we develop a self-supervised learning approach, based on heuristic matching of inline references to figures with figure captions. Experiments in both biomedical and computer science domains show that our model is able to outperform the state of the art despite being self-supervised and therefore not relying on any annotated training data.

Related papers

• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  This dataset includes figures such as schematic diagrams, simulated images, macroscopic/microscopic photos, and experimental visualizations. We developed benchmarks for scientific figure captioning and multiple-choice questions, evaluating six proprietary and over ten open-source models. The dataset and benchmarks will be released to support further research.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• Enhancing Scientific Figure Captioning Through Cross-modal Learning [0.0]
  The volume and diversity of scientific research data have surged, leading to an increase in the number and variety of charts. This paper presents a novel approach to scientific chart title generation, demonstrating its effectiveness in improving the clarity and accessibility of research data.
  arXiv  Detail & Related papers  (2024-06-24T18:08:19Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [80.49349719239584]
  We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following demonstrations for 54 tasks. SciRIFF is the first dataset focused on extracting and synthesizing information from research literature across a wide range of scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval [64.03631654052445]
  Current benchmarks for evaluating MMIR performance in image-text pairing within the scientific domain show a notable gap. We develop a specialised scientific MMIR benchmark by leveraging open-access paper collections. This benchmark comprises 530K meticulously curated image-text pairs, extracted from figures and tables with detailed captions in scientific documents.
  arXiv  Detail & Related papers  (2024-01-24T14:23:12Z)
• CARE: Extracting Experimental Findings From Clinical Literature [29.763929941107616]
  This work presents CARE, a new IE dataset for the task of extracting clinical findings. We develop a new annotation schema capturing fine-grained findings as n-ary relations between entities and attributes. We collect extensive annotations for 700 abstracts from two sources: clinical trials and case reports.
  arXiv  Detail & Related papers  (2023-11-16T10:06:19Z)
• Towards Graph Foundation Models: A Survey and Beyond [66.37994863159861]
  Foundation models have emerged as critical components in a variety of artificial intelligence applications. The capabilities of foundation models to generalize and adapt motivate graph machine learning researchers to discuss the potential of developing a new graph learning paradigm. This article introduces the concept of Graph Foundation Models (GFMs), and offers an exhaustive explanation of their key characteristics and underlying technologies.
  arXiv  Detail & Related papers  (2023-10-18T09:31:21Z)
• State of the Art on Diffusion Models for Visual Computing [191.6168813012954]
  This report introduces the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model. We also give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing. We discuss available datasets, metrics, open challenges, and social implications.
  arXiv  Detail & Related papers  (2023-10-11T05:32:29Z)
• CitationIE: Leveraging the Citation Graph for Scientific Information Extraction [89.33938657493765]
  We use the citation graph of referential links between citing and cited papers. We observe a sizable improvement in end-to-end information extraction over the state-of-the-art.
  arXiv  Detail & Related papers  (2021-06-03T03:00:12Z)
• Semantic and Relational Spaces in Science of Science: Deep Learning Models for Article Vectorisation [4.178929174617172]
  We focus on document-level embeddings based on the semantic and relational aspects of articles, using Natural Language Processing (NLP) and Graph Neural Networks (GNNs) Our results show that using NLP we can encode a semantic space of articles, while with GNN we are able to build a relational space where the social practices of a research community are also encoded.
  arXiv  Detail & Related papers  (2020-11-05T14:57:41Z)
• Generating Knowledge Graphs by Employing Natural Language Processing and Machine Learning Techniques within the Scholarly Domain [1.9004296236396943]
  We present a new architecture that takes advantage of Natural Language Processing and Machine Learning methods for extracting entities and relationships from research publications. Within this research work, we i) tackle the challenge of knowledge extraction by employing several state-of-the-art Natural Language Processing and Text Mining tools. We generated a scientific knowledge graph including 109,105 triples, extracted from 26,827 abstracts of papers within the Semantic Web domain.
  arXiv  Detail & Related papers  (2020-10-28T08:31:40Z)
• Visual Exploration and Knowledge Discovery from Biomedical Dark Data [0.0]
  We employ a natural language processing based pipeline to discover knowledge out of the biomedical dark data. We aim to proffer a potential solution to overcome the problem of analyzing overwhelming amounts of information.
  arXiv  Detail & Related papers  (2020-09-28T04:27:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.