Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models

• URL: http://arxiv.org/abs/2302.04914v3
• Date: Wed, 12 Jun 2024 14:25:15 GMT
• Title: Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models
• Authors: Maciej P. Polak, Shrey Modi, Anna Latosinska, Jinming Zhang, Ching-Wen Wang, Shaonan Wang, Ayan Deep Hazra, Dane Morgan,
• Abstract summary: Large language models (LLMs) are transforming the way humans interact with text. We demonstrate a simple and efficient method for extracting materials data from full-text research papers. This approach requires minimal to no coding or prior knowledge about the extracted property. It offers high recall and nearly perfect precision in the resulting database.
• Score: 5.748877272090607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate and comprehensive material databases extracted from research papers are crucial for materials science and engineering, but their development requires significant human effort. With large language models (LLMs) transforming the way humans interact with text, LLMs provide an opportunity to revolutionize data extraction. In this study, we demonstrate a simple and efficient method for extracting materials data from full-text research papers leveraging the capabilities of LLMs combined with human supervision. This approach is particularly suitable for mid-sized databases and requires minimal to no coding or prior knowledge about the extracted property. It offers high recall and nearly perfect precision in the resulting database. The method is easily adaptable to new and superior language models, ensuring continued utility. We show this by evaluating and comparing its performance on GPT-3 and GPT-3.5/4 (which underlie ChatGPT), as well as free alternatives such as BART and DeBERTaV3. We provide a detailed analysis of the method's performance in extracting sentences containing bulk modulus data, achieving up to 90% precision at 96% recall, depending on the amount of human effort involved. We further demonstrate the method's broader effectiveness by developing a database of critical cooling rates for metallic glasses over twice the size of previous human curated databases.

Related papers

• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Enhancing SLM via ChatGPT and Dataset Augmentation [0.3844771221441211]
  We employ knowledge distillation-based techniques and synthetic dataset augmentation to bridge the performance gap between large language models (LLMs) and small language models (SLMs) Our methods involve two forms of rationale generation--information extraction and informed reasoning--to enrich the ANLI dataset. Our findings reveal that the incorporation of synthetic rationales significantly improves the model's ability to comprehend natural language, leading to 1.3% and 2.3% higher classification accuracy, respectively, on the ANLI dataset.
  arXiv  Detail & Related papers  (2024-09-19T09:24:36Z)
• AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning [93.96463520716759]
  Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and hallucinations. Here, we introduce AvaTaR, a novel and automated framework that optimize an LLM agent to effectively leverage provided tools, improving performance on a given task.
  arXiv  Detail & Related papers  (2024-06-17T04:20:02Z)
• Leveraging Large Language Models for Web Scraping [0.0]
  This research investigates a general-purpose accurate data scraping recipe for RAG models designed for language generation. To capture knowledge in a more modular and interpretable way, we use pre trained language models with a latent knowledge retriever.
  arXiv  Detail & Related papers  (2024-06-12T14:15:15Z)
• Aligning Large Language Models with Self-generated Preference Data [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models (LLMs) with human preferences. Our key idea is leveraging the human prior knowledge within the small (seed) data. We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• Dynamic In-context Learning with Conversational Models for Data Extraction and Materials Property Prediction [0.0]
  PropertyExtractor is an open-source tool that blends zero-shot with few-shot in-context learning. Our tests on material data demonstrate precision and recall that exceed 95% with an error rate of approximately 9%.
  arXiv  Detail & Related papers  (2024-05-16T21:15:51Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models [59.89454513692417]
  Tabular data is often hidden in text, particularly in medical diagnostic reports. We propose a novel, simple, and effective methodology for extracting structured tabular data from textual medical reports, called TEMED-LLM. We demonstrate that our approach significantly outperforms state-of-the-art text classification models in medical diagnostics.
  arXiv  Detail & Related papers  (2023-06-08T09:12:28Z)
• Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering [0.0]
  ChatExtract can fully automate very accurate data extraction with minimal initial effort and background. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs.
  arXiv  Detail & Related papers  (2023-03-07T17:54:53Z)
• Bag of Tricks for Training Data Extraction from Language Models [98.40637430115204]
  We investigate and benchmark tricks for improving training data extraction using a publicly available dataset. The experimental results show that several previously overlooked tricks can be crucial to the success of training data extraction.
  arXiv  Detail & Related papers  (2023-02-09T06:46:42Z)
• Utilizing Domain Knowledge: Robust Machine Learning for Building Energy Prediction with Small, Inconsistent Datasets [1.1081836812143175]
  The demand for a huge amount of data for machine learning (ML) applications is currently a bottleneck. We propose a method to combine prior knowledge with data-driven methods to significantly reduce their data dependency. CBML as the knowledge-encoded data-driven method is examined in the context of energy-efficient building engineering.
  arXiv  Detail & Related papers  (2023-01-23T08:56:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.