Can AI Extract Antecedent Factors of Human Trust in AI? An Application of Information Extraction for Scientific Literature in Behavioural and Computer Sciences

• URL: http://arxiv.org/abs/2412.11344v1
• Date: Mon, 16 Dec 2024 00:02:38 GMT
• Title: Can AI Extract Antecedent Factors of Human Trust in AI? An Application of Information Extraction for Scientific Literature in Behavioural and Computer Sciences
• Authors: Melanie McGrath, Harrison Bailey, Necva Bölücü, Xiang Dai, Sarvnaz Karimi, Cecile Paris,
• Abstract summary: Trust in AI is where factors contributing to human trust in AI applications are studied.<n>With the input of domain experts, we create the first annotated English dataset in this domain.<n>We benchmark it with state-of-the-art methods using large language models in named entity and relation extraction.<n>Our results indicate that this problem requires supervised learning which may not be currently feasible with prompt-based LLMs.
• Score: 9.563656421424728
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Information extraction from the scientific literature is one of the main techniques to transform unstructured knowledge hidden in the text into structured data which can then be used for decision-making in down-stream tasks. One such area is Trust in AI, where factors contributing to human trust in artificial intelligence applications are studied. The relationships of these factors with human trust in such applications are complex. We hence explore this space from the lens of information extraction where, with the input of domain experts, we carefully design annotation guidelines, create the first annotated English dataset in this domain, investigate an LLM-guided annotation, and benchmark it with state-of-the-art methods using large language models in named entity and relation extraction. Our results indicate that this problem requires supervised learning which may not be currently feasible with prompt-based LLMs.

Related papers

• Data Therapist: Eliciting Domain Knowledge from Subject Matter Experts Using Large Language Models [17.006423792670414]
  We present the Data Therapist, a web-based tool that helps domain experts externalize implicit knowledge through a mixed-initiative process. The resulting structured knowledge base can inform both human and automated visualization design.
  arXiv  Detail & Related papers  (2025-05-01T11:10:17Z)
• MoRE-LLM: Mixture of Rule Experts Guided by a Large Language Model [54.14155564592936]
  We propose a Mixture of Rule Experts guided by a Large Language Model (MoRE-LLM) MoRE-LLM steers the discovery of local rule-based surrogates during training and their utilization for the classification task. LLM is responsible for enhancing the domain knowledge alignment of the rules by correcting and contextualizing them.
  arXiv  Detail & Related papers  (2025-03-26T11:09:21Z)
• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Diagnostic Reasoning in Natural Language: Computational Model and Application [68.47402386668846]
  We investigate diagnostic abductive reasoning (DAR) in the context of language-grounded tasks (NL-DAR) We propose a novel modeling framework for NL-DAR based on Pearl's structural causal models. We use the resulting dataset to investigate the human decision-making process in NL-DAR.
  arXiv  Detail & Related papers  (2024-09-09T06:55:37Z)
• Understanding Generative AI Content with Embedding Models [4.662332573448995]
  We show that deep neural networks (DNNs) implicitly engineer features by transforming their input data into hidden feature vectors called embeddings. We find empirical evidence that there is intrinsic separability between real samples and those generated by artificial intelligence (AI)
  arXiv  Detail & Related papers  (2024-08-19T22:07:05Z)
• Knowledge AI: Fine-tuning NLP Models for Facilitating Scientific Knowledge Extraction and Understanding [0.0]
  This project investigates the efficacy of Large Language Models (LLMs) in understanding and extracting scientific knowledge across specific domains. We employ pre-trained models and fine-tune them on datasets in the scientific domain.
  arXiv  Detail & Related papers  (2024-08-04T01:32:09Z)
• From Text to Insight: Large Language Models for Materials Science Data Extraction [4.08853418443192]
  The vast majority of materials science knowledge exists in unstructured natural language.<n>Structured data is crucial for innovative and systematic materials design.<n>The advent of large language models (LLMs) represents a significant shift.
  arXiv  Detail & Related papers  (2024-07-23T22:23:47Z)
• Leveraging Large Language Models for Web Scraping [0.0]
  This research investigates a general-purpose accurate data scraping recipe for RAG models designed for language generation. To capture knowledge in a more modular and interpretable way, we use pre trained language models with a latent knowledge retriever.
  arXiv  Detail & Related papers  (2024-06-12T14:15:15Z)
• Agent-based Learning of Materials Datasets from Scientific Literature [0.0]
  We develop a chemist AI agent, powered by large language models (LLMs), to create structured datasets from natural language text. Our chemist AI agent, Eunomia, can plan and execute actions by leveraging the existing knowledge from decades of scientific research articles.
  arXiv  Detail & Related papers  (2023-12-18T20:29:58Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• Instruct and Extract: Instruction Tuning for On-Demand Information Extraction [86.29491354355356]
  On-Demand Information Extraction aims to fulfill the personalized demands of real-world users. We present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE.
  arXiv  Detail & Related papers  (2023-10-24T17:54:25Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.