Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?

• URL: http://arxiv.org/abs/2407.12626v1
• Date: Wed, 17 Jul 2024 14:52:46 GMT
• Title: Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?
• Authors: Aman Sinha, Timothee Mickus, Marianne Clausel, Mathieu Constant, Xavier Coubez,
• Abstract summary: We discuss how domain specificity and uncertainty awareness can be combined to produce reasonable estimates of a model's own uncertainty. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.
• Score: 4.741884506444161
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in-chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.

Related papers

• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors [61.92704516732144]
  We show that the most robust features for correctness prediction are those that play a distinctive causal role in the model's behavior.<n>We propose two methods that leverage causal mechanisms to predict the correctness of model outputs.
  arXiv  Detail & Related papers  (2025-05-17T00:31:39Z)
• A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation [0.0]
  Advancements in image segmentation play an integral role within the greater scope of Deep Learning-based computer vision. Uncertainty quantification has been extensively studied within this context, enabling expression of model ignorance (epistemic uncertainty) or data ambiguity (aleatoric uncertainty) to prevent uninformed decision making. This work provides a comprehensive overview of probabilistic segmentation by discussing fundamental concepts in uncertainty that govern advancements in the field and the application to various tasks.
  arXiv  Detail & Related papers  (2024-11-25T13:26:09Z)
• Verbalized Probabilistic Graphical Modeling [8.524824578426962]
  We propose Verbalized Probabilistic Graphical Modeling (vPGM) to simulate key principles of Probabilistic Graphical Models (PGMs) in natural language. vPGM bypasses expert-driven model design, making it well-suited for scenarios with limited assumptions or scarce data. Our results indicate that the model effectively enhances confidence calibration and text generation quality.
  arXiv  Detail & Related papers  (2024-06-08T16:35:31Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Determining Domain of Machine Learning Models using Kernel Density Estimates: Applications in Materials Property Prediction [1.8551396341435895]
  We develop a new approach of assessing model domain using kernel density estimation. We show that chemical groups considered unrelated based on established chemical knowledge exhibit significant dissimilarities by our measure. High measures of dissimilarity are associated with poor model performance and poor estimates of model uncertainty.
  arXiv  Detail & Related papers  (2024-05-28T15:41:16Z)
• On the Out of Distribution Robustness of Foundation Models in Medical Image Segmentation [47.95611203419802]
  Foundations for vision and language, pre-trained on extensive sets of natural image and text data, have emerged as a promising approach. We compare the generalization performance to unseen domains of various pre-trained models after being fine-tuned on the same in-distribution dataset. We further developed a new Bayesian uncertainty estimation for frozen models and used them as an indicator to characterize the model's performance on out-of-distribution data.
  arXiv  Detail & Related papers  (2023-11-18T14:52:10Z)
• On the Interventional Kullback-Leibler Divergence [11.57430292133273]
  We introduce the Interventional Kullback-Leibler divergence to quantify both structural and distributional differences between causal models. We propose a sufficient condition on the intervention targets to identify subsets of observed variables on which the models provably agree or disagree.
  arXiv  Detail & Related papers  (2023-02-10T17:03:29Z)
• Multi-modal multi-objective model-based genetic programming to find multiple diverse high-quality models [0.0]
  Genetic programming (GP) is often cited as being uniquely well-suited to contribute to Explainable artificial intelligence (XAI) In this paper, we achieve exactly this with a novel multi-modal multi-tree multi-objective GP approach that extends a modern model-based GP algorithm known as GP-GOMEA.
  arXiv  Detail & Related papers  (2022-03-24T21:35:07Z)
• Uncertainty Modeling for Out-of-Distribution Generalization [56.957731893992495]
  We argue that the feature statistics can be properly manipulated to improve the generalization ability of deep learning models. Common methods often consider the feature statistics as deterministic values measured from the learned features. We improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training.
  arXiv  Detail & Related papers  (2022-02-08T16:09:12Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples [84.8370546614042]
  Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data. Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model. Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
  arXiv  Detail & Related papers  (2020-03-25T11:08:56Z)
• Decision-Making with Auto-Encoding Variational Bayes [71.44735417472043]
  We show that a posterior approximation distinct from the variational distribution should be used for making decisions. Motivated by these theoretical results, we propose learning several approximate proposals for the best model. In addition to toy examples, we present a full-fledged case study of single-cell RNA sequencing.
  arXiv  Detail & Related papers  (2020-02-17T19:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.