Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization

• URL: http://arxiv.org/abs/2607.03704v1
• Date: Sat, 04 Jul 2026 04:49:14 GMT
• Title: Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization
• Abstract summary: We develop a GP-compatible optimization objective and investigate whether temperature optimization improves causality classification agreement.<n>Case-specific temperature selection produced meaningful improvements, supporting temperature optimization for LLM-assisted pharmacovigilance.
• Score: 0.48861336570452174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Growing individual case safety report (ICSR) volumes have intensified demand for scalable automated causality assessment. Large Language Models (LLMs) show promise, yet performance on clinically demanding tasks remains suboptimal and inference-time hyperparameter optimization has not been investigated. Objective: To develop a Gaussian Process (GP)-compatible optimization objective and investigate whether temperature optimization improves LLM-expert agreement on Naranjo causality assessment of FAERS ICSRs. Methods: Expert causality assessments were performed on 723 stratified FAERS cases. OpenAI's GPT-5.2 was evaluated using chain-of-thought (CoT) prompting. Four composite metrics were developed: Weighted Cosine Similarity (WCS), Information-Weighted Agreement Score (IWAS), Entropy-Weighted Agreement and Cosine Similarity Score (EWACS), and Consensus-Weighted Cosine Similarity (CWCS) and Bayesian optimization using a GP surrogate with Probability of Improvement (PoI) acquisition was applied across temperature [0, 2]. Results: GPT-5.2 outperformed prior biomedical LLMs at baseline (T = 0), achieving 74.1% agreement on question 5 and 65.4% on question 10 of Naranjo algorithm. Entropy analysis identified these as the sole informative optimization targets. Temperature showed no systematic population-level effect (\b{eta} = 0.002, p = 0.959). EWACS-guided Bayesian optimization improved causality classification agreement from 45.0% to 72.0% (+27 pp), with the largest gain in Doubtful cases (+42.9 pp). Conclusion: EWACS was identified as the optimal GP-compatible metric. The absence of a universal temperature optimum indicates LLM performance is driven primarily by ICSR content, yet case-specific temperature selection produced meaningful improvements, supporting temperature optimization for LLM-assisted pharmacovigilance.

Related papers

• A Physics-Informed Framework for PID Tuning of Chemical Processes Using Large Language Model Agents [0.0]
  This work formalizes a language-model-assisted PID tuning framework applicable to both large and small language models (LLMs/SLMs)<n>For local deployment, Qwen3-0.6B is adapted through supervised fine-tuning (SFT) with simulation-verified IMC targets and physics-informed group relative policy optimization (PI-GRPO) with non-compensable stability and performance rewards.
  arXiv  Detail & Related papers  (2026-07-29T08:14:31Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization [3.7323766833175718]
  MAGE (Memory-Augmented Goal Prompt Evolution) is a controlled analysis framework for studying component interaction in prompt optimization.<n>Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE), when multiple optimization signals operate within a closed reflective loop.
  arXiv  Detail & Related papers  (2026-07-11T10:05:58Z)
• SAFE ma-QAOA: Surrogate-Assisted and Fine-Tuning Enhanced Multi-Angle QAOA with Parameter Distillation [4.822952500424597]
  Multi-angle Quantum Approximate Optimization Algorithm (ma-QAOA) extends the Quantum Approximate Optimization Algorithm (QAOA)<n>We propose the Surrogate-Assisted and Fine-tuning Enhanced (SAFE) framework for gradient-based optimization.<n>We evaluate SAFE on instances of the Sherrington-Kirk model, two-dimensional square-lattice spin glass, and Max-Cut.
  arXiv  Detail & Related papers  (2026-05-22T08:43:07Z)
• Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean [51.56484100374058]
  This paper presents a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T [0.1, 1.0] to control evaluation rigor.<n>Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI.
  arXiv  Detail & Related papers  (2026-04-04T09:04:38Z)
• Adaptive Conditional Forest Sampling for Spectral Risk Optimisation under Decision-Dependent Uncertainty [1.14219428942199]
  Minimising a spectral risk objective, defined as a convex combination of expected cost and Conditional Value-at-Risk, is challenging.<n>We propose Adaptive Conditional Forest Sampling (ACFS), a four-phase simulation-optimisation framework.<n>ACFS integrates Generalised Random Forests for decision-conditional distribution approximation, CEM-guided global exploration, rank-weighted focused augmentation, and surrogate-to-oracle two-stage reranking.
  arXiv  Detail & Related papers  (2026-03-12T22:53:24Z)
• Empirical Evaluation of QAOA with Zero Noise Extrapolation on NISQ Hardware for Carbon Credit Portfolio Optimization in the Brazilian Cerrado [0.0]
  optimizing carbon credit portfolios is a critical challenge for climate mitigation.<n>This study explores the practical application of the Quantum Approximate Optimization (QAOA) combined with Zero Noise Extrapolation (ZNE) to address a multi-objective territorial planning problem.<n>We model an 88-variable portfolio optimization involving carbon sequestration, biodiversity connectivity, and social impact metrics, executed on intermediate-scale IBM Quantum hardware.
  arXiv  Detail & Related papers  (2026-02-06T18:17:28Z)
• Hierarchical Evaluation Function: A Multi-Metric Approach for Optimizing Demand Forecasting Models [0.479839492673697]
  The Hierarchical Evaluation Function (HEF) is proposed as a multi-metric framework for hyperparameter optimization.<n>HEF integrates explanatory power (R2), sensitivity to extreme errors (RMSE), and average accuracy (MAE)<n>The performance of HEF was assessed using four widely recognized benchmark datasets in the forecasting domain.
  arXiv  Detail & Related papers  (2025-08-18T16:25:49Z)
• ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstruction [25.85736569130897]
  Pairwise evaluation of large language models (LLMs) has become the dominant paradigm for benchmarking open-ended tasks.<n>We show that this critical issue stems largely from low-quality data that contains inherently ambiguous preference pairs.<n>We propose ELSPR, a principled graph-theoretic framework that models pairwise preferences as tournament graphs.
  arXiv  Detail & Related papers  (2025-05-23T10:00:03Z)
• Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data [51.62162460809116]
  We introduce Dynamic Noise Preference Optimization (DNPO) to ensure consistent improvements across iterations.<n>In experiments with Zephyr-7B, DNPO consistently outperforms existing methods, showing an average performance boost of 2.6%.<n> DNPO shows a significant improvement in model-generated data quality, with a 29.4% win-loss rate gap compared to the baseline in GPT-4 evaluations.
  arXiv  Detail & Related papers  (2025-02-08T01:20:09Z)
• On the Convergence of DP-SGD with Adaptive Clipping [56.24689348875711]
  Gradient Descent with gradient clipping is a powerful technique for enabling differentially private optimization.<n>This paper provides the first comprehensive convergence analysis of SGD with quantile clipping (QC-SGD)<n>We show how QC-SGD suffers from a bias problem similar to constant-threshold clipped SGD but can be mitigated through a carefully designed quantile and step size schedule.
  arXiv  Detail & Related papers  (2024-12-27T20:29:47Z)
• EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation [58.546205554954454]
  We propose Enhancing Alignment in MLLMs via Critical Observation (EACO)<n>EACO aligns MLLMs by self-generated preference data using only 5k images economically.<n>EACO reduces the overall hallucinations by 65.6% on HallusionBench and improves the reasoning ability by 21.8% on MME-Cognition.
  arXiv  Detail & Related papers  (2024-12-06T09:59:47Z)
• Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence [66.83161885378192]
  Area under ROC (AUROC) and precision-recall curves (AUPRC) are common metrics for evaluating classification performance for imbalanced problems. We propose a technical method to optimize AUPRC for deep learning.
  arXiv  Detail & Related papers  (2021-04-18T06:22:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.