An overview of model uncertainty and variability in LLM-based sentiment analysis. Challenges, mitigation strategies and the role of explainability

• URL: http://arxiv.org/abs/2504.04462v1
• Date: Sun, 06 Apr 2025 12:20:39 GMT
• Title: An overview of model uncertainty and variability in LLM-based sentiment analysis. Challenges, mitigation strategies and the role of explainability
• Authors: David Herrera-Poyatos, Carlos Peláez-González, Cristina Zuheros, Andrés Herrera-Poyatos, Virilo Tejedor, Francisco Herrera, Rosana Montes,
• Abstract summary: This paper systematically explores the Model Variability Problem (MVP) in large language models (LLMs)<n>MVP is characterized by inconsistent sentiment polarization, uncertainty arising from inference mechanisms, prompt sensitivity, and biases in training data.<n>This study helps develop more reliable, explainable, and robust sentiment analysis models, facilitating their deployment in high-stakes domains such as finance, healthcare, and policymaking.
• Score: 6.791108304863664
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have significantly advanced sentiment analysis, yet their inherent uncertainty and variability pose critical challenges to achieving reliable and consistent outcomes. This paper systematically explores the Model Variability Problem (MVP) in LLM-based sentiment analysis, characterized by inconsistent sentiment classification, polarization, and uncertainty arising from stochastic inference mechanisms, prompt sensitivity, and biases in training data. We analyze the core causes of MVP, presenting illustrative examples and a case study to highlight its impact. In addition, we investigate key challenges and mitigation strategies, paying particular attention to the role of temperature as a driver of output randomness and emphasizing the crucial role of explainability in improving transparency and user trust. By providing a structured perspective on stability, reproducibility, and trustworthiness, this study helps develop more reliable, explainable, and robust sentiment analysis models, facilitating their deployment in high-stakes domains such as finance, healthcare, and policymaking, among others.

Related papers

• Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models [13.216398753024182]
  Large Language Models (LLMs) and Vision-Language Models (VLMs) have become essential to general artificial intelligence. We propose a novel stability measure for LLMs inspired by statistical methods rooted in information geometry. Our results demonstrate the utility of our measure in identifying salient parameters and detecting vulnerable regions in input images or critical dimensions in token embeddings.
  arXiv  Detail & Related papers  (2025-03-28T16:23:59Z)
• Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions [8.069858557211132]
  Large Language Models (LLMs) have shown remarkable capabilities across various tasks, but their deployment in high-stake domains requires consistent performance across multiple interaction rounds.<n>This paper introduces a comprehensive framework for evaluating and improving LLM response consistency, making three key contributions.
  arXiv  Detail & Related papers  (2025-03-28T11:49:56Z)
• REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Models [59.445672459851274]
  REVAL is a comprehensive benchmark designed to evaluate the textbfREliability and textbfVALue of Large Vision-Language Models.<n>REVAL encompasses over 144K image-text Visual Question Answering (VQA) samples, structured into two primary sections: Reliability and Values.<n>We evaluate 26 models, including mainstream open-source LVLMs and prominent closed-source models like GPT-4o and Gemini-1.5-Pro.
  arXiv  Detail & Related papers  (2025-03-20T07:54:35Z)
• Identifying Trustworthiness Challenges in Deep Learning Models for Continental-Scale Water Quality Prediction [64.4881275941927]
  We present the first comprehensive evaluation of trustworthiness in a continental-scale multi-task LSTM model.<n>Our investigation uncovers systematic patterns of model performance disparities linked to basin characteristics.<n>This work serves as a timely call to action for advancing trustworthy data-driven methods for water resources management.
  arXiv  Detail & Related papers  (2025-03-13T01:50:50Z)
• Fine-Grained Bias Detection in LLM: Enhancing detection mechanisms for nuanced biases [0.0]
  This study presents a detection framework to identify nuanced biases in Large Language Models (LLMs)<n>The approach integrates contextual analysis, interpretability via attention mechanisms, and counterfactual data augmentation to capture hidden biases.<n>Results show improvements in detecting subtle biases compared to conventional methods.
  arXiv  Detail & Related papers  (2025-03-08T04:43:01Z)
• An LLM-based Delphi Study to Predict GenAI Evolution [0.6138671548064356]
  This study introduces a novel approach to qualitative forecasting by leveraging Large Language Models.<n>The methodology was applied to explore the future evolution of Generative Artificial Intelligence.
  arXiv  Detail & Related papers  (2025-02-28T14:31:25Z)
• An Overview of Large Language Models for Statisticians [109.38601458831545]
  Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence (AI)<n>This paper explores potential areas where statisticians can make important contributions to the development of LLMs.<n>We focus on issues such as uncertainty quantification, interpretability, fairness, privacy, watermarking and model adaptation.
  arXiv  Detail & Related papers  (2025-02-25T03:40:36Z)
• Investigating the Impact of LLM Personality on Cognitive Bias Manifestation in Automated Decision-Making Tasks [4.65004369765875]
  Personality traits play a crucial role in either amplifying or reducing biases.<n>Conscientiousness and Agreeableness may generally enhance the efficacy of bias mitigation strategies.
  arXiv  Detail & Related papers  (2025-02-20T03:15:54Z)
• Bridging Interpretability and Robustness Using LIME-Guided Model Refinement [0.0]
  Local Interpretable Model-Agnostic Explanations (LIME) systematically enhance model robustness.<n> Empirical evaluations on multiple benchmark datasets demonstrate that LIME-guided refinement not only improves interpretability but also significantly enhances resistance to adversarial perturbations and generalization to out-of-distribution data.
  arXiv  Detail & Related papers  (2024-12-25T17:32:45Z)
• Fairness Increases Adversarial Vulnerability [50.90773979394264]
  This paper shows the existence of a dichotomy between fairness and robustness, and analyzes when achieving fairness decreases the model robustness to adversarial samples. Experiments on non-linear models and different architectures validate the theoretical findings in multiple vision domains. The paper proposes a simple, yet effective, solution to construct models achieving good tradeoffs between fairness and robustness.
  arXiv  Detail & Related papers  (2022-11-21T19:55:35Z)
• Causal Fairness Analysis [68.12191782657437]
  We introduce a framework for understanding, modeling, and possibly solving issues of fairness in decision-making settings. The main insight of our approach will be to link the quantification of the disparities present on the observed data with the underlying, and often unobserved, collection of causal mechanisms. Our effort culminates in the Fairness Map, which is the first systematic attempt to organize and explain the relationship between different criteria found in the literature.
  arXiv  Detail & Related papers  (2022-07-23T01:06:34Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.