Evaluating LLM Usage for Efficient and Explainable Numerical and Classified Implicit Sentiment Analysis of Product Desirability

• URL: http://arxiv.org/abs/2606.23701v1
• Date: Thu, 04 Jun 2026 21:16:00 GMT
• Title: Evaluating LLM Usage for Efficient and Explainable Numerical and Classified Implicit Sentiment Analysis of Product Desirability
• Abstract summary: This paper presents a scalable and interpretable framework that uses large language models (LLMs) to quantify product desirability.<n>The framework also incorporates model confidence ratings and human-readable rationale explanations (xAI)
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Qualitative product feedback can reveal nuanced user experiences, but its implicit sentiment is difficult to measure. This paper presents a scalable and interpretable framework that uses large language models (LLMs) to quantify product desirability from such data. Using two Product Desirability Toolkit (PDT) datasets from ZORQ and CARMA comprising 106 respondent term groupings with gold-standard human annotation, zero-shot continuous numerical sentiment scoring and categorical sentiment classification are evaluated without relying on explicit review scores. Across the datasets, LLMs generated numerical sentiment scores directly from qualitative responses and closely matched expert labels, achieving Pearson correlations up to 0.97 and classification accuracy up to 94%. LLMs maintained robustness even when handling data presented in multiple forms and consistently expressed high confidence. In contrast, lexicon-based and transformer baselines did not produce statistically significant results. Among the models tested, GPT-4o-mini achieved performance comparable to larger models at 94% lower cost, supporting scalable deployment. The framework also incorporates model confidence ratings and human-readable rationale explanations (xAI), improving interpretability, transparency, and trust while supporting practical use in product satisfaction assessment. In general, using the PDT tool as a survey method along with a cost efficient LLM for sentiment analysis has the potential to provide for product evaluation with results that are rich in terms of sentiment scores (both numerical and classified sentiment) and in terms of the high-level user impressions of the product that can be used to identify ideas for product development and improvement, as well as marketing ideas for target audiences.

Related papers

• (Towards) Scalable Reliable Automated Evaluation with Large Language Models [0.0]
  This work introduces a novel evaluation framework designed to approximate expert-level assessments of Large Language Models (LLMs) outputs.<n>An Elo rating system is used to generate stable and interpretable rankings.<n>Preliminary results show that automatically derived rankings correlate well with expert judgments.
  arXiv  Detail & Related papers  (2026-07-30T14:31:07Z)
• DualEval: Joint Model-Item Calibration for Unified LLM Evaluation [67.99119926012686]
  DualEval is a latent model-item calibration framework that represents models and evaluation items in a shared space.<n>We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries.
  arXiv  Detail & Related papers  (2026-06-24T22:40:46Z)
• Latent Performance Profiling of Large Language Models [47.009623327601226]
  We introduce Latent Performance Profiling (LPP) -- a framework that derives task-agnostic diagnostics from hidden activations and output distributions.<n>Unlike static accuracy scores, LPP provides stable, architecture-sensitive signatures across models of similar size.<n>We show that models with similar benchmark scores can exhibit contrasting latent profiles, such as differences in entropy or adaptability.
  arXiv  Detail & Related papers  (2026-05-28T14:41:26Z)
• Leveraging LLMs to Evaluate Usefulness of Document [25.976948104719746]
  We introduce a new user-centric evaluation framework that integrates users' search context and behavioral data into large language models.<n>Our study demonstrates that when well-guided with context and behavioral information, LLMs can accurately evaluate usefulness.<n>We also apply the labels produced by our method to predict user satisfaction, with real-world experiments indicating that these labels substantially improve the performance of satisfaction prediction models.
  arXiv  Detail & Related papers  (2025-06-10T09:44:03Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items [2.9357382494347264]
  Existing benchmarks rely on human or machine annotations that are vulnerable to value-related biases.<n>We propose the Value Portrait benchmark, which consists of items that capture real-life user-LLM interactions.<n>This psychometrically validated approach ensures that items strongly correlated with specific values serve as reliable items for assessing those values.
  arXiv  Detail & Related papers  (2025-05-02T05:26:50Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis [10.133537818749291]
  Large language models (LLMs) have demonstrated significant utilities in real-world applications.<n> Benchmark evaluations are crucial for assessing the capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-13T03:43:33Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.