Evaluating Generative AI Tools for Personalized Offline Recommendations: A Comparative Study

• URL: http://arxiv.org/abs/2508.03710v1
• Date: Tue, 22 Jul 2025 02:47:26 GMT
• Title: Evaluating Generative AI Tools for Personalized Offline Recommendations: A Comparative Study
• Abstract summary: Generative AI tools have become increasingly relevant in supporting personalized recommendations across various domains.<n>This study evaluates the performance and user satisfaction of the five most widely used generative AI tools when recommending non-digital activities tailored to individuals at risk of repetitive strain injury.
• Score: 2.503968994462208
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Generative AI tools have become increasingly relevant in supporting personalized recommendations across various domains. However, their effectiveness in health-related behavioral interventions, especially those aiming to reduce the use of technology, remains underexplored. Aims: This study evaluates the performance and user satisfaction of the five most widely used generative AI tools when recommending non-digital activities tailored to individuals at risk of repetitive strain injury. Method: Following the Goal/Question/Metric (GQM) paradigm, this proposed experiment involves generative AI tools that suggest offline activities based on predefined user profiles and intervention scenarios. The evaluation is focused on quantitative performance (precision, recall, F1-score and MCC-score) and qualitative aspects (user satisfaction and perceived recommendation relevance). Two research questions were defined: RQ1 assessed which tool delivers the most accurate recommendations, and RQ2 evaluated how tool choice influences user satisfaction.

Related papers

• RecThinker: An Agentic Framework for Tool-Augmented Reasoning in Recommendation [63.74915464611075]
  RecThinker is an agentic framework for tool-augmented reasoning in recommendation.<n>We develop a suite of specialized tools for RecThinker, enabling the model to acquire user item-side and collaborative information.
  arXiv  Detail & Related papers  (2026-03-10T16:07:17Z)
• How Effective Are Publicly Accessible Deepfake Detection Tools? A Comparative Evaluation of Open-Source and Free-to-Use Platforms [1.189955933770711]
  Deepfake imagery poses escalating challenges for practitioners tasked with verifying digital media authenticity.<n>This paper presents the first cross-paradigm evaluation of six tools, spanning two complementary detection approaches.
  arXiv  Detail & Related papers  (2026-03-02T14:31:51Z)
• MedAI: Evaluating TxAgent's Therapeutic Agentic Reasoning in the NeurIPS CURE-Bench Competition [6.191248426050678]
  Therapeutic decision-making in clinical medicine requires robust, multi-step reasoning grounded in reliable biomedical knowledge.<n>Agentic AI methods, exemplified by TxAgent, address these challenges through iterative retrieval-augmented generation (RAG)<n>This work presents insights derived from our participation in the CURE-Bench NeurIPS 2025 Challenge, which benchmarks therapeutic-reasoning systems.
  arXiv  Detail & Related papers  (2025-12-12T16:01:48Z)
• Performance Assessment Strategies for Generative AI Applications in Healthcare [1.0486921990935787]
  Generative artificial intelligence (GenAI) represent an emerging paradigm within artificial intelligence, with applications throughout the medical enterprise.<n>We discuss current state-of-the-art methodologies for assessing the performance of GenAI applications in healthcare and medical devices.
  arXiv  Detail & Related papers  (2025-09-09T18:50:26Z)
• TestAgent: An Adaptive and Intelligent Expert for Human Assessment [62.060118490577366]
  We propose TestAgent, a large language model (LLM)-powered agent designed to enhance adaptive testing through interactive engagement.<n>TestAgent supports personalized question selection, captures test-takers' responses and anomalies, and provides precise outcomes through dynamic, conversational interactions.
  arXiv  Detail & Related papers  (2025-06-03T16:07:54Z)
• Performance evaluation of predictive AI models to support medical decisions: Overview and guidance [9.674673298809324]
  We aim to assess the merits of classic and contemporary performance measures when validating predictive AI models for use in medical practice.<n>We discuss 32 performance measures covering five performance domains (discrimination, calibration, overall, classification, and clinical utility) along with accompanying graphical assessments.
  arXiv  Detail & Related papers  (2024-12-13T17:11:47Z)
• Quantifying User Coherence: A Unified Framework for Cross-Domain Recommendation Analysis [69.37718774071793]
  This paper introduces novel information-theoretic measures for understanding recommender systems. We evaluate 7 recommendation algorithms across 9 datasets, revealing the relationships between our measures and standard performance metrics.
  arXiv  Detail & Related papers  (2024-10-03T13:02:07Z)
• Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trials [0.9865722130817715]
  We present an innovative N-of-1 trial study design testing whether implementing a personalized intervention by an online reinforcement learning agent is feasible and effective. The results show that, first, implementing a personalized intervention by an online reinforcement learning agent is feasible. Second, such adaptive interventions have the potential to improve patients' benefits even if only few observations are available.
  arXiv  Detail & Related papers  (2023-09-25T14:08:21Z)
• Learning and Evaluating Human Preferences for Conversational Head Generation [101.89332968344102]
  We propose a novel learning-based evaluation metric named Preference Score (PS) for fitting human preference according to the quantitative evaluations across different dimensions. PS can serve as a quantitative evaluation without the need for human annotation.
  arXiv  Detail & Related papers  (2023-07-20T07:04:16Z)
• A Meta-Learning Method for Estimation of Causal Excursion Effects to Assess Time-Varying Moderation [0.0]
  "DR-W" is a flexible and doubly robust inferential procedure for estimating causal excursion effects from a meta-learner perspective.<n>We present the bidirectional properties of the proposed estimators and compare them with existing methods both theoretically and through simulations.<n>The results show a consistent and more efficient estimate, even with missing observations or uncertain treatment randomization probabilities.
  arXiv  Detail & Related papers  (2023-06-28T15:19:33Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores [85.12096045419686]
  We study the adoption of an algorithmic tool used to assist child maltreatment hotline screening decisions. We first show that humans do alter their behavior when the tool is deployed. We show that humans are less likely to adhere to the machine's recommendation when the score displayed is an incorrect estimate of risk.
  arXiv  Detail & Related papers  (2020-02-19T07:27:32Z)
• Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions [48.91284724066349]
  Off-policy evaluation in reinforcement learning offers the chance of using observational data to improve future outcomes in domains such as healthcare and education. Traditional measures such as confidence intervals may be insufficient due to noise, limited data and confounding. We develop a method that could serve as a hybrid human-AI system, to enable human experts to analyze the validity of policy evaluation estimates.
  arXiv  Detail & Related papers  (2020-02-10T00:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.