Human-AI Interaction and User Satisfaction: Empirical Evidence from Online Reviews of AI Products

• URL: http://arxiv.org/abs/2503.17955v2
• Date: Tue, 25 Mar 2025 01:44:50 GMT
• Title: Human-AI Interaction and User Satisfaction: Empirical Evidence from Online Reviews of AI Products
• Authors: Stefan Pasch, Sun-Young Ha,
• Abstract summary: This study analyzes over 100,000 user reviews of AI-related products from G2, a leading review platform for business software and services.<n>We identify seven core HAI dimensions and examine their coverage and sentiment within the reviews.<n>We find that the sentiment on four HAI dimensions-adaptability, customization, error recovery, and security-is positively associated with overall user satisfaction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human-AI Interaction (HAI) guidelines and design principles have become increasingly important in both industry and academia to guide the development of AI systems that align with user needs and expectations. However, large-scale empirical evidence on how HAI principles shape user satisfaction in practice remains limited. This study addresses that gap by analyzing over 100,000 user reviews of AI-related products from G2, a leading review platform for business software and services. Based on widely adopted industry guidelines, we identify seven core HAI dimensions and examine their coverage and sentiment within the reviews. We find that the sentiment on four HAI dimensions-adaptability, customization, error recovery, and security-is positively associated with overall user satisfaction. Moreover, we show that engagement with HAI dimensions varies by professional background: Users with technical job roles are more likely to discuss system-focused aspects, such as reliability, while non-technical users emphasize interaction-focused features like customization and feedback. Interestingly, the relationship between HAI sentiment and overall satisfaction is not moderated by job role, suggesting that once an HAI dimension has been identified by users, its effect on satisfaction is consistent across job roles.

Related papers

• Reasoning LLMs for User-Aware Multimodal Conversational Agents [3.533721662684487]
  Personalization in social robotics is critical for fostering effective human-robot interactions. This paper proposes a novel framework called USER-LLM R1 for a user-aware conversational agent. Our approach integrates chain-of-thought (CoT) reasoning models to iteratively infer user preferences and vision-language models.
  arXiv  Detail & Related papers  (2025-04-02T13:00:17Z)
• Are We Solving a Well-Defined Problem? A Task-Centric Perspective on Recommendation Tasks [46.705107776194616]
  We analyze RecSys task formulations, emphasizing key components such as input-output structures, temporal dynamics, and candidate item selection. We explore the balance between task specificity and model generalizability, highlighting how well-defined task formulations serve as the foundation for robust evaluation and effective solution development.
  arXiv  Detail & Related papers  (2025-03-27T06:10:22Z)
• Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions [20.986518776079404]
  Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise.<n>Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise.<n>We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user.
  arXiv  Detail & Related papers  (2025-02-25T22:46:51Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.50078821423793]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• Welzijn.AI: A Conversational AI System for Monitoring Mental Well-being and a Use Case for Responsible AI Development [3.257656198821199]
  Welzijn.AI is a digital solution for monitoring mental well-being in the elderly.<n>Technology concerns the description of an open, well-documented and interpretable envisioned architecture.<n>Value concerns stakeholder evaluations of Welzijn.AI.
  arXiv  Detail & Related papers  (2025-02-11T21:59:19Z)
• Evaluating Cultural and Social Awareness of LLM Web Agents [113.49968423990616]
  We introduce CASA, a benchmark designed to assess large language models' sensitivity to cultural and social norms. Our approach evaluates LLM agents' ability to detect and appropriately respond to norm-violating user queries and observations. Experiments show that current LLMs perform significantly better in non-agent environments.
  arXiv  Detail & Related papers  (2024-10-30T17:35:44Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review [48.052150453947405]
  No single term is used across the literature to describe HAII. HAII is most measured with user-related subjective metrics. Researchers and developers need to codify HAII terminology.
  arXiv  Detail & Related papers  (2023-10-05T08:57:17Z)
• Identifying Characteristics of the Agile Development Process That Impact User Satisfaction [3.3748063434734843]
  The purpose of this study is to identify the characteristics of Agile development processes that impact user satisfaction. No metrics conclusively indicate an improved user satisfaction, motivation of the development team, the ability to set appropriate work units, the appropriateness of work rules, and the improvement of code maintainability should be considered as they are correlated with improved user satisfaction.
  arXiv  Detail & Related papers  (2023-06-06T08:09:14Z)
• A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective [0.0]
  User trust in Artificial Intelligence (AI) enabled systems has been increasingly recognized and proven as a key element to fostering adoption. This review aims to provide an overview of the user trust definitions, influencing factors, and measurement methods from 23 empirical studies.
  arXiv  Detail & Related papers  (2023-04-18T07:58:09Z)
• Proactive Detractor Detection Framework Based on Message-Wise Sentiment Analysis Over Customer Support Interactions [60.87845704495664]
  We propose a framework relying solely on chat-based customer support interactions for predicting the recommendation decision of individual users. For our case study, we analyzed a total number of 16.4k users and 48.7k customer support conversations within the financial vertical of a large e-commerce company in Latin America. Our results show that, with respective feature interpretability, it is possible to predict the likelihood of a user to recommend a product or service, based solely on the message-wise sentiment evolution of their CS conversations in a fully automated way.
  arXiv  Detail & Related papers  (2022-11-08T00:43:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.