Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions

• URL: http://arxiv.org/abs/2502.18685v1
• Date: Tue, 25 Feb 2025 22:46:51 GMT
• Title: Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions
• Authors: Shramay Palta, Nirupama Chandrasekaran, Rachel Rudinger, Scott Counts,
• Abstract summary: Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise.<n>Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise.<n>We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user.
• Score: 20.986518776079404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise and the resulting impact on user experience along multiple dimensions. Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise (77% of conversations) which correlates with positive user experience regardless of the user's level of expertise. Misalignment, such that the agent responds at a level of expertise below that of the user, has a negative impact on overall user experience, with the impact more profound for more complex tasks. We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user. Our findings underscore the importance of alignment between user and AI when designing human-centered AI systems, to ensure satisfactory and productive interactions.

Related papers

• Exploring Personality-Aware Interactions in Salesperson Dialogue Agents [21.282523537612477]
  This study explores the influence of user personas, defined using the Myers-Briggs Type Indicator (MBTI), on the interaction quality and performance of sales-oriented dialogue agents. Our findings reveal significant patterns in interaction dynamics, task completion rates, and dialogue naturalness, underscoring the future potential for dialogue agents to refine their strategies.
  arXiv  Detail & Related papers  (2025-04-25T04:10:25Z)
• Should We Tailor the Talk? Understanding the Impact of Conversational Styles on Preference Elicitation in Conversational Recommender Systems [19.830560938115436]
  This study investigates the impact of different conversational styles on preference elicitation, task performance, and user satisfaction with Conversational recommender systems (CRSs) Our results indicate that adapting conversational strategies based on user expertise and allowing flexibility between styles can enhance both user satisfaction and the effectiveness of recommendations in CRSs.
  arXiv  Detail & Related papers  (2025-04-17T17:01:17Z)
• Exploring the Impact of Personality Traits on Conversational Recommender Systems: A Simulation with Large Language Models [70.180385882195]
  This paper introduces a personality-aware user simulation for Conversational Recommender Systems (CRSs) The user agent induces customizable personality traits and preferences, while the system agent possesses the persuasion capability to simulate realistic interaction in CRSs. Experimental results demonstrate that state-of-the-art LLMs can effectively generate diverse user responses aligned with specified personality traits.
  arXiv  Detail & Related papers  (2025-04-09T13:21:17Z)
• Human-AI Interaction and User Satisfaction: Empirical Evidence from Online Reviews of AI Products [0.0]
  This study analyzes over 100,000 user reviews of AI-related products from G2, a leading review platform for business software and services. We identify seven core HAI dimensions and examine their coverage and sentiment within the reviews. We find that the sentiment on four HAI dimensions-adaptability, customization, error recovery, and security-is positively associated with overall user satisfaction.
  arXiv  Detail & Related papers  (2025-03-23T06:16:49Z)
• A Speaker Turn-Aware Multi-Task Adversarial Network for Joint User Satisfaction Estimation and Sentiment Analysis [28.940337080659905]
  We propose a novel Speaker Turn-Aware Multi-Task Adversarial Network (STMAN) for dialogue-level User Satisfaction Estimation (USE) and Sentiment Analysis (SA) We first introduce a multi-task adversarial strategy which trains a task discriminator to make utterance representation more task-specific, and then utilize a speaker-turn aware multi-task interaction strategy to extract the common features which are complementary to each task.
  arXiv  Detail & Related papers  (2024-10-12T15:03:53Z)
• Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations [17.409790984399052]
  This paper explores the impact of multi-modal interactions, which incorporate images and audio alongside text, on user engagement. Our findings reveal a significant enhancement in user engagement with multi-modal interactions compared to text-only dialogues. Results suggest that multi-modal interactions optimize cognitive processing and facilitate richer information comprehension.
  arXiv  Detail & Related papers  (2024-06-21T09:26:55Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems [60.27663010453209]
  We leverage large language models (LLMs) to generate satisfaction-aware counterfactual dialogues. We gather human annotations to ensure the reliability of the generated samples. Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems.
  arXiv  Detail & Related papers  (2024-03-27T23:45:31Z)
• Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents [110.25679611755962]
  Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. We introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries. We empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals.
  arXiv  Detail & Related papers  (2024-02-14T14:36:30Z)
• Affective Conversational Agents: Understanding Expectations and Personal Influences [17.059654991560105]
  We surveyed 745 respondents to understand the expectations and preferences regarding affective skills in various applications. Our results indicate a preference for scenarios that involve human interaction, emotional support, and creative tasks. Overall, the desired affective skills in AI agents depend largely on the application's context and nature.
  arXiv  Detail & Related papers  (2023-10-19T04:33:18Z)
• Understanding How People Rate Their Conversations [73.17730062864314]
  We conduct a study to better understand how people rate their interactions with conversational agents. We focus on agreeableness and extraversion as variables that may explain variation in ratings.
  arXiv  Detail & Related papers  (2022-06-01T00:45:32Z)
• Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy [83.61404191470126]
  We propose a new solution named I-Pro that can learn Proactive policy in the Interactive setting. Specifically, we learn the trade-off via a learned goal weight, which consists of four factors. The experimental results demonstrate I-Pro significantly outperforms baselines in terms of effectiveness and interpretability.
  arXiv  Detail & Related papers  (2022-04-07T14:11:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.