Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution

• URL: http://arxiv.org/abs/2609.07594v1
• Date: Mon, 07 Sep 2026 15:06:21 GMT
• Title: Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution
• Abstract summary: Conversational Task Assistants (CTAs) support users in complex real-world tasks through voice, text, image, and video interactions.<n>Prior user studies have focused on controlled settings, leaving limited understanding of real-world CTA usage at scale.<n>We present a large-scale study of CTA usage based on thousands of users in-the-wild.
• Score: 8.281425921021079
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational Task Assistants (CTAs) are multimodal dialogue systems that support users in complex real-world tasks such as cooking and DIY through voice, text, image, and video interactions. Prior user studies have focused on controlled settings, leaving limited understanding of real-world CTA usage at scale. In this work, we present a large-scale study of CTA usage based on thousands of users in-the-wild. Our large-scale real-world data analysis unveils new understandings of (i) user-CTA interaction flows, (ii) user intents, (iii) user conversational traits, and (iv) behavioral factors associated with user satisfaction. Our findings reveal key opportunities for future research in CTAs, particularly in user interaction design and task engagement, concluding with concrete design guidelines.

Related papers

• VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions [63.13827503828231]
  We introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions.<n>Results show that real-world personalization remains highly challenging even for state-of-the-art models.
  arXiv  Detail & Related papers  (2026-05-26T15:07:38Z)
• Human vs. Agent in Task-Oriented Conversations [22.743152820695588]
  This work presents the first systematic comparison between large language models (LLMs)-simulated users and human users in personalized task-oriented conversations.<n>Our analysis reveals significant behavioral differences between the two user types in problem-solving approaches.
  arXiv  Detail & Related papers  (2025-09-22T11:30:39Z)
• Generative Interfaces for Language Models [70.25765232527762]
  We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with up to a 72% improvement in human preference.
  arXiv  Detail & Related papers  (2025-08-26T17:43:20Z)
• UserBench: An Interactive Gym Environment for User-Centric Agents [110.77212949007958]
  Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, but their ability to proactively collaborate with users remains underexplored.<n>We introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions.
  arXiv  Detail & Related papers  (2025-07-29T17:34:12Z)
• RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing [133.0641538589466]
  RMTBench is a comprehensive textbfuser-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds.<n>Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications.<n>By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements.
  arXiv  Detail & Related papers  (2025-07-27T16:49:47Z)
• Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild [10.23533525266164]
  Large language models (LLMs) are used in complex writing to steer generations to better fit their needs.<n>We conduct a large-scale analysis of this collaborative behavior for users engaged in writing tasks in the wild.<n>We identify prototypical behaviors in how users interact with LLMs in prompts following their original request.
  arXiv  Detail & Related papers  (2025-05-21T21:13:01Z)
• Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions [20.986518776079404]
  Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise.<n>Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise.<n>We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user.
  arXiv  Detail & Related papers  (2025-02-25T22:46:51Z)
• Survey of User Interface Design and Interaction Techniques in Generative AI Applications [79.55963742878684]
  We aim to create a compendium of different user-interaction patterns that can be used as a reference for designers and developers alike. We also strive to lower the entry barrier for those attempting to learn more about the design of generative AI applications.
  arXiv  Detail & Related papers  (2024-10-28T23:10:06Z)
• A Speaker Turn-Aware Multi-Task Adversarial Network for Joint User Satisfaction Estimation and Sentiment Analysis [28.940337080659905]
  We propose a novel Speaker Turn-Aware Multi-Task Adversarial Network (STMAN) for dialogue-level User Satisfaction Estimation (USE) and Sentiment Analysis (SA) We first introduce a multi-task adversarial strategy which trains a task discriminator to make utterance representation more task-specific, and then utilize a speaker-turn aware multi-task interaction strategy to extract the common features which are complementary to each task.
  arXiv  Detail & Related papers  (2024-10-12T15:03:53Z)
• Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations [17.409790984399052]
  This paper explores the impact of multi-modal interactions, which incorporate images and audio alongside text, on user engagement. Our findings reveal a significant enhancement in user engagement with multi-modal interactions compared to text-only dialogues. Results suggest that multi-modal interactions optimize cognitive processing and facilitate richer information comprehension.
  arXiv  Detail & Related papers  (2024-06-21T09:26:55Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users [51.34484827552774]
  We release the Multi-User MultiWOZ dataset: task-oriented dialogues among two users and one agent. These dialogues reflect interesting dynamics of collaborative decision-making in task-oriented scenarios. We propose a novel task of multi-user contextual query rewriting: to rewrite a task-oriented chat between two users as a concise task-oriented query.
  arXiv  Detail & Related papers  (2023-10-31T14:12:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.