Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking

• URL: http://arxiv.org/abs/2511.23376v1
• Date: Fri, 28 Nov 2025 17:21:41 GMT
• Title: Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking
• Authors: Li Siyan, Jason Zhang, Akash Maharaj, Yuanming Shi, Yunyao Li,
• Abstract summary: Expert users have different systematic preferences in task-oriented dialogues.<n>We built a version of an enterprise AI assistant with passive personalization.<n>Preliminary results indicate that passive personalization helps reduce task load and improve assistant perception.<n>These findings underscore the importance of combining active and passive personalization to optimize user experience and effectiveness in enterprise task-oriented environments.
• Score: 29.26173340915243
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Novice and expert users have different systematic preferences in task-oriented dialogues. However, whether catering to these preferences actually improves user experience and task performance remains understudied. To investigate the effects of expertise-based personalization, we first built a version of an enterprise AI assistant with passive personalization. We then conducted a user study where participants completed timed exams, aided by the two versions of the AI assistant. Preliminary results indicate that passive personalization helps reduce task load and improve assistant perception, but reveal task-specific limitations that can be addressed through providing more user agency. These findings underscore the importance of combining active and passive personalization to optimize user experience and effectiveness in enterprise task-oriented environments.

Related papers

• PersonaDual: Balancing Personalization and Objectivity via Adaptive Reasoning [57.10486355722133]
  PersonaDual supports both general-purpose objective reasoning and personalized reasoning in a single model.<n>Experiments on objective and personalized benchmarks show that PersonaDual preserves the benefits of personalization while reducing interference.
  arXiv  Detail & Related papers  (2026-01-13T16:02:35Z)
• Training Proactive and Personalized LLM Agents [107.57805582180315]
  We introduce PPP, a multi-objective reinforcement learning approach that jointly optimize all three dimensions: Productivity, Proactivity, and Personalization.<n>Experiments show that agents trained with PPP achieve substantial improvements over strong baselines such as GPT-5 (+21.6 on average)<n>This work demonstrates that explicitly optimizing for user-centered interaction is critical for building practical and effective AI agents.
  arXiv  Detail & Related papers  (2025-11-04T02:59:36Z)
• One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning [52.966712416640085]
  We propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies.<n>SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-29T08:54:58Z)
• ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation [26.512935389758727]
  We introduce ProPerSim, a task and simulation framework for developing AI assistants capable of making timely, personalized recommendations.<n>We propose ProPerAssistant, a retrieval-augmented, preference-aligned assistant that continually learns and adapts through user feedback.
  arXiv  Detail & Related papers  (2025-09-26T00:57:27Z)
• PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants [31.486658078902025]
  We introduce PersonaLens, a benchmark for evaluating personalization in task-oriented AI assistants.<n>Our benchmark features diverse user profiles equipped with rich preferences and interaction histories, along with two specialized LLM-based agents.<n>We reveal significant variability in their personalization capabilities, providing crucial insights for advancing conversational AI systems.
  arXiv  Detail & Related papers  (2025-06-11T16:16:07Z)
• PersonaAgent: When Large Language Model Agents Meet Personalization at Test Time [87.99027488664282]
  PersonaAgent is a framework designed to address versatile personalization tasks.<n>It integrates a personalized memory module and a personalized action module.<n>Test-time user-preference alignment strategy ensures real-time user preference alignment.
  arXiv  Detail & Related papers  (2025-06-06T17:29:49Z)
• Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization [26.34637576545121]
  LLM-powered embodied agents have shown success on conventional object-rearrangement tasks.<n>However, providing personalized assistance that leverages user-specific knowledge from past interactions presents new challenges.<n>We investigate these challenges through the lens of agents' memory utilization.
  arXiv  Detail & Related papers  (2025-05-22T08:00:10Z)
• Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks [51.15473776489712]
  We introduce a simple yet effective task selection method that leverages instruction information alone to identify relevant tasks. Our method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Experimental results demonstrate that training on a small set of tasks, chosen solely on the instructions, results in substantial improvements in performance on benchmarks such as P3, Big-Bench, NIV2, and Big-Bench Hard.
  arXiv  Detail & Related papers  (2024-04-25T08:49:47Z)
• User Modeling Challenges in Interactive AI Assistant Systems [3.1204913702660475]
  Interactive Artificial Intelligent(AI) assistant systems are designed to offer timely guidance to help human users to complete a variety tasks. One of the remaining challenges is to understand user's mental states during the task for more personalized guidance. In this work, we analyze users' mental states during task executions and investigate the capabilities and challenges for large language models to interpret user profiles for more personalized user guidance.
  arXiv  Detail & Related papers  (2024-03-29T11:54:13Z)
• Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement [79.2400720115588]
  We introduce Persona-DB, a simple yet effective framework consisting of a hierarchical construction process to improve generalization across task contexts.<n>In the evaluation of response prediction, Persona-DB demonstrates superior context efficiency in maintaining accuracy with a significantly reduced retrieval size.<n>Our experiments also indicate a marked improvement of over 10% under cold-start scenarios, when users have extremely sparse data.
  arXiv  Detail & Related papers  (2024-02-16T20:20:43Z)
• From User Surveys to Telemetry-Driven AI Agents: Exploring the Potential of Personalized Productivity Solutions [21.79433247723466]
  Information workers increasingly struggle with productivity challenges in modern workplaces.<n>Despite availability of productivity metrics through enterprise tools, workers often fail to translate this data into actionable insights.<n>We present a comprehensive, user-centric approach to address these challenges through AI-based productivity agents tailored to users' needs.
  arXiv  Detail & Related papers  (2024-01-17T04:20:10Z)
• Assisting Human Decisions in Document Matching [52.79491990823573]
  We devise a proxy matching task that allows us to evaluate which kinds of assistive information improve decision makers' performance. We find that providing black-box model explanations reduces users' accuracy on the matching task. On the other hand, custom methods that are designed to closely attend to some task-specific desiderata are found to be effective in improving user performance.
  arXiv  Detail & Related papers  (2023-02-16T17:45:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.