Facilitating Longitudinal Interaction Studies of AI Systems

• URL: http://arxiv.org/abs/2508.10252v1
• Date: Thu, 14 Aug 2025 00:38:23 GMT
• Title: Facilitating Longitudinal Interaction Studies of AI Systems
• Abstract summary: UIST researchers develop tools to address user challenges.<n>User interactions with AI evolve over time through learning, adaptation, and repurposing, making one time evaluations insufficient.<n>Our workshop aims to tackle these challenges and prepare researchers with practical strategies for longitudinal studies.
• Score: 45.430296301751895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: UIST researchers develop tools to address user challenges. However, user interactions with AI evolve over time through learning, adaptation, and repurposing, making one time evaluations insufficient. Capturing these dynamics requires longer-term studies, but challenges in deployment, evaluation design, and data collection have made such longitudinal research difficult to implement. Our workshop aims to tackle these challenges and prepare researchers with practical strategies for longitudinal studies. The workshop includes a keynote, panel discussions, and interactive breakout groups for discussion and hands-on protocol design and tool prototyping sessions. We seek to foster a community around longitudinal system research and promote it as a more embraced method for designing, building, and evaluating UIST tools.

Related papers

• A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inference [72.19201465132619]
  We present feedback-driven Test-Time Intelligence (TTI) as a unified perspective for understanding such deployment-time improvement.<n>We use this view to relate test-time adaptation, test-time learning, and test-time scaling, highlighting both their distinctions and their growing overlap in hybrid systems.<n>Our goal is to provide a coherent foundation and research roadmap for the study of self-improving AI systems at test time.
  arXiv  Detail & Related papers  (2026-09-01T11:41:39Z)
• Intentmaking and Sensemaking: Human Interaction with AI-Guided Mathematical Discovery [72.49593499512092]
  We identify and characterize a distinct workflow we term intentmaking.<n>We frame this as a natural extension to sensemaking.<n>Our documentation of these themes suggests an approach to designing AI tools for scientific discovery.
  arXiv  Detail & Related papers  (2026-05-07T09:30:25Z)
• Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset [47.98539809308384]
  We analyze the Asta Interaction dataset, a large-scale resource comprising over 200,000 user queries and interaction logs.<n>We characterize query patterns, engagement behaviors, and how usage evolves with experience.<n>We release the anonymized dataset and analysis with a new query taxonomy to inform future designs of real-world AI research assistants.
  arXiv  Detail & Related papers  (2026-02-26T18:40:28Z)
• ECHO: An Open Research Platform for Evaluation of Chat, Human Behavior, and Outcomes [6.989051035721272]
  ECHO is an open research platform designed to support mixed-method studies of human interaction with both conversational AI systems and Web search engines.<n>It enables researchers from varying disciplines to orchestrate end-to-end experimental that integrate consent and background surveys, chat-based and search-based information-seeking sessions, writing or judgment tasks, and pre-task evaluations within a unified, low-coding-load framework.
  arXiv  Detail & Related papers  (2026-02-10T21:10:38Z)
• Towards AI-Supported Research: a Vision of the TIB AIssistant [6.36260975777314]
  We present the vision of the TIB AIssistant, a domain-agnostic human-machine collaborative platform designed to support researchers across disciplines in scientific discovery.<n>We describe the conceptual framework, system architecture, and implementation of an early prototype that demonstrates the feasibility and potential impact of our approach.
  arXiv  Detail & Related papers  (2025-12-18T12:08:46Z)
• Deep Research: A Systematic Survey [118.82795024422722]
  Deep Research (DR) aims to combine the reasoning capabilities of large language models with external tools, such as search engines.<n>This survey presents a comprehensive and systematic overview of deep research systems.
  arXiv  Detail & Related papers  (2025-11-24T15:28:28Z)
• StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation [13.425907817210184]
  Design processes involve exploration, iteration, and movement across interconnected stages such as persona creation, problem framing, solution ideation, and prototyping.<n>To better understand these challenges, we conducted a formative study with 15 participants-comprised of UX practitioners, students, and instructors.<n>Based on the findings, we developed StoryEnsemble, a tool that integrates AI into a node-link interface and leverages forward and backward propagation to support dynamic exploration and iteration across the design process.
  arXiv  Detail & Related papers  (2025-08-05T07:47:23Z)
• Large Language Model Agent: A Survey on Methodology, Applications and Challenges [88.3032929492409]
  Large Language Model (LLM) agents, with goal-driven behaviors and dynamic adaptation capabilities, potentially represent a critical pathway toward artificial general intelligence.<n>This survey systematically deconstructs LLM agent systems through a methodology-centered taxonomy.<n>Our work provides a unified architectural perspective, examining how agents are constructed, how they collaborate, and how they evolve over time.
  arXiv  Detail & Related papers  (2025-03-27T12:50:17Z)
• Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM [29.73055078727462]
  Vital Insight is a novel, LLM-assisted, prototype system to enable human-in-the-loop inference (sensemaking) and visualizations of multi-modal passive sensing data from smartphones and wearables.<n>We observe experts' interactions with it and develop an expert sensemaking model that explains how experts move between direct data representations and AI-supported inferences.
  arXiv  Detail & Related papers  (2024-10-18T21:56:35Z)
• Constraining Participation: Affordances of Feedback Features in Interfaces to Large Language Models [49.74265453289855]
  Large language models (LLMs) are now accessible to anyone with a computer, a web browser, and an internet connection via browser-based interfaces. This paper examines the affordances of interactive feedback features in ChatGPT's interface, analysing how they shape user input and participation in iteration.
  arXiv  Detail & Related papers  (2024-08-27T13:50:37Z)
• Acceleron: A Tool to Accelerate Research Ideation [15.578814192003437]
  Acceleron is a research accelerator for different phases of the research life cycle. It guides researchers through the formulation of a comprehensive research proposal, encompassing a novel research problem. We leverage the reasoning and domain-specific skills of Large Language Models (LLMs) to create an agent-based architecture.
  arXiv  Detail & Related papers  (2024-03-07T10:20:06Z)
• Investigating Collaborative Data Practices: a Case Study on Artificial Intelligence for Healthcare Research [1.3178083420209858]
  We look at the collaborative data practices of research consortia tasked with applying AI tools to understand and manage multiple long-term conditions in the UK. Our findings reveal the adaptation of tools that are used for sharing knowledge and the tailoring of information based on the audience. We identify meetings as the key setting for facilitating exchanges between disciplines and allowing for the blending and creation of knowledge.
  arXiv  Detail & Related papers  (2023-11-30T10:19:33Z)
• Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? [62.59699229202307]
  Despite advances in AI, it remains a significant challenge to develop interactive task guidance systems. We created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance.
  arXiv  Detail & Related papers  (2023-11-01T15:13:49Z)
• Towards Formalizing HRI Data Collection Processes [4.090390588417062]
  We contribute a clearly defined process to collect data with three steps for machine learning modeling purposes. Specifically, we discuss our data collection goal and how we worked to encourage well-covered and abundant participant responses.
  arXiv  Detail & Related papers  (2022-03-16T04:59:18Z)
• Stakeholder Participation in AI: Beyond "Add Diverse Stakeholders and Stir" [76.44130385507894]
  This paper aims to ground what we dub a 'participatory turn' in AI design by synthesizing existing literature on participation and through empirical analysis of its current practices. Based on our literature synthesis and empirical research, this paper presents a conceptual framework for analyzing participatory approaches to AI design.
  arXiv  Detail & Related papers  (2021-11-01T17:57:04Z)
• Human-Robot Collaboration and Machine Learning: A Systematic Review of Recent Research [69.48907856390834]
  Human-robot collaboration (HRC) is the approach that explores the interaction between a human and a robot. This paper proposes a thorough literature review of the use of machine learning techniques in the context of HRC.
  arXiv  Detail & Related papers  (2021-10-14T15:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.