ECHO: An Open Research Platform for Evaluation of Chat, Human Behavior, and Outcomes

• URL: http://arxiv.org/abs/2602.10295v1
• Date: Tue, 10 Feb 2026 21:10:38 GMT
• Title: ECHO: An Open Research Platform for Evaluation of Chat, Human Behavior, and Outcomes
• Authors: Jiqun Liu, Nischal Dinesh, Ran Yu,
• Abstract summary: ECHO is an open research platform designed to support mixed-method studies of human interaction with both conversational AI systems and Web search engines.<n>It enables researchers from varying disciplines to orchestrate end-to-end experimental that integrate consent and background surveys, chat-based and search-based information-seeking sessions, writing or judgment tasks, and pre-task evaluations within a unified, low-coding-load framework.
• Score: 6.989051035721272
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: ECHO (Evaluation of Chat, Human behavior, and Outcomes) is an open research platform designed to support reproducible, mixed-method studies of human interaction with both conversational AI systems and Web search engines. It enables researchers from varying disciplines to orchestrate end-to-end experimental workflows that integrate consent and background surveys, chat-based and search-based information-seeking sessions, writing or judgment tasks, and pre- and post-task evaluations within a unified, low-coding-load framework. ECHO logs fine-grained interaction traces and participant responses, and exports structured datasets for downstream analysis. By supporting both chat and search alongside flexible evaluation instruments, ECHO lowers technical barriers for studying learning, decision making, and user experience across different information access paradigms, empowering researchers from information retrieval, HCI, and the social sciences to conduct scalable and reproducible human-centered AI evaluations.

Related papers

• Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset [47.98539809308384]
  We analyze the Asta Interaction dataset, a large-scale resource comprising over 200,000 user queries and interaction logs.<n>We characterize query patterns, engagement behaviors, and how usage evolves with experience.<n>We release the anonymized dataset and analysis with a new query taxonomy to inform future designs of real-world AI research assistants.
  arXiv  Detail & Related papers  (2026-02-26T18:40:28Z)
• LISP -- A Rich Interaction Dataset and Loggable Interactive Search Platform [10.637323019551035]
  We present a reusable dataset and accompanying infrastructure for studying human search behavior in Interactive Information Retrieval (IIR)<n>The dataset combines detailed interaction logs from 61 participants with user characteristics, including perceptual speed, topic-specific interest, search expertise, and demographic information.
  arXiv  Detail & Related papers  (2026-01-14T10:49:13Z)
• "My Boyfriend is AI": A Computational Analysis of Human-AI Companionship in Reddit's AI Community [28.482163389070646]
  We present the first large-scale computational analysis of r/MyBoyfriendIsAI, Reddit's primary AI companion community.<n>Our findings reveal how community members' AI companionship emerges unintentionally through functional use rather than deliberate seeking.
  arXiv  Detail & Related papers  (2025-09-14T19:00:40Z)
• User-centric evaluation of explainability of AI with and for humans: a comprehensive empirical study [5.775094401949666]
  This study is located in the Human-Centered Artificial Intelligence (HCAI) It focuses on the results of a user-centered assessment of commonly used eXplainable Artificial Intelligence (XAI) algorithms.
  arXiv  Detail & Related papers  (2024-10-21T12:32:39Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• Can AI Serve as a Substitute for Human Subjects in Software Engineering Research? [24.39463126056733]
  This vision paper proposes a novel approach to qualitative data collection in software engineering research by harnessing the capabilities of artificial intelligence (AI) We explore the potential of AI-generated synthetic text as an alternative source of qualitative data. We discuss the prospective development of new foundation models aimed at emulating human behavior in observational studies and user evaluations.
  arXiv  Detail & Related papers  (2023-11-18T14:05:52Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• Human-Robot Collaboration and Machine Learning: A Systematic Review of Recent Research [69.48907856390834]
  Human-robot collaboration (HRC) is the approach that explores the interaction between a human and a robot. This paper proposes a thorough literature review of the use of machine learning techniques in the context of HRC.
  arXiv  Detail & Related papers  (2021-10-14T15:14:33Z)
• Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots [51.091235903442715]
  This paper makes an attempt to explore the impact of utilizing personas that describe either self or partner speakers on the task of response selection. Four persona fusion strategies are designed, which assume personas interact with contexts or responses in different ways. Empirical studies on the Persona-Chat dataset show that the partner personas can improve the accuracy of response selection.
  arXiv  Detail & Related papers  (2021-05-19T10:32:30Z)
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach [84.02388020258141]
  We propose a new framework named ENIGMA for estimating human evaluation scores based on off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.
  arXiv  Detail & Related papers  (2021-02-20T03:29:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.