What Do We Mean by 'Pilot Study': Early Findings from a Meta-Review of Pilot Study Reporting at CHI

• URL: http://arxiv.org/abs/2602.13488v1
• Date: Fri, 13 Feb 2026 21:48:30 GMT
• Title: What Do We Mean by 'Pilot Study': Early Findings from a Meta-Review of Pilot Study Reporting at CHI
• Abstract summary: The role of pilot studies in HCI remains conceptually vague and empirically underexamined.<n>Many papers reference pilots 'in passing', without details about design, outcomes, or how the pilot informed the main study.<n>This variability suggests a methodological blind spot in our community.
• Score: 24.978257584329018
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pilot studies (PS) are ubiquitous in HCI research. CHI papers routinely reference 'pilot studies', 'pilot tests', or 'preliminary studies' to justify design decisions, verify procedures, or motivate methodological choices. Yet despite their frequency, the role of pilot studies in HCI remains conceptually vague and empirically underexamined. Unlike fields such as medicine, nursing, and education, where pilot and feasibility studies have well-established definitions, guidelines, reporting standards and even a dedicated research journal, the CHI community lacks a shared understanding of what constitutes a pilot study, why they are conducted, and how they should be reported. Many papers reference pilots 'in passing', without details about design, outcomes, or how the pilot informed the main study. This variability suggests a methodological blind spot in our community.

Related papers

• Research Methodologies for Cybersecurity in Enterprise Environments: A Narrative Review, Synthesis and Executable Guide [0.0]
  This paper provides a narrative review and synthesis of methodological practices across a verified corpus of 151 works.<n>We organise these practices into eleven methodology families, detailing for each what questions it answers, the strength of its supporting evidence, and its common failure modes.<n>Every protocol is also visually mapped to make the sequence, decisions, and threats legible at a glance.
  arXiv  Detail & Related papers  (2026-08-25T17:38:00Z)
• Personalized Auto-Research: Towards a True AI Co-Scientist [90.01790420664169]
  Given a research goal, state-of-the-art systems remain researcher-agnostic.<n>What counts as novel, valuable, or feasible depends on the researcher.<n>We propose a framework that threads a graph-grounded researcher context through retrieval, hypothesis search, experimentation, writing, and review.
  arXiv  Detail & Related papers  (2026-08-14T20:43:27Z)
• Confounder Detection via Treatment Intent: A New Observational Study Design [69.50062870487349]
  We introduce a new study design called confounder detection via treatment intent.<n>The idea is to query a human expert who makes treatment decisions, and ask them to compare pairs of units.<n>We show empirical evidence that electronic health records collected in ICUs are subject to unobserved confounding.
  arXiv  Detail & Related papers  (2026-05-26T00:41:47Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Large Language Models as Co-Pilots for Causal Inference in Medical Studies [0.0]
  We explore the use of large language models (LLMs) as co-pilot tools to assist researchers in identifying study design flaws that undermine the validity of causal inferences. We propose a conceptual framework for LLMs as causal co-pilots that encode domain knowledge across various fields, engaging with researchers in natural language interactions to provide contextualized assistance in study design.
  arXiv  Detail & Related papers  (2024-07-26T22:43:15Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and Guidelines [12.782288692460565]
  A lack of details surrounding pilot studies in crowdsourcing research hinders the replication of studies and the reproduction of findings. We conducted a systematic literature review on the current state of pilot study reporting at the intersection of crowdsourcing and HCI research. We formulate a set of best practice guidelines for reporting crowd pilot studies in crowdsourcing research.
  arXiv  Detail & Related papers  (2023-12-13T12:13:40Z)
• De-identification of clinical free text using natural language processing: A systematic review of current approaches [48.343430343213896]
  Natural language processing has repeatedly demonstrated its feasibility in automating the de-identification process. Our study aims to provide systematic evidence on how the de-identification of clinical free text has evolved in the last thirteen years.
  arXiv  Detail & Related papers  (2023-11-28T13:20:41Z)
• Too Good To Be True: performance overestimation in (re)current practices for Human Activity Recognition [49.1574468325115]
  sliding windows for data segmentation followed by standard random k-fold cross validation produce biased results. It is important to raise awareness in the scientific community about this problem, whose negative effects are being overlooked. Several experiments with different types of datasets and different types of classification models allow us to exhibit the problem and show it persists independently of the method or dataset.
  arXiv  Detail & Related papers  (2023-10-18T13:24:05Z)
• Incentive-Theoretic Bayesian Inference for Collaborative Science [59.15962177829337]
  We study hypothesis testing when there is an agent with a private prior about an unknown parameter. We show how the principal can conduct statistical inference that leverages the information that is revealed by an agent's strategic behavior.
  arXiv  Detail & Related papers  (2023-07-07T17:59:01Z)
• Navigating the reporting guideline environment for computational pathology: A review [0.685316573653194]
  The aim of this work is to highlight resources and reporting guidelines available to researchers working in computational pathology. Items were compiled to create a summary for easy identification of useful resources and guidance. Over 70 published resources applicable to pathology AI research were identified.
  arXiv  Detail & Related papers  (2023-01-03T23:17:51Z)
• A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature [4.339510167603376]
  The goal of this study is to characterize the reporting of empiricism in Computing Education Research (CER) literature. We conducted an SLR of 427 papers published during 2014 and 2015 in five CER venues. Over 80% of papers had some form of empirical evaluation.
  arXiv  Detail & Related papers  (2021-07-02T16:37:29Z)
• All Factors Should Matter! Reference Checklist for Describing Research Conditions in Pursuit of Comparable IVR Experiments [5.874802930380899]
  A significant problem with immersive virtual reality (IVR) experiments is the ability to compare research conditions. VR kits and tactile environments are complex and diverse but researchers from different fields, e.g. ICT, psychology, or marketing, often neglect to describe them with a level of detail sufficient to their research on the landscape. This article presents a reference checklist for describing research conditions of experiments.
  arXiv  Detail & Related papers  (2021-01-04T23:45:52Z)
• A Survey on Causal Inference [64.45536158710014]
  Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics. Various causal effect estimation methods for observational data have sprung up.
  arXiv  Detail & Related papers  (2020-02-05T21:35:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.