Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis

• URL: http://arxiv.org/abs/2603.05510v1
• Date: Sun, 18 Jan 2026 02:38:01 GMT
• Title: Exploring Human-in-the-Loop Themes in AI Application Development: An Empirical Thematic Analysis
• Abstract summary: Human-in-the-Loop (HITL) and Human-Centered AI (HCAI) principles are widely acknowledged.<n>But operational guidance for structuring roles, checkpoints, and feedback mechanisms remains fragmented.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developing and deploying AI applications in organizations is challenging when human decision authority and oversight are underspecified across the system lifecycle. Although Human-in-the-Loop (HITL) and Human-Centered AI (HCAI) principles are widely acknowledged, operational guidance for structuring roles, checkpoints, and feedback mechanisms remains fragmented. We report a multi-source qualitative study: a retrospective diary study of a customer-support chatbot and semi-structured interviews with eight AI experts from academia and industry. Through five-cycle thematic analysis of 1,435 codewords, we derive four themes: AI Governance and Human Authority, Human-in-the-Loop Iterative Refinement, AI System Lifecycle and Operational Constraints, and Human-AI Team Collaboration and Coordination. These themes provide empirical inputs for subsequent HITL framework design and validation.

Related papers

• From Challenge to Change: Design Principles for AI Transformations [9.232567192178836]
  The rapid rise of Artificial Intelligence (AI) is reshaping Software Engineering (SE)<n>This paper proposes a Behavioral Software Engineering (BSE)-informed, human-centric framework to support SE organizations during early AI adoption.<n>The framework comprises nine dimensions: AI Strategy Design, AI Strategy Evaluation, Collaboration, Communication, Governance and Ethics, Leadership, Organizational Culture, Organizational Dynamics, and Up-skilling.
  arXiv  Detail & Related papers  (2025-12-05T08:45:14Z)
• Classifying Epistemic Relationships in Human-AI Interaction: An Exploratory Approach [0.6906005491572401]
  This study examines how users form relationships with AI-how they assess, trust, and collaborate with it in research and teaching contexts.<n>Based on 31 interviews with academics across disciplines, we developed a five-part codebook and identified five relationship types.
  arXiv  Detail & Related papers  (2025-08-02T23:41:28Z)
• AI4Research: A Survey of Artificial Intelligence for Scientific Research [55.5452803680643]
  We present a comprehensive survey on AI for Research (AI4Research)<n>We first introduce a systematic taxonomy to classify five mainstream tasks in AI4Research.<n>We identify key research gaps and highlight promising future directions.
  arXiv  Detail & Related papers  (2025-07-02T17:19:20Z)
• A Question Bank to Assess AI Inclusivity: Mapping out the Journey from Diversity Errors to Inclusion Excellence [5.364403920214549]
  This paper introduces a structured AI inclusivity question bank, a comprehensive set of 253 questions designed to evaluate AI inclusivity.<n>The development of the question bank involved an iterative, multi-source approach, incorporating insights from literature reviews, D&I guidelines, Responsible AI frameworks.<n>The simulated evaluation, conducted with 70 AI-generated personas related to different AI jobs, assessed the question bank's relevance and effectiveness for AI inclusivity.
  arXiv  Detail & Related papers  (2025-06-23T11:48:38Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Human-Centered AI Transformation: Exploring Behavioral Dynamics in Software Engineering [6.126394204968227]
  This study uses Behavioral Software Engineering as a lens to examine these often-overlooked dimensions of AI transformation. Our findings reveal six key challenges tied to these BSE aspects that the organizations face during their AI transformation.
  arXiv  Detail & Related papers  (2024-11-13T15:29:24Z)
• Position: Towards Bidirectional Human-AI Alignment [109.57781720848669]
  We argue that the research community should explicitly define and critically reflect on "alignment" to account for the bidirectional and dynamic relationship between humans and AI.<n>We introduce the Bidirectional Human-AI Alignment framework, which not only incorporates traditional efforts to align AI with human values but also introduces the critical, underexplored dimension of aligning humans with AI.
  arXiv  Detail & Related papers  (2024-06-13T16:03:25Z)
• On some Foundational Aspects of Human-Centered Artificial Intelligence [52.03866242565846]
  There is no clear definition of what is meant by Human Centered Artificial Intelligence. This paper introduces the term HCAI agent to refer to any physical or software computational agent equipped with AI components. We see the notion of HCAI agent, together with its components and functions, as a way to bridge the technical and non-technical discussions on human-centered AI.
  arXiv  Detail & Related papers  (2021-12-29T09:58:59Z)
• Stakeholder Participation in AI: Beyond "Add Diverse Stakeholders and Stir" [76.44130385507894]
  This paper aims to ground what we dub a 'participatory turn' in AI design by synthesizing existing literature on participation and through empirical analysis of its current practices. Based on our literature synthesis and empirical research, this paper presents a conceptual framework for analyzing participatory approaches to AI design.
  arXiv  Detail & Related papers  (2021-11-01T17:57:04Z)
• Human-Centered AI for Data Science: A Systematic Approach [48.71756559152512]
  Human-Centered AI (HCAI) refers to the research effort that aims to design and implement AI techniques to support various human tasks. We illustrate how we approach HCAI using a series of research projects around Data Science (DS) works as a case study.
  arXiv  Detail & Related papers  (2021-10-03T21:47:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.