How to Elicit Explainability Requirements? A Comparison of Interviews, Focus Groups, and Surveys

• URL: http://arxiv.org/abs/2505.23684v3
• Date: Wed, 09 Jul 2025 17:13:26 GMT
• Title: How to Elicit Explainability Requirements? A Comparison of Interviews, Focus Groups, and Surveys
• Abstract summary: This study examines the efficiency and effectiveness of three commonly used elicitation methods - focus groups, interviews, and online surveys.<n>The results show that interviews were the most efficient, capturing the highest number of distinct needs per participant per time spent.<n>We recommend a hybrid approach combining surveys and interviews to balance efficiency and coverage.
• Score: 2.30929645503432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As software systems grow increasingly complex, explainability has become a crucial non-functional requirement for transparency, user trust, and regulatory compliance. Eliciting explainability requirements is challenging, as different methods capture varying levels of detail and structure. This study examines the efficiency and effectiveness of three commonly used elicitation methods - focus groups, interviews, and online surveys - while also assessing the role of taxonomy usage in structuring and improving the elicitation process. We conducted a case study at a large German IT consulting company, utilizing a web-based personnel management software. A total of two focus groups, 18 interviews, and an online survey with 188 participants were analyzed. The results show that interviews were the most efficient, capturing the highest number of distinct needs per participant per time spent. Surveys collected the most explanation needs overall but had high redundancy. Delayed taxonomy introduction resulted in a greater number and diversity of needs, suggesting that a two-phase approach is beneficial. Based on our findings, we recommend a hybrid approach combining surveys and interviews to balance efficiency and coverage. Future research should explore how automation can support elicitation and how taxonomies can be better integrated into different methods.

Related papers

• CoWeaver: A Bi-directional, Learnable and Explainable Matching Engine for Mixed Human-Agent Science Collaboration [29.03592005523777]
  LLM-based agents excel at writing articles, coding and information retrieval.<n>We propose COWEAVER, a bidirectional, learnable and explainable algorithm to match scientists and form strong collaborations within a human-agent network.
  arXiv  Detail & Related papers  (2026-07-17T01:35:12Z)
• From Chat to Interview: Agentic Requirements Elicitation with an Experience Ontology [35.19488207612296]
  This paper proposes an interview agent named OntoAgent for the elicitation of requirements guided by an experience ontology.<n>OntoAgent automatically analyzes domain-specific requirements descriptions to construct an experience ontology.<n>The results show that OntoAgent significantly outperforms existing baselines in both elicitation effectiveness and questioning efficiency.
  arXiv  Detail & Related papers  (2026-05-07T08:03:03Z)
• KARL: Knowledge Agents via Reinforcement Learning [63.627906947205624]
  We present a system for training enterprise search agents via reinforcement learning.<n> KARLBench is a multi-capability evaluation suite spanning six distinct search regimes.<n>We show that models trained across heterogeneous search behaviors generalize substantially better than those optimized for any single benchmark.
  arXiv  Detail & Related papers  (2026-03-05T14:30:25Z)
• Lessons from the Field: An Adaptable Lifecycle Approach to Applied Dialogue Summarization [12.244621802309352]
  We present an industry case study on developing an agentic system to summarize multi-party interactions.<n>We share practical insights spanning the full development lifecycle to guide practitioners in building reliable, adaptable summarization systems.
  arXiv  Detail & Related papers  (2026-01-13T16:04:11Z)
• Questionnaire meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses [3.8293581919117123]
  Large language models (LLMs) excel at few-shot reasoning over open-ended text.<n>Current retrieval and survey analysis tools are typically designed for humans in the workflow.<n>We introduce QASU, a benchmark that probes six structural skills, including answer lookup, respondent count, and multi-hop inference.<n>Experiments show that choosing an effective format and prompt combination can improve accuracy by up to 8.8% points compared to suboptimal formats.
  arXiv  Detail & Related papers  (2025-10-30T08:18:37Z)
• Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization [86.98098988779809]
  We propose SummQ, a novel adversarial multi-agent framework for long document summarization.<n>Our approach employs summary generators and reviewers that work collaboratively to create and evaluate comprehensive summaries.<n>We evaluate SummQ on three widely used long document summarization benchmarks.
  arXiv  Detail & Related papers  (2025-09-25T08:36:19Z)
• A Multi-To-One Interview Paradigm for Efficient MLLM Evaluation [63.76972456980632]
  We propose a multi-to-one interview paradigm for efficient MLLM evaluation.<n>Our framework consists of (i) a two-stage interview strategy with pre-interview and formal interview phases, (ii) dynamic adjustment of weights to ensure fairness, and (iii) an adaptive mechanism for question difficulty-level chosen.
  arXiv  Detail & Related papers  (2025-09-18T12:07:40Z)
• LLMREI: Automating Requirements Elicitation Interviews with LLMs [47.032121951473435]
  This study introduces LLMREI, a chat bot designed to conduct requirements elicitation interviews with minimal human intervention.<n>We evaluated its performance in 33 simulated stakeholder interviews.<n>Our findings indicate that LLMREI makes a similar number of errors compared to human interviewers, is capable of extracting a large portion of requirements, and demonstrates a notable ability to generate highly context-dependent questions.
  arXiv  Detail & Related papers  (2025-07-03T12:18:05Z)
• Feedback-Aware Monte Carlo Tree Search for Efficient Information Seeking in Goal-Oriented Conversations [10.352944689413398]
  We introduce a novel approach to adaptive question-asking through a combination of Large Language Models (LLM) for generating questions that maximize information gain.<n>We present two key innovations: (1) an adaptive MCTS algorithm that balances exploration and exploitation for efficient search over potential questions; and (2) a clustering-based feedback algorithm that leverages prior experience to guide future interactions.
  arXiv  Detail & Related papers  (2025-01-25T03:42:22Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that the pointwise mutual information between a context and a question is an effective gauge for language model performance.<n>We propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• A Comprehensive Survey on Retrieval Methods in Recommender Systems [32.1847120460637]
  This survey explores the critical yet often overlooked retrieval stage of recommender systems. To achieve precise and efficient personalized retrieval, we summarize existing work in three key areas. We highlight current industrial applications through a case study on retrieval practices at a specific company.
  arXiv  Detail & Related papers  (2024-07-11T07:09:59Z)
• Thread: A Logic-Based Data Organization Paradigm for How-To Question Answering with Retrieval Augmented Generation [49.36436704082436]
  How-to questions are integral to decision-making processes and require dynamic, step-by-step answers. We propose Thread, a novel data organization paradigm aimed at enabling current systems to handle how-to questions more effectively.
  arXiv  Detail & Related papers  (2024-06-19T09:14:41Z)
• Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search [89.1772985740272]
  In mixed-initiative conversational search systems, clarifying questions are used to help users who struggle to express their intentions in a single query. We hypothesize that in scenarios where multimodal information is pertinent, the clarification process can be improved by using non-textual information. We collect a dataset named Melon that contains over 4k multimodal clarifying questions, enriched with over 14k images. Several analyses are conducted to understand the importance of multimodal contents during the query clarification phase.
  arXiv  Detail & Related papers  (2024-02-12T16:04:01Z)
• Promoting Research Collaboration with Open Data Driven Team Recommendation in Response to Call for Proposals [10.732914229005903]
  We describe a novel system to recommend teams using a variety of AI methods. We create teams to maximize goodness along a metric balancing short- and long-term objectives.
  arXiv  Detail & Related papers  (2023-09-18T00:04:08Z)
• Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement [0.0]
  This paper reports on two experiments investigating using non-expert crowdworkers and comparative judgement to evaluate student data. We found that using comparative judgement substantially improved inter-rater reliability on both tasks.
  arXiv  Detail & Related papers  (2023-05-22T10:22:14Z)
• Reinforcement Learning Guided Multi-Objective Exam Paper Generation [21.945655389912112]
  We propose a reinforcement learning guided Multi-Objective Exam Paper Generation framework, termed MOEPG. It simultaneously optimize three exam domain-specific objectives including difficulty degree, distribution of exam scores, and skill coverage. We show that MOEPG is feasible in addressing the multiple dilemmas of exam paper generation scenario.
  arXiv  Detail & Related papers  (2023-03-02T07:55:52Z)
• Scaling up Search Engine Audits: Practical Insights for Algorithm Auditing [68.8204255655161]
  We set up experiments for eight search engines with hundreds of virtual agents placed in different regions. We demonstrate the successful performance of our research infrastructure across multiple data collections. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms across long periods of time.
  arXiv  Detail & Related papers  (2021-06-10T15:49:58Z)
• Mining Implicit Relevance Feedback from User Behavior for Web Question Answering [92.45607094299181]
  We make the first study to explore the correlation between user behavior and passage relevance. Our approach significantly improves the accuracy of passage ranking without extra human labeled data. In practice, this work has proved effective to substantially reduce the human labeling cost for the QA service in a global commercial search engine.
  arXiv  Detail & Related papers  (2020-06-13T07:02:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.