Are You a Real Software Engineer? Best Practices in Online Recruitment for Software Engineering Studies

• URL: http://arxiv.org/abs/2402.01925v1
• Date: Fri, 2 Feb 2024 21:53:28 GMT
• Title: Are You a Real Software Engineer? Best Practices in Online Recruitment for Software Engineering Studies
• Authors: Adam Alami and Mansooreh Zahedi and Neil Ernst
• Abstract summary: Previous studies reported mixed outcomes and challenges in leveraging online platforms for the recruitment of qualified software engineers. We propose best practices for recruiting and screening participants to enhance the quality and relevance of both qualitative and quantitative software engineering (SE) research samples.
• Score: 4.247193377317027
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Online research platforms, such as Prolific, offer rapid access to diverse participant pools but also pose unique challenges in participant qualification and skill verification. Previous studies reported mixed outcomes and challenges in leveraging online platforms for the recruitment of qualified software engineers. Drawing from our experience in conducting three different studies using Prolific, we propose best practices for recruiting and screening participants to enhance the quality and relevance of both qualitative and quantitative software engineering (SE) research samples. We propose refined best practices for recruitment in SE research on Prolific. (1) Iterative and controlled prescreening, enabling focused and manageable assessment of submissions (2) task-oriented and targeted questions that assess technical skills, knowledge of basic SE concepts, and professional engagement. (3) AI detection to verify the authenticity of free-text responses. (4) Qualitative and manual assessment of responses, ensuring authenticity and relevance in participant answers (5) Additional layers of prescreening are necessary when necessary to collect data relevant to the topic of the study. (6) Fair or generous compensation post-qualification to incentivize genuine participation. By sharing our experiences and lessons learned, we contribute to the development of effective and rigorous methods for SE empirical research. particularly the ongoing effort to establish guidelines to ensure reliable data collection. These practices have the potential to transferability to other participant recruitment platforms.

Related papers

• Using Large Language Models to Develop Requirements Elicitation Skills [1.1473376666000734]
  We propose conditioning a large language model to play the role of the client during a chat-based interview. We find that both approaches provide sufficient information for participants to construct technically sound solutions.
  arXiv  Detail & Related papers  (2025-03-10T19:27:38Z)
• CritiQ: Mining Data Quality Criteria from Human Preferences [70.35346554179036]
  We introduce CritiQ, a novel data selection method that automatically mines criteria from human preferences for data quality. CritiQ Flow employs a manager agent to evolve quality criteria and worker agents to make pairwise judgments. We demonstrate the effectiveness of our method in the code, math, and logic domains.
  arXiv  Detail & Related papers  (2025-02-26T16:33:41Z)
• ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification [53.80183105328448]
  Refine via Intrinsic Self-Verification (ReVISE) is an efficient framework that enables LLMs to self-correct their outputs through self-verification. Our experiments on various reasoning tasks demonstrate that ReVISE achieves efficient self-correction and significantly improves reasoning performance.
  arXiv  Detail & Related papers  (2025-02-20T13:50:02Z)
• CyberMentor: AI Powered Learning Tool Platform to Address Diverse Student Needs in Cybersecurity Education [6.267144136593821]
  Many non-traditional students in cybersecurity programs often lack access to advice from peers, family members and professors. This paper introduces an application designed to provide comprehensive support by answering questions related to knowledge, skills, and career preparation advice. We developed a learning tool platform, CyberMentor, to address the diverse needs and pain points of students in cybersecurity.
  arXiv  Detail & Related papers  (2025-01-16T18:00:06Z)
• NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA [49.74911193222192]
  The competition introduced a dataset of real invoice documents, along with associated questions and answers. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold.
  arXiv  Detail & Related papers  (2024-11-06T07:51:19Z)
• Preliminary Insights on Industry Practices for Addressing Fairness Debt [4.546982900370235]
  This study explores how software professionals identify and address biases in AI systems within the software industry. Our paper presents initial evidence on addressing fairness debt and provides a foundation for developing structured guidelines to manage fairness-related issues in AI systems.
  arXiv  Detail & Related papers  (2024-09-04T04:18:42Z)
• AI Research is not Magic, it has to be Reproducible and Responsible: Challenges in the AI field from the Perspective of its PhD Students [1.1922075410173798]
  We surveyed 28 AI doctoral candidates from 13 European countries. Challenges underscore the findability and quality of AI resources such as datasets, models, and experiments. There is need for immediate adoption of responsible and reproducible AI research practices.
  arXiv  Detail & Related papers  (2024-08-13T12:19:02Z)
• A pragmatic look at education and training of software test engineers: Further cooperation of academia and industry is needed [1.516251872371896]
  It is important for both university educators and trainers in industry to be aware of the status of software testing education in academia. This paper provides a pragmatic overview of the issue, presents several recommendations, and hopes to trigger further discussions.
  arXiv  Detail & Related papers  (2024-08-12T13:39:52Z)
• Fairness in AI-Driven Recruitment: Challenges, Metrics, Methods, and Future Directions [0.0]
  Big data and machine learning has led to a rapid transformation in the traditional recruitment process. Given the prevalence of AI-based recruitment, there is growing concern that human biases may carry over to decisions made by these systems. This paper provides a comprehensive overview of this emerging field by discussing the types of biases encountered in AI-driven recruitment.
  arXiv  Detail & Related papers  (2024-05-30T05:25:14Z)
• Facilitating Multi-Role and Multi-Behavior Collaboration of Large Language Models for Online Job Seeking and Recruiting [51.54907796704785]
  Existing methods rely on modeling the latent semantics of resumes and job descriptions and learning a matching function between them. Inspired by the powerful role-playing capabilities of Large Language Models (LLMs), we propose to introduce a mock interview process between LLM-played interviewers and candidates. We propose MockLLM, a novel applicable framework that divides the person-job matching process into two modules: mock interview generation and two-sided evaluation in handshake protocol.
  arXiv  Detail & Related papers  (2024-05-28T12:23:16Z)
• LOVA3: Learning to Visual Question Answering, Asking and Assessment [61.51687164769517]
  Question answering, asking, and assessment are three innate human traits crucial for understanding the world and acquiring knowledge. Current Multimodal Large Language Models (MLLMs) primarily focus on question answering, often neglecting the full potential of questioning and assessment skills. We introduce LOVA3, an innovative framework named "Learning tO Visual question Answering, Asking and Assessment"
  arXiv  Detail & Related papers  (2024-05-23T18:21:59Z)
• SurveyAgent: A Conversational System for Personalized and Efficient Research Survey [50.04283471107001]
  This paper introduces SurveyAgent, a novel conversational system designed to provide personalized and efficient research survey assistance to researchers. SurveyAgent integrates three key modules: Knowledge Management for organizing papers, Recommendation for discovering relevant literature, and Query Answering for engaging with content on a deeper level. Our evaluation demonstrates SurveyAgent's effectiveness in streamlining research activities, showcasing its capability to facilitate how researchers interact with scientific literature.
  arXiv  Detail & Related papers  (2024-04-09T15:01:51Z)
• Quality Assurance for Artificial Intelligence: A Study of Industrial Concerns, Challenges and Best Practices [14.222404866137756]
  We report on the challenges and best practices of quality assurance for AI systems (QA4AI) Our findings suggest correctness as the most important property, followed by model relevance, efficiency and deployability. We identify 21 QA4AI practices across each stage of AI development.
  arXiv  Detail & Related papers  (2024-02-26T08:31:45Z)
• Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning [65.12226891589592]
  This paper proposes a new approach to automated game validation and testing. Our method leverages a data-driven imitation learning technique, which requires little effort and time and no knowledge of machine learning or programming.
  arXiv  Detail & Related papers  (2022-08-15T11:08:44Z)
• Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots [51.091235903442715]
  This paper makes an attempt to explore the impact of utilizing personas that describe either self or partner speakers on the task of response selection. Four persona fusion strategies are designed, which assume personas interact with contexts or responses in different ways. Empirical studies on the Persona-Chat dataset show that the partner personas can improve the accuracy of response selection.
  arXiv  Detail & Related papers  (2021-05-19T10:32:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.