Psychometrics in Behavioral Software Engineering: A Methodological
Introduction with Guidelines
- URL: http://arxiv.org/abs/2005.09959v4
- Date: Tue, 8 Jun 2021 13:10:37 GMT
- Title: Psychometrics in Behavioral Software Engineering: A Methodological
Introduction with Guidelines
- Authors: Daniel Graziotin, Per Lenberg, Robert Feldt, Stefan Wagner
- Abstract summary: We provide an introduction to psychometric theory for the evaluation of measurement instruments for software engineering researchers.
We detail activities used when operationalizing new psychological constructs, such as item pooling, item review, pilot testing, item analysis, factor analysis, statistical property of items, reliability, validity, and fairness in testing and test bias.
We hope to encourage a culture change in SE research towards the adoption of established methods from psychology.
- Score: 19.40714760075466
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A meaningful and deep understanding of the human aspects of software
engineering (SE) requires psychological constructs to be considered. Psychology
theory can facilitate the systematic and sound development as well as the
adoption of instruments (e.g., psychological tests, questionnaires) to assess
these constructs. In particular, to ensure high quality, the psychometric
properties of instruments need evaluation. In this paper, we provide an
introduction to psychometric theory for the evaluation of measurement
instruments for SE researchers. We present guidelines that enable using
existing instruments and developing new ones adequately. We conducted a
comprehensive review of the psychology literature framed by the Standards for
Educational and Psychological Testing. We detail activities used when
operationalizing new psychological constructs, such as item pooling, item
review, pilot testing, item analysis, factor analysis, statistical property of
items, reliability, validity, and fairness in testing and test bias. We provide
an openly available example of a psychometric evaluation based on our
guideline. We hope to encourage a culture change in SE research towards the
adoption of established methods from psychology. To improve the quality of
behavioral research in SE, studies focusing on introducing, validating, and
then using psychometric instruments need to be more common.
Related papers
- Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models [57.518784855080334]
Large Language Models (LLMs) have demonstrated exceptional task-solving capabilities, increasingly adopting roles akin to human-like assistants.
This paper presents a framework for investigating psychology dimension in LLMs, including psychological identification, assessment dataset curation, and assessment with results validation.
We introduce a comprehensive psychometrics benchmark for LLMs that covers six psychological dimensions: personality, values, emotion, theory of mind, motivation, and intelligence.
arXiv Detail & Related papers (2024-06-25T16:09:08Z) - Mental health of computing professionals and students: A systematic literature review [2.532202013576547]
We evaluate the state-of-the-art of research in mental health and well-being interventions, assessments, and concerns like anxiety and depression in computer science and computing education.
arXiv Detail & Related papers (2024-05-06T12:31:34Z) - Survey of Computerized Adaptive Testing: A Machine Learning Perspective [66.26687542572974]
Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees.
This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method.
arXiv Detail & Related papers (2024-03-31T15:09:47Z) - LLM Agents for Psychology: A Study on Gamified Assessments [71.08193163042107]
Psychological measurement is essential for mental health, self-understanding, and personal development.
PsychoGAT (Psychological Game AgenTs) achieves statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity.
arXiv Detail & Related papers (2024-02-19T18:00:30Z) - Precision psychiatry: predicting predictability [0.0]
I review ten challenges in the field of precision psychiatry.
Need for studies on real-world populations and realistic clinical outcome definitions.
Consider treatment-related factors such as placebo effects and non-adherence to prescriptions.
arXiv Detail & Related papers (2023-06-21T13:10:46Z) - Concepts and Experiments on Psychoanalysis Driven Computing [0.0]
This research investigates the effective incorporation of the human factor and user perception in text-based interactive media.
We use the notion of Lacanian discourse types to capture and deeply understand real characteristics, qualities and contents of texts.
This is the first time computational methods are systematically combined with psychoanalysis.
arXiv Detail & Related papers (2022-09-29T19:27:22Z) - Interpretability by design using computer vision for behavioral sensing
in child and adolescent psychiatry [3.975358343371988]
We use machine learning to derive behavioral codes or concepts of a gold standard behavioral rating system.
Our ratings were comparable to human expert ratings for negative emotions, activity-level/arousal and anxiety.
arXiv Detail & Related papers (2022-07-11T09:07:08Z) - Evaluating and Inducing Personality in Pre-trained Language Models [78.19379997967191]
We draw inspiration from psychometric studies by leveraging human personality theory as a tool for studying machine behaviors.
To answer these questions, we introduce the Machine Personality Inventory (MPI) tool for studying machine behaviors.
MPI follows standardized personality tests, built upon the Big Five Personality Factors (Big Five) theory and personality assessment inventories.
We devise a Personality Prompting (P2) method to induce LLMs with specific personalities in a controllable way.
arXiv Detail & Related papers (2022-05-20T07:32:57Z) - AGENT: A Benchmark for Core Psychological Reasoning [60.35621718321559]
Intuitive psychology is the ability to reason about hidden mental variables that drive observable actions.
Despite recent interest in machine agents that reason about other agents, it is not clear if such agents learn or hold the core psychology principles that drive human reasoning.
We present a benchmark consisting of procedurally generated 3D animations, AGENT, structured around four scenarios.
arXiv Detail & Related papers (2021-02-24T14:58:23Z) - Opportunities of a Machine Learning-based Decision Support System for
Stroke Rehabilitation Assessment [64.52563354823711]
Rehabilitation assessment is critical to determine an adequate intervention for a patient.
Current practices of assessment mainly rely on therapist's experience, and assessment is infrequently executed due to the limited availability of a therapist.
We developed an intelligent decision support system that can identify salient features of assessment using reinforcement learning.
arXiv Detail & Related papers (2020-02-27T17:04:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.