An unsupervised learning approach to evaluate questionnaire data -- what one can learn from violations of measurement invariance

• URL: http://arxiv.org/abs/2312.06309v1
• Date: Mon, 11 Dec 2023 11:31:41 GMT
• Title: An unsupervised learning approach to evaluate questionnaire data -- what one can learn from violations of measurement invariance
• Authors: Max Hahn-Klimroth, Paul W. Dierkes, Matthias W. Kleespies
• Abstract summary: This paper promotes an unsupervised learning-based approach to such research data. It works in three phases: data preparation, clustering of questionnaires, and measuring similarity based on the obtained clustering and the properties of each group. It provides a natural comparison between groups and a natural description of the response patterns of the groups.
• Score: 2.4762962548352467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In several branches of the social sciences and humanities, surveys based on standardized questionnaires are a prominent research tool. While there are a variety of ways to analyze the data, some standard procedures have become established. When those surveys want to analyze differences in the answer patterns of different groups (e.g., countries, gender, age, ...), these procedures can only be carried out in a meaningful way if there is measurement invariance, i.e., the measured construct has psychometric equivalence across groups. As recently raised as an open problem by Sauerwein et al. (2021), new evaluation methods that work in the absence of measurement invariance are needed. This paper promotes an unsupervised learning-based approach to such research data by proposing a procedure that works in three phases: data preparation, clustering of questionnaires, and measuring similarity based on the obtained clustering and the properties of each group. We generate synthetic data in three data sets, which allows us to compare our approach with the PCA approach under measurement invariance and under violated measurement invariance. As a main result, we obtain that the approach provides a natural comparison between groups and a natural description of the response patterns of the groups. Moreover, it can be safely applied to a wide variety of data sets, even in the absence of measurement invariance. Finally, this approach allows us to translate (violations of) measurement invariance into a meaningful measure of similarity.

Related papers

• A structured regression approach for evaluating model performance across intersectional subgroups [53.91682617836498]
  Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups. We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
  arXiv  Detail & Related papers  (2024-01-26T14:21:45Z)
• Measuring Adversarial Datasets [28.221635644616523]
  Researchers have curated various adversarial datasets for capturing model deficiencies that cannot be revealed in standard benchmark datasets. There is still no methodology to measure the intended and unintended consequences of those adversarial transformations. We conducted a systematic survey of existing quantifiable metrics that describe text instances in NLP tasks.
  arXiv  Detail & Related papers  (2023-11-06T22:08:16Z)
• Assaying Out-Of-Distribution Generalization in Transfer Learning [103.57862972967273]
  We take a unified view of previous work, highlighting message discrepancies that we address empirically. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting.
  arXiv  Detail & Related papers  (2022-07-19T12:52:33Z)
• Estimating Structural Disparities for Face Models [54.062512989859265]
  In machine learning, disparity metrics are often defined by measuring the difference in the performance or outcome of a model, across different sub-populations. We explore performing such analysis on computer vision models trained on human faces, and on tasks such as face attribute prediction and affect estimation.
  arXiv  Detail & Related papers  (2022-04-13T05:30:53Z)
• Spectral Clustering with Variance Information for Group Structure Estimation in Panel Data [7.712669451925186]
  We first conduct a local analysis which reveals that the variances of the individual coefficient estimates contain useful information for the estimation of group structure. We then propose a method to estimate unobserved groupings for general panel data models that explicitly account for the variance information.
  arXiv  Detail & Related papers  (2022-01-05T19:16:16Z)
• Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties [62.997667081978825]
  In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class.
  arXiv  Detail & Related papers  (2021-12-15T18:56:39Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Quantitative Evaluations on Saliency Methods: An Experimental Study [6.290238942982972]
  We briefly summarize the status quo of the metrics, including faithfulness, localization, false-positives, sensitivity check, and stability. We conclude that among all the methods we compare, no single explanation method dominates others in all metrics.
  arXiv  Detail & Related papers  (2020-12-31T14:13:30Z)
• Measuring Disentanglement: A Review of Metrics [2.959278299317192]
  Learning to disentangle and represent factors of variation in data is an important problem in AI. We propose a new taxonomy in which all metrics fall into one of three families: intervention-based, predictor-based and information-based. We conduct extensive experiments, where we isolate representation properties to compare all metrics on many aspects.
  arXiv  Detail & Related papers  (2020-12-16T21:28:25Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.