Methodological Variation in Studying Staff and Student Perceptions of AI

• URL: http://arxiv.org/abs/2602.11158v1
• Date: Mon, 22 Dec 2025 11:04:46 GMT
• Title: Methodological Variation in Studying Staff and Student Perceptions of AI
• Authors: Juliana Gerard, Morgan Macleod, Kelly Norwood, Aisling Reid, Muskaan Singh,
• Abstract summary: We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure.<n>To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups.<n>We show that different analyses can produce different results - for a single data source.
• Score: 1.701796716399534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we compare methodological approaches for comparing student and staff perceptions, and ask: how much do these measures vary across different approaches? We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure, or with a mixed methods approach that compares two distinct data sources - e.g. a quantitative questionnaire with qualitative comments. To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups. We conduct two analyses for each data source: with a sentiment and stance analysis, we measure overall negativity/positivity of the comments and focus group conversations, respectively. Meanwhile, word clouds from the comments and a thematic analysis of the focus groups provide further detail on the content of this qualitative data - particularly the thematic analysis, which includes both similarities and differences between students and staff. We show that different analyses can produce different results - for a single data source. This variation stems from the construct being evaluated - an overall measure of positivity/negativity can produce a different picture from more detailed content-based analyses. We discuss the implications of this variation for institutional contexts, and for the comparisons from previous studies.

Related papers

• An unsupervised learning approach to evaluate questionnaire data -- what one can learn from violations of measurement invariance [2.4762962548352467]
  This paper promotes an unsupervised learning-based approach to such research data. It works in three phases: data preparation, clustering of questionnaires, and measuring similarity based on the obtained clustering and the properties of each group. It provides a natural comparison between groups and a natural description of the response patterns of the groups.
  arXiv  Detail & Related papers  (2023-12-11T11:31:41Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• A Fair and Comprehensive Comparison of Multimodal Tweet Sentiment Analysis Methods [3.8142537449670963]
  We present a comprehensive experimental evaluation and comparison with six state-of-the-art methods. Results are presented for two different publicly available benchmark datasets of tweets and corresponding images.
  arXiv  Detail & Related papers  (2021-06-16T14:44:48Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation [3.624563211765782]
  We show that best choice of evaluation method can vary from one aspect to another. We show that the total number of annotators can have a strong impact on study power. Current statistical analysis methods can inflate type I error rates up to eight-fold.
  arXiv  Detail & Related papers  (2021-01-27T10:14:15Z)
• Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale [12.883536911500062]
  We study claim quality assessment irrespective of discussed aspects by comparing different revisions of the same claim. We propose two tasks: assessing which claim of a revision pair is better, and ranking all versions of a claim by quality.
  arXiv  Detail & Related papers  (2021-01-25T17:32:04Z)
• Through the Data Management Lens: Experimental Analysis and Evaluation of Fair Classification [75.49600684537117]
  Data management research is showing an increasing presence and interest in topics related to data and algorithmic fairness. We contribute a broad analysis of 13 fair classification approaches and additional variants, over their correctness, fairness, efficiency, scalability, and stability. Our analysis highlights novel insights on the impact of different metrics and high-level approach characteristics on different aspects of performance.
  arXiv  Detail & Related papers  (2021-01-18T22:55:40Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning [141.2562447971]
  We propose to solve a few-shot (or low-shot) visual reasoning problem, by resorting to analogical reasoning. We extract structural relationships between elements in both domains, and enforce them to be as similar as possible with analogical learning. We validate our method on RAVEN dataset, on which it outperforms state-of-the-art method, with larger gains when the training data is scarce.
  arXiv  Detail & Related papers  (2020-07-23T14:00:34Z)
• On the Ambiguity of Rank-Based Evaluation of Entity Alignment or Link Prediction Methods [27.27230441498167]
  We take a closer look at the evaluation of two families of methods for enriching information from knowledge graphs: Link Prediction and Entity Alignment. In particular, we demonstrate that all existing scores can hardly be used to compare results across different datasets. We show that this leads to various problems in the interpretation of results, which may support misleading conclusions.
  arXiv  Detail & Related papers  (2020-02-17T12:26:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.