Predicting respondent difficulty in web surveys: A machine-learning approach based on mouse movement features

• URL: http://arxiv.org/abs/2011.06916v1
• Date: Thu, 5 Nov 2020 10:54:33 GMT
• Title: Predicting respondent difficulty in web surveys: A machine-learning approach based on mouse movement features
• Authors: Amanda Fern\'andez-Fontelo, Pascal J. Kieslich, Felix Henninger, Frauke Kreuter and Sonja Greven
• Abstract summary: This paper explores the predictive value of mouse-tracking data with regard to respondents' difficulty. We use data from a survey on respondents' employment history and demographic information. We develop a personalization method that adjusts for respondents' baseline mouse behavior and evaluate its performance.
• Score: 3.6944296923226316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A central goal of survey research is to collect robust and reliable data from respondents. However, despite researchers' best efforts in designing questionnaires, respondents may experience difficulty understanding questions' intent and therefore may struggle to respond appropriately. If it were possible to detect such difficulty, this knowledge could be used to inform real-time interventions through responsive questionnaire design, or to indicate and correct measurement error after the fact. Previous research in the context of web surveys has used paradata, specifically response times, to detect difficulties and to help improve user experience and data quality. However, richer data sources are now available, in the form of the movements respondents make with the mouse, as an additional and far more detailed indicator for the respondent-survey interaction. This paper uses machine learning techniques to explore the predictive value of mouse-tracking data with regard to respondents' difficulty. We use data from a survey on respondents' employment history and demographic information, in which we experimentally manipulate the difficulty of several questions. Using features derived from the cursor movements, we predict whether respondents answered the easy or difficult version of a question, using and comparing several state-of-the-art supervised learning methods. In addition, we develop a personalization method that adjusts for respondents' baseline mouse behavior and evaluate its performance. For all three manipulated survey questions, we find that including the full set of mouse movement features improved prediction performance over response-time-only models in nested cross-validation. Accounting for individual differences in mouse movements led to further improvements.

Related papers

• Deep Learning-Based Object Pose Estimation: A Comprehensive Survey [73.74933379151419]
  We discuss the recent advances in deep learning-based object pose estimation. Our survey also covers multiple input data modalities, degrees-of-freedom of output poses, object properties, and downstream tasks.
  arXiv  Detail & Related papers  (2024-05-13T14:44:22Z)
• Qsnail: A Questionnaire Dataset for Sequential Question Generation [76.616068047362]
  We present the first dataset specifically constructed for the questionnaire generation task, which comprises 13,168 human-written questionnaires. We conduct experiments on Qsnail, and the results reveal that retrieval models and traditional generative models do not fully align with the given research topic and intents. Despite enhancements through the chain-of-thought prompt and finetuning, questionnaires generated by language models still fall short of human-written questionnaires.
  arXiv  Detail & Related papers  (2024-02-22T04:14:10Z)
• Continually Improving Extractive QA via Human Feedback [59.49549491725224]
  We study continually improving an extractive question answering (QA) system via human user feedback. We conduct experiments involving thousands of user interactions under diverse setups to broaden the understanding of learning from feedback over time.
  arXiv  Detail & Related papers  (2023-05-21T14:35:32Z)
• Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets [73.2096288987301]
  We propose a simple approach that uses a small amount of downstream expert data to selectively query relevant behaviors from an offline, unlabeled dataset. We observe that our method learns to query only the relevant transitions to the task, filtering out sub-optimal or task-irrelevant data. Our simple querying approach outperforms more complex goal-conditioned methods by 20% across simulated and real robotic manipulation tasks from images.
  arXiv  Detail & Related papers  (2023-04-18T05:42:53Z)
• When Respondents Don't Care Anymore: Identifying the Onset of Careless Responding [0.0]
  We propose a novel method for identifying the onset of careless responding for each participant. It is based on combined measurements of multiple dimensions in which carelessness may manifest. It is highly flexible, based on machine learning, and provides statistical guarantees on its performance.
  arXiv  Detail & Related papers  (2023-03-13T15:10:30Z)
• Classification ensembles for multivariate functional data with application to mouse movements in web surveys [3.6944296923226316]
  We propose new ensemble models for multivariate functional data classification as combinations of weak learners. We apply these ensemble models to identify respondents' difficulty with survey questions, with the aim to improve survey data quality.
  arXiv  Detail & Related papers  (2022-05-26T14:12:24Z)
• Open vs Closed-ended questions in attitudinal surveys -- comparing, combining, and interpreting using natural language processing [3.867363075280544]
  Topic Modeling could significantly reduce the time to extract information from open-ended responses. Our research uses Topic Modeling to extract information from open-ended questions and compare its performance with closed-ended responses.
  arXiv  Detail & Related papers  (2022-05-03T06:01:03Z)
• Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices [0.0]
  The research directions of QA field are analyzed based on the type of question, answer type, source of evidence-answer, and modeling approach. This detailed followed by open challenges of the field like automatic question generation, similarity detection and, low resource availability for a language.
  arXiv  Detail & Related papers  (2021-12-07T08:53:40Z)
• Mining Implicit Relevance Feedback from User Behavior for Web Question Answering [92.45607094299181]
  We make the first study to explore the correlation between user behavior and passage relevance. Our approach significantly improves the accuracy of passage ranking without extra human labeled data. In practice, this work has proved effective to substantially reduce the human labeling cost for the QA service in a global commercial search engine.
  arXiv  Detail & Related papers  (2020-06-13T07:02:08Z)
• Review-guided Helpful Answer Identification in E-commerce [38.276241153439955]
  Product-specific community question answering platforms can greatly help address the concerns of potential customers. The user-provided answers on such platforms often vary a lot in their qualities. Helpfulness votes from the community can indicate the overall quality of the answer, but they are often missing.
  arXiv  Detail & Related papers  (2020-03-13T11:34:29Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.