Quality meets Diversity: A Model-Agnostic Framework for Computerized Adaptive Testing

• URL: http://arxiv.org/abs/2101.05986v1
• Date: Fri, 15 Jan 2021 06:48:50 GMT
• Title: Quality meets Diversity: A Model-Agnostic Framework for Computerized Adaptive Testing
• Authors: Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, Shijin Wang
• Abstract summary: Computerized Adaptive Testing (CAT) is emerging as a promising testing application in many scenarios. We propose a novel framework, namely Model-Agnostic Adaptive Testing (MAAT) for CAT solution.
• Score: 60.38182654847399
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computerized Adaptive Testing (CAT) is emerging as a promising testing application in many scenarios, such as education, game and recruitment, which targets at diagnosing the knowledge mastery levels of examinees on required concepts. It shows the advantage of tailoring a personalized testing procedure for each examinee, which selects questions step by step, depending on her performance. While there are many efforts on developing CAT systems, existing solutions generally follow an inflexible model-specific fashion. That is, they need to observe a specific cognitive model which can estimate examinee's knowledge levels and design the selection strategy according to the model estimation. In this paper, we study a novel model-agnostic CAT problem, where we aim to propose a flexible framework that can adapt to different cognitive models. Meanwhile, this work also figures out CAT solution with addressing the problem of how to generate both high-quality and diverse questions simultaneously, which can give a comprehensive knowledge diagnosis for each examinee. Inspired by Active Learning, we propose a novel framework, namely Model-Agnostic Adaptive Testing (MAAT) for CAT solution, where we design three sophisticated modules including Quality Module, Diversity Module and Importance Module. Extensive experimental results on two real-world datasets clearly demonstrate that our MAAT can support CAT with guaranteeing both quality and diversity perspectives.

Related papers

• UKTF: Unified Knowledge Tracing Framework for Subjective and Objective Assessments [3.378008889662775]
  Knowledge tracing technology can establish knowledge state models based on learners' historical answer data. This study proposes a unified knowledge tracing model that integrates both objective and subjective test questions.
  arXiv  Detail & Related papers  (2024-11-08T04:58:19Z)
• A Survey of Models for Cognitive Diagnosis: New Developments and Future Directions [66.40362209055023]
  This paper aims to provide a survey of current models for cognitive diagnosis, with more attention on new developments using machine learning-based methods. By comparing the model structures, parameter estimation algorithms, model evaluation methods and applications, we provide a relatively comprehensive review of the recent trends in cognitive diagnosis models.
  arXiv  Detail & Related papers  (2024-07-07T18:02:00Z)
• Survey of Computerized Adaptive Testing: A Machine Learning Perspective [66.26687542572974]
  Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method.
  arXiv  Detail & Related papers  (2024-03-31T15:09:47Z)
• Testing for Fault Diversity in Reinforcement Learning [13.133263651395865]
  We argue that policy testing should not find as many failures as possible (e.g., inputs that trigger similar car crashes) but rather aim at revealing as informative and diverse faults as possible in the model. We show that QD optimisation, while being conceptually simple and generally applicable, finds effectively more diverse faults in the decision model.
  arXiv  Detail & Related papers  (2024-03-22T09:46:30Z)
• SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models [71.78800549517298]
  Continual learning (CL) ability is vital for deploying large language models (LLMs) in the dynamic world. Existing methods devise the learning module to acquire task-specific knowledge with parameter-efficient tuning (PET) block and the selection module to pick out the corresponding one for the testing input. We propose a novel Shared Attention Framework (SAPT) to align the PET learning and selection via the Shared Attentive Learning & Selection module.
  arXiv  Detail & Related papers  (2024-01-16T11:45:03Z)
• Designing Novel Cognitive Diagnosis Models via Evolutionary Multi-Objective Neural Architecture Search [13.9289351255891]
  We propose to automatically design novel cognitive diagnosis models by evolutionary multi-objective neural architecture search (NAS) Experiments on two real-world datasets demonstrate that the cognitive diagnosis models searched by the proposed approach exhibit significantly better performance than existing models and also hold as good interpretability as human-designed models.
  arXiv  Detail & Related papers  (2023-07-10T09:09:26Z)
• Zero-shot Model Diagnosis [80.36063332820568]
  A common approach to evaluate deep learning models is to build a labeled test set with attributes of interest and assess how well it performs. This paper argues the case that Zero-shot Model Diagnosis (ZOOM) is possible without the need for a test set nor labeling.
  arXiv  Detail & Related papers  (2023-03-27T17:59:33Z)
• MetaQA: Combining Expert Agents for Multi-Skill Question Answering [49.35261724460689]
  We argue that despite the promising results of multi-dataset models, some domains or QA formats might require specific architectures. We propose to combine expert agents with a novel, flexible, and training-efficient architecture that considers questions, answer predictions, and answer-prediction confidence scores.
  arXiv  Detail & Related papers  (2021-12-03T14:05:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.