Experiences and Lessons Learned Creating and Validating Concept Inventories for Cybersecurity

• URL: http://arxiv.org/abs/2004.05248v1
• Date: Fri, 10 Apr 2020 22:40:04 GMT
• Title: Experiences and Lessons Learned Creating and Validating Concept Inventories for Cybersecurity
• Authors: Alan T. Sherman, Geoffrey L. Herman, Linda Oliva, Peter A. H. Peterson, Enis Golaszewski, Seth Poulsen, Travis Scheponik, Akshita Gorti
• Abstract summary: Cybersecurity Concept Inventory (CCI) is for students who have recently completed any first course in cybersecurity. The Cybersecurity Curriculum Assessment (CCA) is for students who have recently completed an undergraduate major or track in cybersecurity. Each assessment tool comprises 25 multiple-choice questions (MCQs) of various difficulties that target the same five core concepts.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We reflect on our ongoing journey in the educational Cybersecurity Assessment Tools (CATS) Project to create two concept inventories for cybersecurity. We identify key steps in this journey and important questions we faced. We explain the decisions we made and discuss the consequences of those decisions, highlighting what worked well and what might have gone better. The CATS Project is creating and validating two concept inventories---conceptual tests of understanding---that can be used to measure the effectiveness of various approaches to teaching and learning cybersecurity. The Cybersecurity Concept Inventory (CCI) is for students who have recently completed any first course in cybersecurity; the Cybersecurity Curriculum Assessment (CCA) is for students who have recently completed an undergraduate major or track in cybersecurity. Each assessment tool comprises 25 multiple-choice questions (MCQs) of various difficulties that target the same five core concepts, but the CCA assumes greater technical background. Key steps include defining project scope, identifying the core concepts, uncovering student misconceptions, creating scenarios, drafting question stems, developing distractor answer choices, generating educational materials, performing expert reviews, recruiting student subjects, organizing workshops, building community acceptance, forming a team and nurturing collaboration, adopting tools, and obtaining and using funding. Creating effective MCQs is difficult and time-consuming, and cybersecurity presents special challenges. Because cybersecurity issues are often subtle, where the adversarial model and details matter greatly, it is challenging to construct MCQs for which there is exactly one best but non-obvious answer. We hope that our experiences and lessons learned may help others create more effective concept inventories and assessments in STEM.

Related papers

• Construction and Preliminary Validation of a Dynamic Programming Concept Inventory [0.7389633345370871]
  Concept inventories are standardized assessments that evaluate student understanding of key concepts within academic disciplines. While prevalent across STEM fields, their development lags for advanced computer science topics like dynamic programming (DP) We detail the iterative process used to formulate multiple-choice questions targeting known student misconceptions about DP concepts identified through prior research studies. We conducted a preliminary psychometric validation by administering the D PCI to 172 undergraduate CS students finding our questions to be of appropriate difficulty and effectively discriminating between differing levels of student understanding.
  arXiv  Detail & Related papers  (2024-11-22T01:01:43Z)
• SoK: Unifying Cybersecurity and Cybersafety of Multimodal Foundation Models with an Information Theory Approach [58.93030774141753]
  Multimodal foundation models (MFMs) represent a significant advancement in artificial intelligence. This paper conceptualizes cybersafety and cybersecurity in the context of multimodal learning. We present a comprehensive Systematization of Knowledge (SoK) to unify these concepts in MFMs, identifying key threats.
  arXiv  Detail & Related papers  (2024-11-17T23:06:20Z)
• Using Real-world Bug Bounty Programs in Secure Coding Course: Experience Report [1.099532646524593]
  Training new cybersecurity professionals is a challenging task due to the broad scope of the area. We propose a solution: integrating a real-world bug bounty programme into cybersecurity curriculum. We let students choose to participate in a bug bounty programme as an option for the semester assignment in a secure coding course.
  arXiv  Detail & Related papers  (2024-04-18T09:53:49Z)
• Cybersecurity in Motion: A Survey of Challenges and Requirements for Future Test Facilities of CAVs [11.853500347907826]
  Cooperative Intelligent Transportation Systems (C-ITSs) are at the forefront of this evolution. This paper presents an envisaged Cybersecurity Centre of Excellence (CSCE) designed to bolster research, testing, and evaluation of the cybersecurity of C-ITSs.
  arXiv  Detail & Related papers  (2023-12-22T13:42:53Z)
• Introducing and Interfacing with Cybersecurity -- A Cards Approach [5.269622526990732]
  The National Cyber Security Centre published a Cybersecurity Body of Knowledge (CyBOK) CyBOK contains over 1000 pages of in-depth material and may not be easy to navigate for novice individuals. We propose the use of a playing cards format to provide introductory cybersecurity knowledge.
  arXiv  Detail & Related papers  (2023-07-31T10:01:42Z)
• An Exploratory Study on the Evidence of Hackathons' Role in Solving OSS Newcomers' Challenges [54.56931759953522]
  We aim to understand and discuss the challenges newcomers face when joining an OSS project. We collect evidence on how hackathons were used to address those challenges.
  arXiv  Detail & Related papers  (2023-05-16T15:40:19Z)
• Quiz-based Knowledge Tracing [61.9152637457605]
  Knowledge tracing aims to assess individuals' evolving knowledge states according to their learning interactions. QKT achieves state-of-the-art performance compared to existing methods.
  arXiv  Detail & Related papers  (2023-04-05T12:48:42Z)
• Graph Mining for Cybersecurity: A Survey [61.505995908021525]
  The explosive growth of cyber attacks nowadays, such as malware, spam, and intrusions, caused severe consequences on society. Traditional Machine Learning (ML) based methods are extensively used in detecting cyber threats, but they hardly model the correlations between real-world cyber entities. With the proliferation of graph mining techniques, many researchers investigated these techniques for capturing correlations between cyber entities and achieving high performance.
  arXiv  Detail & Related papers  (2023-04-02T08:43:03Z)
• XSS for the Masses: Integrating Security in a Web Programming Course using a Security Scanner [3.387494280613737]
  Cybersecurity education is an important part of undergraduate computing curricula. Many institutions teach it only in dedicated courses or tracks. An alternative approach is to integrate cybersecurity concepts across non-security courses.
  arXiv  Detail & Related papers  (2022-04-26T16:20:36Z)
• Dos and Don'ts of Machine Learning in Computer Security [74.1816306998445]
  Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance. We identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible.
  arXiv  Detail & Related papers  (2020-10-19T13:09:31Z)
• Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain [58.30296637276011]
  This paper summarizes the latest research on adversarial attacks against security solutions based on machine learning techniques. It is the first to discuss the unique challenges of implementing end-to-end adversarial attacks in the cyber security domain.
  arXiv  Detail & Related papers  (2020-07-05T18:22:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.