Crafting Tomorrow's Evaluations: Assessment Design Strategies in the Era of Generative AI

• URL: http://arxiv.org/abs/2405.01805v1
• Date: Fri, 3 May 2024 01:28:21 GMT
• Title: Crafting Tomorrow's Evaluations: Assessment Design Strategies in the Era of Generative AI
• Authors: Rajan Kadel, Bhupesh Kumar Mishra, Samar Shailendra, Samia Abid, Maneeha Rani, Shiva Prasad Mahato,
• Abstract summary: GenAI has had an intense impact on education, significantly disrupting the assessment design and evaluation methodologies. There are several concerns primarily centred around academic integrity, authenticity, equity of access, assessment evaluation methodology, and feedback. This article discusses the challenges, and opportunities that need to be addressed for the assessment design and evaluation.
• Score: 0.02638878351659022
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GenAI has gained the attention of a myriad of users in almost every profession. Its advancement has had an intense impact on education, significantly disrupting the assessment design and evaluation methodologies. Despite the potential benefits and possibilities of GenAI in the education sector, there are several concerns primarily centred around academic integrity, authenticity, equity of access, assessment evaluation methodology, and feedback. Consequently, academia is encountering challenges in assessment design that are essential to retaining academic integrity in the age of GenAI. In this article, we discuss the challenges, and opportunities that need to be addressed for the assessment design and evaluation. The article also highlights the importance of clear policy about the usage of GenAI in completing assessment tasks, and also in design approaches to ensure academic integrity and subject learning. Additionally, this article also provides assessment categorisation based on the use of GenAI to cultivate knowledge among students and academic professionals. It also provides information on the skills necessary to formulate and articulate problems and evaluate the task, enabling students and academics to effectively utilise GenAI tools.

Related papers

• Dimensions of Generative AI Evaluation Design [51.541816010127256]
  We propose a set of general dimensions that capture critical choices involved in GenAI evaluation design. These dimensions include the evaluation setting, the task type, the input source, the interaction style, the duration, the metric type, and the scoring method.
  arXiv  Detail & Related papers  (2024-11-19T18:25:30Z)
• What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation [57.550045763103334]
  evaluating a story can be more challenging than other generation evaluation tasks. We first summarize existing storytelling tasks, including text-to-text, visual-to-text, and text-to-visual. We propose a taxonomy to organize evaluation metrics that have been developed or can be adopted for story evaluation.
  arXiv  Detail & Related papers  (2024-08-26T20:35:42Z)
• Framework for Adoption of Generative Artificial Intelligence (GenAI) in Education [0.032771631221674334]
  This study focuses on examining the university education system and assessing the opportunities and challenges related to incorporating GenAI in teaching and learning. It identifies a gap and the absence of a comprehensive framework that obstructs the effective integration of GenAI within the academic environment.
  arXiv  Detail & Related papers  (2024-07-21T12:02:28Z)
• Understanding Student and Academic Staff Perceptions of AI Use in Assessment and Feedback [0.0]
  The rise of Artificial Intelligence (AI) and Generative Artificial Intelligence (GenAI) in higher education necessitates assessment reform. This study addresses a critical gap by exploring student and academic staff experiences with AI and GenAI tools. An online survey collected data from 35 academic staff and 282 students across two universities in Vietnam and one in Singapore.
  arXiv  Detail & Related papers  (2024-06-22T10:25:01Z)
• ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models [53.00812898384698]
  We argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. We propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.
  arXiv  Detail & Related papers  (2024-05-28T22:45:28Z)
• Higher education assessment practice in the era of generative AI tools [0.37282630026096586]
  This study experimented using three assessment instruments from data science, data analytics, and construction management disciplines. Our findings revealed that GenAI tools exhibit subject knowledge, problem-solving, analytical, critical thinking, and presentation skills. Based on our findings, we made recommendations on how AI tools can be utilised for teaching and learning in HE.
  arXiv  Detail & Related papers  (2024-04-01T10:43:50Z)
• The AI Assessment Scale (AIAS) in action: A pilot implementation of GenAI supported assessment [0.0]
  The rapid adoption of Generative Artificial Intelligence (GenAI) technologies in higher education has raised concerns about academic integrity, assessment practices, and student learning. This paper presents the findings of a pilot study conducted at British University Vietnam (BUV) exploring the implementation of the Artificial Intelligence Assessment Scale (AIAS) The AIAS consists of five levels, ranging from 'No AI' to 'Full AI', enabling educators to design assessments that focus on areas requiring human input and critical thinking.
  arXiv  Detail & Related papers  (2024-03-15T08:00:02Z)
• The AI Assessment Scale (AIAS): A Framework for Ethical Integration of Generative AI in Educational Assessment [0.0]
  We outline a practical, simple, and sufficiently comprehensive tool to allow for the integration of GenAI tools into educational assessment. The AI Assessment Scale (AIAS) empowers educators to select the appropriate level of GenAI usage in assessments. By adopting a practical, flexible approach, the AIAS can form a much-needed starting point to address the current uncertainty and anxiety regarding GenAI in education.
  arXiv  Detail & Related papers  (2023-12-12T09:08:36Z)
• Evaluating General-Purpose AI with Psychometrics [43.85432514910491]
  We discuss the need for a comprehensive and accurate evaluation of general-purpose AI systems such as large language models. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation.
  arXiv  Detail & Related papers  (2023-10-25T05:38:38Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
  The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal. In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
  arXiv  Detail & Related papers  (2021-01-19T12:23:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.