Klyshbekova, M. orcid.org/0000-0002-7463-356X and Abbott, P. orcid.org/0000-0002-4680-0754 (2024) ChatGPT and assessment in higher education: a magic wand or a disruptor? Electronic Journal of e-Learning, 22 (2). ISSN 1479-4403
Abstract
There is a current debate about the extent to which ChatGPT, a natural language AI chatbot, can disrupt processes in higher education settings. The chatbot is capable of not only answering queries in a human-like way within seconds but can also provide long tracts of texts which can be in the form of essays, emails, and coding. In this study, in the context of higher education settings, by adopting an experimental design approach, we applied ChatGPT-3 to a traditional form of assessment to determine its capabilities and limitations. Specifically, we tested its ability to produce an essay on a topic of our choice, created a rubric, and assessed the produced work in accordance with the designed rubric. We then evaluated the chatbot’s work by assessing ChatGPT’s application of its rubric according to a modified version of Paul’s (2005) Intellectual Standards rubric. Using Christensen et al.’s (2015) framework on disruptive innovations, our study found that ChatGPT was capable of completing the set tasks competently, quickly, and easily, like a “magic wand”. However, our findings also challenge the extent to which all of the ChatGPT’s demonstrated capabilities can disrupt this traditional form of assessment, given that there are aspects of its construction and evaluation that the technology is not yet able to replicate as a human expert would. These limitations of the chatbot can provide us with an opportunity for addressing vulnerabilities in traditional forms of assessment in higher education that are subject to academic integrity issues posed by this form of AI. We conclude the article with implications for teachers and higher education institutions by urging them to reconsider and revisit their practices when it comes to assessment.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2024 Maira Klyshbekova, Pamela Abbott. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License: https://creativecommons.org/licenses/by-nc-nd/4.0/ |
Keywords: | ChatGPT; Higher education; Artificial intelligence (AI); Assessment; Academic integrity |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) The University of Sheffield > Faculty of Social Sciences (Sheffield) > School of Education (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 20 Sep 2024 11:01 |
Last Modified: | 20 Sep 2024 11:01 |
Published Version: | http://dx.doi.org/10.34190/ejel.21.5.3114 |
Status: | Published |
Publisher: | Academic Conferences International Ltd |
Refereed: | Yes |
Identification Number: | 10.34190/ejel.21.5.3114 |
Related URLs: | |
Sustainable Development Goals: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:217458 |
Download
Filename: 3.+EL+3114+Final++Klyshbekova+and+Abbott+typeset+060224.pdf
Licence: CC-BY 4.0