Williams, S.C. orcid.org/0000-0003-1770-1797, Starup-Hansen, J., Funnell, J.P. et al. (6 more authors) (2024) Can ChatGPT outperform a neurosurgical trainee? A prospective comparative study. British Journal of Neurosurgery. ISSN 0268-8697
Abstract
Purpose
This study aimed to compare the performance of ChatGPT, a large language model (LLM), with human neurosurgical applicants in a neurosurgical national selection interview, to assess the potential of artificial intelligence (AI) and LLMs in healthcare and provide insights into their integration into the field.
Methods
In a prospective comparative study, a set of neurosurgical national selection-style interview questions were asked to eight human participants and ChatGPT in an online interview. All participants were doctors currently practicing in the UK who had applied for a neurosurgical National Training Number. Interviews were recorded, anonymised, and scored by three neurosurgical consultants with experience as interviewers for national selection. Answers provided by ChatGPT were used as a template for a virtual interview. Interview transcripts were subsequently scored by neurosurgical consultants using criteria utilised in real national selection interviews. Overall interview score and subdomain scores were compared between human participants and ChatGPT.
Results
For overall score, ChatGPT fell behind six human competitors and did not achieve a mean score higher than any individuals who achieved training positions. Several factors, including factual inaccuracies and deviations from expected structure and style may have contributed to ChatGPT's underperformance.
Conclusions
LLMs such as ChatGPT have huge potential for integration in healthcare. However, this study emphasises the need for further development to address limitations and challenges. While LLMs have not surpassed human performance yet, collaboration between humans and AI systems holds promise for the future of healthcare.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2024 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
Keywords: | AI; Artificial intelligence; ChatGPT; healthcare; large language model; natural language processing; neurosurgery |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Medicine, Dentistry and Health (Sheffield) > The Medical School (Sheffield) |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 22 Mar 2024 15:39 |
Last Modified: | 22 Mar 2024 15:39 |
Status: | Published online |
Publisher: | Informa UK Limited |
Refereed: | Yes |
Identification Number: | 10.1080/02688697.2024.2308222 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:210730 |
Download
Filename: Can ChatGPT outperform a neurosurgical trainee A prospective comparative study.pdf
Licence: CC-BY 4.0