Gibbons, C, Richards, S orcid.org/0000-0003-1416-0569, Valdeas, JM et al. (1 more author) (2017) Supervised machine learning algorithms can classify open-text feedback of doctor performance with human-level accuracy. Journal of Medical Internet Research, 19 (3). e65. ISSN 1439-4456
Abstract
Background: Machine learning techniques may be an effective and efficient way to classify open-text reports on doctor’s activity for the purposes of quality assurance, safety, and continuing professional development. Objective: The objective of the study was to evaluate the accuracy of machine learning algorithms trained to classify open-text reports of doctor performance and to assess the potential for classifications to identify significant differences in doctors’ professional performance in the United Kingdom. Methods: We used 1636 open-text comments (34,283 words) relating to the performance of 548 doctors collected from a survey of clinicians’ colleagues using the General Medical Council Colleague Questionnaire (GMC-CQ). We coded 77.75% (1272/1636) of the comments into 5 global themes (innovation, interpersonal skills, popularity, professionalism, and respect) using a qualitative framework. We trained 8 machine learning algorithms to classify comments and assessed their performance using several training samples. We evaluated doctor performance using the GMC-CQ and compared scores between doctors with different classifications using t tests. Results: Individual algorithm performance was high (range F score=.68 to .83). Interrater agreement between the algorithms and the human coder was highest for codes relating to “popular” (recall=.97), “innovator” (recall=.98), and “respected” (recall=.87) codes and was lower for the “interpersonal” (recall=.80) and “professional” (recall=.82) codes. A 10-fold cross-validation demonstrated similar performance in each analysis. When combined together into an ensemble of multiple algorithms, mean human-computer interrater agreement was .88. Comments that were classified as “respected,” “professional,” and “interpersonal” related to higher doctor scores on the GMC-CQ compared with comments that were not classified (P<.05). Scores did not vary between doctors who were rated as popular or innovative and those who were not rated at all (P>.05). Conclusions: Machine learning algorithms can classify open-text feedback of doctor performance into multiple themes derived by human raters with high performance. Colleague open-text comments that signal respect, professionalism, and being interpersonal may be key indicators of doctor’s performance.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | ©Chris Gibbons, Suzanne Richards, Jose Maria Valderas, John Campbell. Originally published in the Journal of Medical Internet Research (http://www .jmir.org), 15.03.2017. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.or g/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www .jmir.org/, as well as this copyright and license information must be included. |
Keywords: | machine learning; surveys and questionnaires; feedback; data mining; work performance |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Medicine and Health (Leeds) > School of Medicine (Leeds) > Leeds Institute of Health Sciences (Leeds) > Academic Unit of Primary Care (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 23 Mar 2017 09:55 |
Last Modified: | 05 Oct 2017 16:26 |
Published Version: | http://www.jmir.org/2017/3/e65/ |
Status: | Published |
Publisher: | JMIR Publications |
Identification Number: | 10.2196/jmir.6533 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:114024 |