Thelwall, M., Lehtisaari, M., Katsirea, E. et al. (2 more authors) (2025) Does ChatGPT ignore article retractions and other reliability concerns? Learned Publishing, 38 (4). e2018. ISSN: 0953-1513
Abstract
Large language models (LLMs) like ChatGPT seem to be increasingly used for information seeking and analysis, including to support academic literature reviews. To test whether the results might sometimes include retracted research, we identified 217 retracted or otherwise concerning academic studies with high altmetric scores and asked ChatGPT 4o-mini to evaluate their quality 30 times each. Surprisingly, none of its 6510 reports mentioned that the articles were retracted or had relevant errors, and it gave 190 relatively high scores (world leading, internationally excellent, or close). The 27 articles with the lowest scores were mostly accused of being weak, although the topic (but not the article) was described as controversial in five cases (e.g., about hydroxychloroquine for COVID-19). In a follow-up investigation, 61 claims were extracted from retracted articles from the set, and ChatGPT 4o-mini was asked 10 times whether each was true. It gave a definitive yes or a positive response two-thirds of the time, including for at least one statement that had been shown to be false over a decade ago. The results therefore emphasise, from an academic knowledge perspective, the importance of verifying information from LLMs when using them for information seeking or analysis.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2025 The Author(s). Learned Publishing published by John Wiley & Sons Ltd on behalf of ALPSP. This is an open access article under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits use, distribution and reproduction in any medium, provided the original work is properly cited. |
Keywords: | altmetrics; ChatGPT; large language models; retraction; science communication |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Funding Information: | Funder Grant number EUROPEAN MEDIA AND INFORMATION FUND UNSPECIFIED |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 07 Aug 2025 15:46 |
Last Modified: | 07 Aug 2025 15:46 |
Status: | Published |
Publisher: | Wiley |
Refereed: | Yes |
Identification Number: | 10.1002/leap.2018 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:229642 |