Thelwall, M. orcid.org/0000-0001-6065-205X and Kurt, Z. (Accepted: 2025) Research evaluation with ChatGPT: is it age, country, length, or field biased? Scientometrics. ISSN: 0138-9130 (In Press)
Abstract
Some research now suggests that ChatGPT can estimate the quality of journal articles from their titles and abstracts. This has created the possibility to use ChatGPT quality scores, perhaps alongside citation-based formulae, to support peer review for research evaluation. Nevertheless, ChatGPT’s internal processes are effectively opaque, despite it writing a report to support its scores, and its biases are unknown. This article investigates whether publication date and field are biasing factors. Based on submitting a monodisciplinary journal-balanced set of 117,650 articles from 26 fields published in the years 2003, 2008, 2013, 2018 and 2023 to ChatGPT 4o-mini, the results show that average scores increased over time, and this was not due to author nationality or title and abstract length changes. The results also varied substantially between fields, and first author countries. In addition, articles with longer abstracts tended to receive higher scores, mostly due to such articles tending to be better (e.g., more likely to be in higher impact journals) but also partly due to ChatGPT analysing more text. For the most accurate research quality evaluation results from ChatGPT, it is important to normalise ChatGPT scores for field and year and check for anomalies caused by sets of articles with short abstracts.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © Akadémiai Kiadó Zrt 2025 |
Keywords: | ChatGPT; research impact; publication date; research excellence framework |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Funding Information: | Funder Grant number UK RESEARCH AND INNOVATION UKRI1079 |
Depositing User: | Symplectic Sheffield |
Date Deposited: | 31 Jul 2025 09:24 |
Last Modified: | 31 Jul 2025 09:24 |
Status: | In Press |
Publisher: | Springer |
Refereed: | Yes |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:229531 |
Download
Filename: ChatGPT biases preprint.pdf
