Which abstract stylistic features fool ChatGPT research evaluations?

Large Language Models (LLMs) have the potential to be used to support research evaluation and have a moderate capability to estimate the research quality of a journal article from its title and abstract. This paper assesses whether there are abstract textual features unrelated to research quality that may influence ChatGPT’s scores. Using a dataset of 99,277 journal articles submitted to the UK-wide Research Excellence Framework (REF) 2021 assessments, we calculated several readability indicators from abstracts and correlated them with ChatGPT scores and departmental REF scores (all fields) and individual article scores (health and life sciences). From the results, linguistic complexity and length correlated positively with both ChatGPT scores and a departmental average proxy for REF expert scores in most broad fields (Units of Assessment). They were also more strongly associated with ChatGPT research quality scores than with the proxy for REF expert scores in many subject areas, although this may have been due to the influence of the averaging process in the proxy for some. In the health and life sciences, where individual article scores were available, syllables per word and words per sentence had stronger correlations with individual ChatGPT scores than with individual article quality scores. Although cause-and-effect was not tested, these results suggest that ChatGPT may be more likely than human experts to reward linguistic complexity, with a potential bias towards longer and less readable abstracts in many fields. The apparent preference of LLMs for complex language is an undesirable feature for practical applications of LLMs for research quality evaluation, unless solutions can be found.