ChatGPT ranking of business and management journals with article quality scores

Purpose

Business and management journal rankings are controversial but influential for scholars seeking publishing venues and for appointment, tenure and promotion committees needing to evaluate applicants’ work. Whilst some prominent rankings are citation-based, others are constructed by field experts. This article assesses whether large language models (LLMs) can provide credible new business and management journal rankings.

Design/methodology/approach

Based on mean ChatGPT 4o-mini scores for business and management articles published between 2014 and 2020 and submitted to the UK Research Excellence Framework (REF) 2021, ChatGPT-based rankings were compared with expert rankings from the Australian Business Deans Council (ABDC) and the Chartered Association of Business Schools (CABS), weighted normalised citation-based rankings, mean REF citation scores and mean REF departmental quality scores.

Findings

For the 43 journals with at least 50 articles and data from all six sources, the ChatGPT scores correlated more strongly with expert rankings (CABS: 0.438 and ABCD: 0.510) than any of the citation rankings except Scimago Journal Rank (SJR) for one of the two (CABS: 0.664 and ABCD: 0.360). Journal scores calculated from REF departmental quality score rankings had the highest Spearman correlations with the established rankings, however (CABS: 0.717 and ABCD: 0.583). If rankings based on REF departmental quality scores are taken as optimal, then ChatGPT scores have the highest correlation with this (0.830), greater even than with the two expert rankings.

Originality/value

ChatGPT-based journal quality scores are a plausible new ranking mechanism for business and management journals and may be superior to citation-based rankings in some cases, potentially providing more current, finer-grained and cheaper results.