ChatGPT estimates of the quality of published conference papers from their titles and abstracts

Purpose: Although citation-based indicators are sometimes used to help evaluate the quality of papers in conference-based fields, conferences seem to be less systematically indexed in the major citation indexes, and the value of citation counts as research quality indicators for them is unknown. In response, this article investigates whether ChatGPT might provide a suitable alternative.

Design/methodology/approach: ChatGPT was used to assign a quality score to conference papers from 2020 from the 23 narrow fields that are partly conference-based in the sense of having at least 30% conference papers indexed in Scopus. The results were compared with citation counts and conference rankings.

Findings: ChatGPT research quality score predictions based on paper titles and abstracts alone correlated positively and statistically significantly with citation rates in all fields at the paper level and mostly at the conference level. Expert-based citation-informed conference rankings conformed slightly more closely with geometric mean citation rates than with mean ChatGPT scores.

Research limitations/implications: No direct measure of paper research quality was used.

Originality/value: Whilst the evidence tends to support the value of both citations and ChatGPT as research quality indicators, it suggests that citations may be better, at least for older research, and that ChatGPT is a reasonable alternative for research that is too new to have attracted many citations, at least in conference-based fields.