Do Large Language Models know basic facts about journal articles?

Purpose - There is increasing use of Large Language Models (LLMs) in information science, including to evaluate academic journal articles. Despite this, it is unclear whether they “know” about articles in the sense of being able to answer simple questions about individual papers without web searches.

Design/methodology/approach – Four questions were asked ofChatGPT 4o-mini about 64,055 academic journal articles (excluding reviews) from 2021, identified by their titles and abstracts, with uncited and highly cited articles also assessed by ChatGPT 4.1 and five open weights LLMs.

Findings – The results were mostly incorrect, even for the most cited articles from that year. In particular, ChatGPT 4o-mini and the open weights LLMs had almost no knowledge of an article’s first author affiliation, rarely knew the publishing journal and usually guessed the publication year wrong, although ChatGPT 4o-mini was 42% correct for Physical Review B. Even ChatGPT 4.1 could only identify a small majority of the journals for the top cited papers of the year.

Practical implications – Smaller LLMs’ lack of basic knowledge about articles suggests that when they are asked to evaluate them without web searches, they will rarely cheat by eliciting citation information or journal reputation but will instead answer based on the article text because they may not associate online criticisms with individual articles.

Originality/value – This is the first investigation of the ability of LLMs to recall basic facts about journal articles.