Zheng, E. orcid.org/0000-0001-8759-3643, Fu, H. orcid.org/0000-0002-1534-9374, Thelwall, M. orcid.org/0000-0001-6065-205X et al. (1 more author) (2025) Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models. Journal of the Association for Information Science and Technology. ISSN: 2330-1635
Abstract
Timely detection of problematic research is essential for safeguarding scientific integrity. To explore whether social media commentary can serve as an early indicator of potentially problematic articles, this study analyzed 3815 tweets referencing 604 retracted articles and 3373 tweets referencing 668 comparable non-retracted articles. Tweets critical of the articles were identified through both human annotation and large language models (LLMs). Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction, compared to only 1.5% of non-retracted articles, highlighting the potential of tweets as early warning signals of retraction. However, critical tweets identified by LLMs (GPT-4o mini, Gemini 2.0 Flash-Lite, and Claude 3.5 Haiku) only partially aligned with human annotation, suggesting that fully automated monitoring of post-publication discourse should be applied with caution. A human–AI collaborative approach may offer a more reliable and scalable alternative, with human expertise helping to filter out tweets critical of issues unrelated to the research integrity of the articles. Overall, this study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2025 The Author(s). Journal of the Association for Information Science and Technology published by Wiley Periodicals LLC on behalf of Association for Information Science and Technology. This is an open access article under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits use, distribution and reproduction in any medium, provided the original work is properly cited. |
Dates: |
|
Institution: | The University of Sheffield |
Academic Units: | ?? Sheffield.IJC ?? The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield) |
Funding Information: | Funder Grant number EUROPEAN MEDIA AND INFORMATION FUND UNSPECIFIED |
Date Deposited: | 29 Sep 2025 13:15 |
Last Modified: | 29 Sep 2025 13:21 |
Status: | Published online |
Publisher: | Wiley |
Refereed: | Yes |
Identification Number: | 10.1002/asi.70028 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:232306 |