What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield

Background: There is a growing interest in using and analyzing altmetric data for quantifying the impact of research, especially societal impact (Bornmann, 2014, Thelwall et al., 2016, Haunschild and Bornmann, 2017). This study therefore aimed to explore the usefulness of Altmetric.com data as a means of identifying and categorizing the policy impact of research articles from a single center (the University of Sheffield). Method: This study has only included published research articles from authors at the University of Sheffield and indexed in the Altmetric.com database. Altmetric data on policy impact was sourced from Altmetric.com following a data request and included citations up until February 2017. Supplementary Altmetric.com data, including news media, blogs, Mendeley saves, and Wikipedia citations, were also gathered. Results: Altmetric.com data did enable the identification of policy documents that cited relevant articles. In total, 1,463 pieces of published research from authors at the University of Sheffield were found to be cited by between 1 and 13 policy documents. 21 research articles (1%) were listed as being cited in five or more policy documents; 21 (1%) in four policy documents; 50 (3%) in three documents; 186 (13%) in two documents; and 1,185 (81%) in one document. Of those 1,463 outputs, 1,449 (99%) were journal articles, 13 were books, and 1 was a book chapter (less than 1%). The time lag from the publication of the research to its citation in policy documents ranged from 3 months to 31 years. Analysis of the 92 research articles cited in three or more policy documents indicated that the research topics with the greatest policy impact were medicine, dentistry, and health, followed by social science and pure science. The Altmetric.com data enabled an in-depth assessment of the 21 research articles cited in five or more policy documents. However, errors of attribution and designation were found in the Altmetric.com data. These findings might be generalizable to other institutions similar in organizational structure to The University of Sheffield. Conclusion: Within the limitations of the current text-mining system, Altmetric.com can offer important and highly accessible data on the policy impact of an organization’s published research articles, but caution must be exercised when seeking to use this data, especially in terms of providing evidence of policy impact.