Many analysts, one dataset: making transparent how variations in analytical choices affect results

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts’ prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.

Metadata

Item Type:	Article
Authors/Creators:	Silberzahn, R. Uhlmann, E.L. Martin, D.P. Anselmi, P. Aust, F. Awtrey, E. Bahník, V.S. Bai, F. Bannard, C. Bonnier, E. Carlsson, R. Cheung, F. Christensen, G. Clay, R. Craig, M.A. Rosa, A.D. Dam, L. Evans, M.H. Cervantes, I.F. Fong, N. Gamez-Djokic, M. Glenz, A. Gordon-McKeon, S. Heaton, T.J. Hederos, K. Heene, M. Mohr, A.J.H. Högden, F. Hui, K. Johannesson, M. Kalodimos, J. Kaszubowski, E. Kennedy, D.M. Lei, R. Lindsay, T.A. Liverani, S. Madan, C.R. Molden, D. Molleman, E. Morey, R.D. Mulder, L.B. Nijstad, B.R. Pope, N.G. Pope, B. Prenoveau, J.M. Rink, F. Robusto, E. Roderique, H. Sandberg, A. Schlüter, E. Schönbrodt, F.D. Sherman, M.F. Sommer, S.A. Sotak, K. Spain, S. Spörlein, C. Stafford, T. https://orcid.org/0000-0002-8089-9479 Stefanutti, L. Tauber, S. Ullrich, J. Vianello, M. Wagenmakers, E.-J. Witkowiak, M. Yoon, S. Nosek, B.A.
Copyright, Publisher and Additional Information:	© The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
Keywords:	crowdsourcing science; data analysis; scientific transparency; open data; open materials
Dates:	Accepted: 24 April 2017 Published (online): 23 August 2018 Published: 1 September 2018
Institution:	The University of Sheffield
Academic Units:	The University of Sheffield > Faculty of Science (Sheffield) > Department of Psychology (Sheffield) The University of Sheffield > Faculty of Science (Sheffield) > School of Mathematics and Statistics (Sheffield)
Depositing User:	Symplectic Sheffield
Date Deposited:	09 May 2019 13:27
Last Modified:	09 May 2019 13:27
Status:	Published
Publisher:	SAGE Publications
Refereed:	Yes
Identification Number:	10.1177/2515245917747646
Open Archives Initiative ID (OAI ID):	oai:eprints.whiterose.ac.uk:145795

Download

Published Version

Filename: 2515245917747646.pdf

Licence: CC-BY-NC 4.0

CLICK TO DOWNLOAD

CORE (COnnecting REpositories)

Many analysts, one dataset: making transparent how variations in analytical choices affect results

Abstract

Metadata

Download

Published Version

Export

Statistics