Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

Abstract

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions.

Metadata

Item Type:	Article
Authors/Creators:	Coretta, Stefano Casillas, Joseph V. Roessig, Simon (xqb506@york.ac.uk) Franke, Michael Ahn, Byron Al-Hoorie, Ali H. Al-Tamimi, Jalal Alotaibi, Najd E. AlShakhori, Mohammed K. Altmiller, Ruth M. Arantes, Pablo Athanasopoulou, Angeliki Baese-Berk, Melissa M. Bailey, George https://orcid.org/0000-0001-5137-8394 Sangma, Cheman Baira A Beier, Eleonora J. Benavides, Gabriela M. Benker, Nicole BensonMeyer, Emelia P. Benway, Nina R. Berry, Grant M. Bing, Liwen Bjorndahl, Christina Bolyanatz, Mariška Braver, Aaron Brown, Alicia M. Brown, Alicia M. Brugos, Alejna Buchanan, Erin M. Butlin, Tanna Buxó-Lugo, Andrés Caillol, Coline Cangemi, Francesco Carignan, Christopher Carraturo, Sita Caudrelier, Tiphaine Chodroff, Eleanor https://orcid.org/0000-0003-4549-5919 Cohn, Michelle Cronenberg, Johanna Crouzet, Olivier Dagar, Erica L. Dawson, Charlotte Diantoro, Carissa A. Dokovova, Marie Drake, Shiloh Du, Fengting Dubuis, Margaux Duême, Florent Durward, Matthew Egurtzegi, Ander Elsherif, Mahmoud M. Esser, Janina Ferragne, Emmanuel Ferreira, Fernanda Fink, Lauren K. Finley, Sara Foster, Kurtis Foulkes, Paul https://orcid.org/0000-0001-9481-1004 Franzke, Rosa Frazer-McKee, Gabriel Fromont, Robert García, Christina Geller, Jason Grasso, Camille L. Greca, Pia Grice, Martine Grose-Hodge, Magdalena S. Gully, Amelia J. https://orcid.org/0000-0002-8600-121X Halfacre, Caitlin Hauser, Ivy Hay, Jen Haywood, Robert Hellmuth, Sam https://orcid.org/0000-0002-0062-904X Hilger, Allison I. Holliday, Nicole Hoogland, Damar Huang, Yaqian Hughes, Vincent https://orcid.org/0000-0002-4660-979X Isasa, Ane Icardo Ilchovska, Zlatomira G. Jeon, Hae-Sung Jones, Jacq Junges, Mágat N. Kaefer, Stephanie Kaland, Constantijn Kelley, Matthew C. Kelly, Niamh E. Kettig, Thomas (tk1026@york.ac.uk) Khattab, Ghada Koolen, Ruud Krahmer, Emiel Krajewska, Dorota Krug, Andreas Kumar, Abhilasha A. Lander, Anna Lentz, Tomas O. Li, Wanyin Li, Yanyu Lialiou, Maria Ronaldo M. Lima, Jr. Lo, Justin J. H. Otero, Julio Cesar Lopez Mackay, Bradley MacLeod, Bethany Mallard, Mel McConnellogue, Carol-Ann Mary Moroz, George Murali, Mridhula Nalborczyk, Ladislas Nenadić, Filip Nieder, Jessica Nikolić, Dušan Nogueira, Francisco G. S. Offerman, Heather M. Passoni, Elisa Pélissier, Maud Perry, Scott J. Pfiffner, Alexandra M. Proctor, Michael Rhodes, Ryan Rodríguez, Nicole Roepke, Elizabeth Röer, Jan P. Sbacco, Lucia Scarborough, Rebecca Schaeffler, Felix Schleef, Erik Schmitz, Dominic Shiryaev, Alexander Sóskuthy, Márton (ms1341@york.ac.uk) Spaniol, Malin Stanley, Joseph A. Strickler, Alyssa Tavano, Alessandro Tomaschek, Fabian Tucker, Benjamin V. Turnbull, Rory Ugwuanyi, Kingsley O. Urrestarazu-Porta, Iñigo Vijver, Ruben van de Engen, Kristin J. Van Miltenburg, Emiel van Wang, Bruce Xiao Warner, Natasha Wehrle, Simon Westerbeek, Hans Wiener, Seth Winters, Stephen Wong, Sidney G.-J. Wood, Anna Wottawa, Jane Xu, Chenzi (cx936@york.ac.uk) Zárate-Sández, Germán Zellou, Georgia Zhang, Cong Zhu, Jian Roettger, Timo B.
Copyright, Publisher and Additional Information:	This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details
Dates:	Accepted: 2 February 2023 Published: 2023
Institution:	The University of York
Academic Units:	The University of York > Faculty of Arts and Humanities (York) > Language and Linguistic Science (York)
Depositing User:	Pure (York)
Date Deposited:	03 Feb 2023 14:10
Last Modified:	14 Apr 2024 23:15
Published Version:	https://doi.org/10.1177/25152459231162567
Status:	Published
Refereed:	Yes
Identification Number:	https://doi.org/10.1177/25152459231162567
Related URLs:	https://doi.org/10.1177/2515245923116256...

Download

Accepted Version

Filename: 48843699_File000000_1202459385.pdf

Description: 48843699_File000000_1202459385

CLICK TO DOWNLOAD

CORE (COnnecting REpositories)

Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

Abstract

Metadata

Download

Accepted Version

Export

Statistics