White Rose University Consortium logo
University of Leeds logo University of Sheffield logo York University logo

Countering method bias in questionnaire-based user studies

Gorrell, G., Ford, N., Madden, A., Holdridge, P. and Eaglestone, B. (2011) Countering method bias in questionnaire-based user studies. Journal of Documentation, 67 (3). pp. 507-524. ISSN 0022-0418


Download (176Kb)


Purpose - This paper seeks to discuss reliability problems associated with questionnaires, commonly employed in library and information science. It aims to focus on the effects of "common method variance" (CMV), which is a form of bias, and ways of countering these effects.

Design/methodology/approach - The paper critically reviews the use of existing tools for demonstrating reliability in questionnaire-based studies. In particular, it focuses on Cronbach's alpha, "Harman's single factor test" and Lindell and Whitney's "marker variable" approach. The paper introduces an illustrative case study based on the work on metacognition and web searching. It goes on to make recommendations for researchers considering using a questionnaire-based approach.

Findings - CMV is a problem affecting questionnaire-based studies in different disciplines across social and information science. Where questionnaire items are more abstract, CMV has been found to be more of a problem. The widely used Cronbach alpha measure, of the reliability of a questionnaire, may often be affected by CMV. Where method bias dominates the results, a high alpha score may do no more than indicate that the questionnaire consistently allows participants to accede to their biases. "Harman's single factor test" is criticised as lacking sufficient foundation, and it is concluded that it should not be used. The marker variable approach is a useful option but must be applied with caution.

Practical implications - A number of practical implications for researchers are drawn. Questionnaire-based work investigating abstract constructs should be assumed to carry a high likelihood of CMV, and therefore should attempt to avoid it and demonstrate the degree of success in this regard. Otherwise, interpretation of the results should assume CMV. A number of approaches to assessing and isolating CMV are discussed.

Originality/value - The paper draws attention to a problem that is arguably often overlooked in questionnaire-based studies, namely method bias. It discusses a number of approaches whereby it may be identified and controlled.

Item Type: Article
Copyright, Publisher and Additional Information: © 2011 Emerald Group Publishing. This is an author produced version of a paper subsequently published in Journal of Documentation. Uploaded in accordance with the publisher's self-archiving policy.
Keywords: Research methods; Tests and testing; Variance
Institution: The University of Sheffield
Academic Units: The University of Sheffield > Faculty of Engineering (Sheffield) > Department of Computer Science (Sheffield)
The University of Sheffield > Faculty of Social Sciences (Sheffield) > Information School (Sheffield)
Depositing User: Miss Anthea Tucker
Date Deposited: 06 Jul 2011 13:06
Last Modified: 08 Feb 2013 17:32
Published Version: http://dx.doi.org/10.1108/00220411111124569
Status: Published
Publisher: Emerald
Refereed: Yes
Identification Number: 10.1108/00220411111124569
URI: http://eprints.whiterose.ac.uk/id/eprint/43123

Actions (repository staff only: login required)