Objectives To describe and analyze the user experience of a late-phase prototype of Tool for Addressing Conflicts of Interest in Trials (TACIT). The tool was developed primarily for guiding systematic reviewers in collecting, processing, and interpreting information on conflicts of interest in included randomized trials. Study Design and Setting A prototype tool was piloted in two settings: 1) We contacted potential participants through our network, social media, and snowball sampling. Twenty-two experienced Cochrane reviewers were invited to a virtual training module applying the tool on a preselected set of trials, and feedback was collected both during a 60-minute group session and in individual semi-structured interviews of 25-55 minutes; 2) We contacted corresponding authors of systematic review protocols from 2022. Authors of 115 protocols were invited to apply the tool on trials included in their own reviews, and feedback was collected using an online questionnaire. Recordings from the group session and interviews were transcribed and imported to Nvivo 12. Feedback was analyzed using thematic cross-case analysis. Questionnaire data were analyzed quantitatively. Based on our experience, we developed a short guide for planning of study appraisal tool piloting. Results Eleven experienced Cochrane reviewers and 14 systematic review teams provided feedback. Feedback was organized in four themes: general impressions, tool concepts, content and layout, and practical usage. Users generally had a favorable overall impression of the tool but suggested enhancing user friendliness and more guidance on how to make judgments based on limited conflicts of interest information and when it was reasonable to stop searching for additional information. Of 14 systematic review teams, two rated the overall impression as excellent, 11 as good, one as neutral, and no ratings of poor or very poor. The guide included nine key issues to consider when planning study appraisal tool piloting. Conclusion Users of a prototype tool for addressing conflicts of interest in trials generally had a positive experience, but more guidance allowing ease of use and how to deal with limited information on conflicts of interest was suggested. The feedback guided a subsequent tool adjustment, including an Excel-based application, and we provide a short guide for planning piloting of future study appraisal tools. Plain Language Summary This study describes user experience of a prototype of a tool for addressing conflicts of interest in randomized clinical trials included in systematic reviews. Eleven participants who had published multiple Cochrane Reviews gave feedback in interviews after using the tool in a virtual training module. Fourteen additional review teams gave feedback via a questionnaire after having used the tool in their own systematic reviews. Users generally had a positive experience, but adaptations to make the tool easier to use were suggested. The feedback guided a subsequent tool adjustment, including development of an Excel-based application. Based on our experience, we also provide a short guide for planning piloting of future study appraisal tools.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)