The importance of adequate user feedback for developing appraisal tools for evidence synthesis-the case of TACIT (Tool for Addressing Conflicts of Interest in Trials)

BACKGROUND AND OBJECTIVE: The validity of randomized trials included in evidence syntheses is typically assessed using appraisal tools. Such tools are often developed by a group of experts and usually involve some feedback from intended tool users. STUDY DESIGN AND SETTING: In this commentary, we provide an overview of user testing procedures in the development of four selected appraisal tools for evidence synthesis of randomized trials and describe and discuss the user testing procedures in TACIT, a new Tool for Addressing Conflicts of Interest in Trials (TACIT) that we have developed. RESULTS: The procedures for user feedback varied between the selected appraisal tools in relation to the type of users and trials involved, in which setting user testing was done, and how feedback was collected. In TACIT, informal user feedback was provided in the early phase of tool development and formalized feedback was provided to a late draft of the tool. The formalized feedback involved workshop participants using the tool on a preselected set of trials and systematic reviewers using the tool in their own review. Feedback formats included both semistructured interviews and a questionnaire. CONCLUSIONS: Adequate user feedback is important for the development of appraisal tools, and we suggest striving for an iterative process of tool revisions informed by different groups of intended users applying the tool in different contexts and providing feedback using diverse formats. Feedback from users after public release of the tool, but before final publication, has the potential to reach a large number of users applying the tool on a broad set of trials. We encourage systematic reviewers to apply the publicly released version of TACIT and contact us with feedback.