Optimising error rates in programmes of pilot and definitive trials using Bayesian statistical decision theory

Pilot trials are often conducted in advance of definitive trials to assess their feasibility and to inform their design. Although pilot trials typically collect primary endpoint data, preliminary tests of effectiveness have been discouraged given their typically low power. Power could be increased at the cost of a higher type I error rate, but there is little methodological guidance on how to determine the optimal balance between these operating characteristics. We consider a Bayesian decision-theoretic approach to this problem, introducing a utility function and defining an optimal pilot and definitive trial programme as that which maximises expected utility. We base utility on changes in average primary outcome, the cost of sampling, treatment costs, and the decision-maker’s attitude to risk. We apply this approach to re-design OK-Diabetes, a pilot trial of a complex intervention with a continuous primary outcome with known standard deviation. We then examine how optimal programme characteristics vary with the parameters of the utility function. We find that the conventional approach of not testing for effectiveness in pilot trials can be considerably sub-optimal.