Treatment effects may remain the same even when trial participants differed from the target population

Objective RCTs have been criticised for lacking external validity. We assessed whether a trial in people with type I diabetes mellitus (T1DM) mirrored the wider population, and applied sample-weighting methods to assess the impact of differences on our trial's findings.

Study design and setting The REPOSE trial was nested within a large UK cohort capturing demographic, clinical and quality of life (QoL) data for people with T1DM undergoing structured diabetes-specific education. We firstly assessed whether our RCT participants were comparable to this cohort using propensity score modelling. Following this we re-weighted the trial population to better match the wider cohort and re-estimated the treatment effect.

Results Trial participants differed from the cohort in regards to sex, weight, HbA1c and also QoL and satisfaction with current treatment. Nevertheless, the treatment effects derived from alternative model weightings were similar to that of the original RCT.

Conclusions Our RCT participants differed in composition to the wider population but the original findings were unaffected by sampling adjustments. We encourage investigators take steps to address criticisms of generalisability, but doing so is problematic: external data, even if available, may contain limited information and analyses can be susceptible to model misspecification.