Evaluating a pilot nationwide licencing OSCE for internationally-qualified registered nurses: Making better decisions through mixed-methods triangulation

Background Nurse migration across global workplaces presents multiple benefits, yet challenges exist in ensuring registration processes in new countries are high-quality, sustainable, and defensible. Performance assessments such as Objective Structured Clinical Examinations (OSCEs) provide one approach to assess whether internationally-qualified health professionals reach a standard of clinical competency for safe registration. This paper presents how integration of a range of data provides improved evaluation of a licencing OSCE for registered nurses (RNs). Objective To determine the reliability, discriminant validity, cost-effectiveness, and acceptability of a pilot licencing OSCE for RNs. Methods A 10-station OSCE was developed and piloted with 59 recently qualified RNs. A mixed-methods evaluation was undertaken. Internal consistency was assessed using Cronbach's α, and OSCE reliability via Generalizability statistics. R2 and Inter-grade discrimination was determined to examine the variation in checklist scores due to global grades. Discriminant validity was evaluated by linking OSCE performance to educational attainment, and a logistic regression model was fitted to identify performance predictors beyond knowledge and OSCE complexity. Cost data were collected through expert consultation and observation, and analysed using a consolidated economic framework. Acceptability data were collected through stakeholder focus groups and analysed using the framework method. All data were triangulated to underpin the OSCE evaluation. Results With 10 stations, acceptable generalisability (G-coefficient = 0.75) and high internal-consistency reliability (α = 0.76) were demonstrated. Possessing a postgraduate nursing qualification (p < 0.030) and between-circuit differences (p < 0.035) appeared significant predictors of candidate performance. OSCE stakeholder acceptability was categorised into 12 themes related to either candidate factors or OSCE design. Triangulation of psychometrics and cost data indicated 10 stations were optimal. Triangulation of psychometrics and acceptability data revealed increasing time to complete tasks, improving candidate familiarity with equipment, documentation and/or room layout, and clarity with station objectives could improve station metrics. Conclusions A holistic approach to determining the quality of an OSCE utilising mixed-methods evaluation sheds additional light on OSCE performance more than any one method in isolation. This paper provides a methodological framework for OSCE assessment in licencing and other high-stakes settings.