Development of robust methods of assessment of clinical competency in ophthalmic dispensing – results of a pilot trial

Purpose: We evaluated whether the use of Objective Structured Clinical Examinations (OSCEs) combined with established standard setting procedures, could be used to define standards for the achievement of clinical competencies in ophthalmic dispensing. Methods: Ten OSCE stations were created to assess a range of professional competencies for dispensing opticians. For each station, examiners made a global judgment about student performance using a rating scale (bad fail, fail, borderline, pass, good pass). Examiners were instructed not to base their rating on the checklist marks. We also introduced performance based standard setting using a linear regression method. The rating of the student was plotted against marks obtained for the station and a line of best fit was derived from the data. The pass mark for each station was set at the mark for the borderline rating. Results: The average pass mark across stations was 57%. Students with higher marks also tended to be rated more highly by examiners; however, this was not universal. The slopes of the regression lines were significantly greater than zero across stations suggesting that the checklists were appropriate. Feedback from students and examiners was positive. Conclusions: Our study has demonstrated that the OSCE format and use of standard setting procedures is a viable approach to assess clinical competencies in ophthalmic dispensing. More data are required to confirm the reliability of the stations over repeated use.