Evaluating the assessments in an orthopaedic residency program

Smith, Nicholas Carl (2015) Evaluating the assessments in an orthopaedic residency program. Masters thesis, Memorial University of Newfoundland.

[English] PDF - Accepted Version Available under License - The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission. Download (1MB)

Abstract

Introduction: Orthopaedic surgical education has undergone major change in the last 15 years. Work hour restrictions, public accountability, and government pressures have led to a paradigm shift in the execution of surgical training. The Royal College of Physicians and Surgeons of Canada (RCPSC) is adopting a competency-based training model in an attempt to ensure the quality of its future surgeons. Objective: To evaluate the reliability and validity of assessment methods of orthopaedic surgery residents as defined by the RCPSC’s CanMEDs framework. Methods: A critical appraisal was undertaken that indicated a paucity of studies evaluating strategies for assessing surgical competencies in residency training programs. Staff surgeons assessed residents in day-to-day performance of duties using the Interprofessional Collaborator Assessment Rubric (ICAR) and the Surgical Encounters Form (SEF). The assessments were collected and measurements of percent agreement, Cronbach’s alpha, and Fleiss Kappa were obtained. Results: For the ICAR percent agreement was 80.6 percent. Cronbach’s Alpha measure averaged 0.662 and the mean Fleiss Kappa score was -0.218 (95% CI -0.400 to -0.089). For the SEF percent agreement was 90.9 percent. Cronbach’s alpha averaged 0.865, 0.920, 0.934, 1.00 and 1.00 for the Medical expert, Technical skills, Communicator, Collaborator, and Advocate roles respectively. The mean Fleiss Kappa score was 0.147 (95% CI -0.071 to 0.364). Conclusion: Low inter-rater reliability results suggest low levels of assessor agreement and subsequently invalid assessment measures. Modification to assessment methods will be required before a valid competency-training program can be fully adopted.

Actions (login required)

View Item