Evaluating the assessments in an orthopaedic residency program

Smith, Nicholas Carl (2015) Evaluating the assessments in an orthopaedic residency program. Masters thesis, Memorial University of Newfoundland.

[img] [English] PDF - Accepted Version
Available under License - The author retains copyright ownership and moral rights in this thesis. Neither the thesis nor substantial extracts from it may be printed or otherwise reproduced without the author's permission.

Download (1MB)


Introduction: Orthopaedic surgical education has undergone major change in the last 15 years. Work hour restrictions, public accountability, and government pressures have led to a paradigm shift in the execution of surgical training. The Royal College of Physicians and Surgeons of Canada (RCPSC) is adopting a competency-based training model in an attempt to ensure the quality of its future surgeons. Objective: To evaluate the reliability and validity of assessment methods of orthopaedic surgery residents as defined by the RCPSC’s CanMEDs framework. Methods: A critical appraisal was undertaken that indicated a paucity of studies evaluating strategies for assessing surgical competencies in residency training programs. Staff surgeons assessed residents in day-to-day performance of duties using the Interprofessional Collaborator Assessment Rubric (ICAR) and the Surgical Encounters Form (SEF). The assessments were collected and measurements of percent agreement, Cronbach’s alpha, and Fleiss Kappa were obtained. Results: For the ICAR percent agreement was 80.6 percent. Cronbach’s Alpha measure averaged 0.662 and the mean Fleiss Kappa score was -0.218 (95% CI -0.400 to -0.089). For the SEF percent agreement was 90.9 percent. Cronbach’s alpha averaged 0.865, 0.920, 0.934, 1.00 and 1.00 for the Medical expert, Technical skills, Communicator, Collaborator, and Advocate roles respectively. The mean Fleiss Kappa score was 0.147 (95% CI -0.071 to 0.364). Conclusion: Low inter-rater reliability results suggest low levels of assessor agreement and subsequently invalid assessment measures. Modification to assessment methods will be required before a valid competency-training program can be fully adopted.

Item Type: Thesis (Masters)
URI: http://research.library.mun.ca/id/eprint/8422
Item ID: 8422
Additional Information: Includes bibliographical references (pages 60-71).
Keywords: orthopaedic, surgery, evaluation, resident, reliability
Department(s): Medicine, Faculty of
Date: May 2015
Date Type: Submission
Library of Congress Subject Heading: Orthopedic surgery--Study and teaching; Residents (Medicine)--Training of; Surgeons--Training of; Clinical competence; Surgery--Evaluation
Medical Subject Heading: Orthopedics; Internship and Residency; Clinical Competence--Evaluation Studies

Actions (login required)

View Item View Item


Downloads per month over the past year

View more statistics