Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively

Donald D. Anderson, Steven Long, Geb W. Thomas, Matthew D. Putnam, Joan E. Bechtold, Matthew D. Karam

Research output: Contribution to journalArticlepeer-review

24 Scopus citations

Abstract

Background Performance assessment in skills training is ideally based on objective, reliable, and clinically relevant indicators of success. The Objective Structured Assessment of Technical Skill (OSATS) is a reliable and valid tool that has been increasingly used in orthopaedic skills training. It uses a global rating approach to structure expert evaluation of technical skills with the experts working from a list of operative competencies that are each rated on a 5-point Likert scale anchored by behavioral descriptors. Given the observational nature of its scoring, the OSATS might not effectively assess the quality of surgical results. Questions/purposes (1) Does OSATS scoring in an intraarticular fracture reduction training exercise correlate with the quality of the reduction? (2) Does OSATS scoring in a cadaveric extraarticular fracture fixation exercise correlate with the mechanical integrity of the fixation? Methods Orthopaedic residents at the University of Iowa (six postgraduate year [PGY]-1s) and at the University of Minnesota (seven PGY-1s and eight PGY-2s) undertook a skills training exercise that involved reducing a simulated intraarticular fracture under fluoroscopic guidance. Iowa residents participated three times during 1 month, and Minnesota residents participated twice with 1 month between their two sessions. A fellowship-trained orthopaedic traumatologist rated each performance using a modified OSATS scoring scheme. The quality of the articular reduction obtained was then directly measured. Regression analysis was performed between OSATS scores and two metrics of articular reduction quality: articular surface deviation and estimated contact stress. Another skills training exercise involved fixing a simulated distal radius fracture in a cadaveric specimen. Thirty residents, distributed across four PGY classes (PGY-2 and PGY-3, n = 8 each; PGY-4 and PGY-5, n = 7 each), simultaneously completed the exercise at individual stations. One of three faculty hand surgeons independently scored each performance using a validated OSATS scoring system. The mechanical integrity of each fixation construct was then assessed in a materials testing machine. Regression analysis was performed between OSATS scores and two metrics of fixation integrity: stiffness and failure load. Results In the intraarticular fracture model, OSATS scores did not correlate with articular reduction quality (maximum surface deviations: R = 0.17, p = 0.25; maximum contact stress: R = 0.22, p = 0.13). Similarly in the cadaveric extraarticular fracture model, OSATS scores did not correlate with the integrity of the mechanical fixation (stiffness: R = 0.10, p = 0.60; failure load: R = 0.30, p = 0.10). Conclusions OSATS scoring methods do not effectively assess the quality of the surgical result. Efforts must be made to incorporate assessment metrics that reflect the quality of the surgical result. Clinical Relevance New objective, reliable, and clinically relevant measures of the quality of the surgical result obtained by a trainee are urgently needed. For intraarticular fracture reduction and extraarticular fracture fixation, direct physical measurement of reduction quality and of mechanical integrity of fixation, respectively, meet this need.

Original languageEnglish (US)
Pages (from-to)874-881
Number of pages8
JournalClinical orthopaedics and related research
Volume474
Issue number4
DOIs
StatePublished - Oct 26 2016

Bibliographical note

Funding Information:
The institutions of the authors received funding from the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under award numbers P50AR48939 (DDA) and P50AR055533 (DDA, GWT), from the National Board of Medical Examiners® (NBME®) Edward J. Stemmler, MD Medical Education Research Fund (DDA, GWT, MDK), from Core Competency Innovation Grants provided by the OMeGA Medical Grants Association (DDA), and from the Orthopaedic Trauma Association (MDK, DDA). All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research® editors and board members are on file with the publication and can be viewed on request. Clinical Orthopaedics and Related Research® neither advocates nor endorses the use of any treatment, drug, or device. Readers are encouraged to always seek additional information, including FDA-approval status, of any drug or device prior to clinical use. Each author certifies that his or her institution approved the human protocol for this investigation, that all investigations were conducted in conformity with ethical principles of research, and that informed consent for participation in the study was obtained.

Funding Information:
The institutions of the authors received funding from the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under award numbers P50AR48939 (DDA) and P50AR055533 (DDA, GWT), from the National Board of Medical Examiners® (NBME®) Edward J. Stemmler, MD Medical Education Research Fund (DDA, GWT, MDK), from Core Competency Innovation Grants provided by the OMeGA Medical Grants Association (DDA), and from the Orthopaedic Trauma Association (MDK, DDA).

Funding Information:
The institutions of the authors received funding from the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under award numbers P50AR48939 (DDA) and P50AR055533 (DDA, GWT), from the National Board of Medical Examiners? (NBME?) Edward J. Stemmler, MD Medical Education Research Fund (DDA, GWT, MDK), from Core Competency Innovation Grants provided by the OMeGA Medical Grants Association (DDA), and from the Orthopaedic Trauma Association (MDK, DDA).

Fingerprint Dive into the research topics of 'Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively'. Together they form a unique fingerprint.

Cite this