Study On The Reliability And Validity Of OSCE

Objectives Objective Structured Clinical Examination (OSCE), a multidimensional practical examination of clinical skills, has an established place in the assessment of medical students. In keeping up with this fast evolution of assessment methods and to promote a qualification in China medical education, West China Medical School of Sichuan University initiated a process to add OSCE to final examination. Reliability and validity are fundamental considerations of any assessment process. A handful of studies have shown formal psychometric characteristics of this tool in assessing clinical performance. However, its quality, especially the reliability and validity have so far received very little research attention in China. It is crucial that the OSCE be completely evaluated before the numerous resources necessary to develop for medical education are expended. The purpose of this study is to investigate the reliability and the validity of the OSCE and factors that influence reliability, provide guiding principles from which educators may design the evaluation strategies.Methods Traditional measurement considerations based on Classical Test Theory (CTT) and the relationship with other measures were examined. Generalizability Theory (GT) and Multivariate Generalizability Theory (MGT) were used to estimate measurement errors and generalizability coefficients, estimate the overall reliability of the examination, the reliability of the subtests and their respective contributions to overall reliability.Results The OSCE overall reliability, Cronbach's alpha was ranged from 0.42 to 0.54, achieved an acceptable reliability sufficient to evaluate the clinical competence. Construct validity studies indicated that three traitswere involved in the analysis: patient management skills, data-driven reasoning skills, basic clinical practical skills. The overall generalizability coefficient of the OSCE was 0.4422, which was considered moderate but acceptable. Except for the Non-SP based stations, reliability estimates for the subtests were acceptable. Non-SP based stations contributed negatively to reliability of the whole test. Increasing the stations for all the subtests would increase reliability and a range of generalizability coefficient estimates was provided. The G-study and D-study results suggest that a composite universe score approach was not ideal both because variance distributions among subtests were inconsistent with the planned score weighting and because correlations among them were low.Conclusions Several suggestions were provided for the Medical Examination Authority and future study.?An OSCE with 8 SP-based stations and 15 Non-SP stations would be required in the future. ? Several areas for improvement were identified. The adjustment for the OSCE was made by checking the Non-SP based stations. ?Longer tests tend to be more reliable than short ones. ?Case specificity are the likely factors in lowering reliability. ? The components testing different aspects of clinical skills must be carefully balanced to ensure reliability. ?The OSCE is an important addition to the traditional tools for evaluation.