A Preliminary Study On The Rater Effect Of Standardized Patients In OSCE

IntroductionAn Objective Structured Clinical Examination (OSCE) is considered to be an effective method for evaluating clinical skills, which has been used for assessment in various medical disciplines wordwide.Standardized Patients (SP)-based OSCEs are subject to many potential measurement errors, including inappropriate case content, inaccurate or inconsistent portrayal of the case by SPs, and various types of rater errors, such as rater leniency, rater inconsistency, and rater bias.Along with the development of OSCE, more and more academician tried to use modern educational measurement theory to analysis many potential measurement errors in OSCE. Many-faceted Rasch model (MFRM) is an extension of the one-parameter item response theory (IRT) model, which allows the estimation of rater severity, task difficulty, and the difficulty of any other facets affecting the performance of examinees.We conducted a MFRM analysis utilizing the FACETS program, in which the probability of receiving any given rating was determined by the students' competence, SP severity and item difficulty et al. We use the model estimating clinical skills of the students, rater errors of SP and the application of rating scales, in order to provide scientific basis for the quality control of OSCE scores, SP training and revising rating scale.Subjects and Methods23 SPs participated in the examination of 2009 five-year undergraduated clinical medical students of China Medical University are used as the research subjects, as so as 352 clinical medical students of China Medical University participated in the exam. OSCE was taken in the Clinical skills Training Center of the First Affiliated Hospital of China Medical University.23 SPs including 7 male and 16 female,352 students including 154 male and 198 female.We conducted many facets Rasch model using FACETS3.64.0 to analyse the data of this OSCE.Results1. Data and model fittingData and model fit better.2. The outcome of MFRM analysisSP21 is the most stringent standardized patient, while the most lenient standardized patient is SP27; The most difficult item is Item 9, Physical examination, the easiest item is Item 1 and 2. Item 1 is Friendly communication and Item 2 is Respectful treatment.3. Facet of standardized patients(1) SP severitySP severity measures ranges from-1.35-0.74 logits. SP21 is the most severity standardized patient with 0.74 logits (SE= 0.03). SP27 is the most leniency standardized patient with-1.35 logits (SE=0.04). SPs are significantly different in their severity measures. OSCE administrator should arrange the severity SP and the leniency SP into the same group according to SP severity measures to ensure the quality of OSCE.(2) SP self-consistency effectThe infit values of SP10, SP33, SP31 are 4.16,1.9,1.52. Because of infit values more than 1.5, these SPs are inconsistent; The infit value of SP15,SP8,SP24 are all 0.5, These three SP are overly consistent raters.â‘ The inconsistent SPTake SP10 and SP 33 for example. In our study, MFRM provided 100 abnormal scores exceeding the expected value over two standard deviations. When evaluating candidates on Item 4, SP10 provided 40 abnormal scores. As to Item 8, SP10 provided 36 abnormal scores. These indicated SP10 didn't apprehend Item 4 and Item 8 well.SP33 provided level 0 when evaluated Student 259 on the Item 10. But according the outcome, MFRM expected SP33 provide 2.7 score, real score departed 3.6 standard deviations from expecting score.â‘¡The overly consistent ratersStandardized patients 24,8,15 of infit values are 0.5, They had evaluated 90 candidates rarely used level 0 and level 4, but level 1,2,3 frequently.4. Facet of candidates352 candidates'competence measures were normally distributed, ranging from-1.1-1.13 logits, with a whole distance of 2.24 logits. Candidate 156 with 1.1 logits (SE= 0.17) have the highest competence, the minimum competence is Candidate 268 with-1.13 logits (SE= 0.13).5. Facet of itemsThe most difficult item is Item 9, Physical examination, the easiest item is Item 1 and 2. Item 1 is Friendly communication and Item 2 is Respectful treatment.Conclusions 1. Overall, the difficulty of the examination fit for the candidates, RUCIS fit for the students in this study.2. Rater severity of SP differed from each other significantly. Strict standardized patients can consider a more stringent level of standardized patients and standardized patients with the more stringent test in the same group in order to increase the fairness of the examination.3. Standardized patient ratings for individual characteristics of their rating scale and rating scale in the understanding of individual topics for training.4. On the rating scale in the individual subject difficult to improve. Good understanding on the subject of standardized patient presentation to improve.