Measurement of candidates, examiners, and skills across different performance assessment formats

Organizations providing certification for medical specialists use performance and product assessments in which experienced practitioners (i.e., examiners) rate candidates on the degree to which they demonstrate proficiency on skills that are key to safe and effective practice, such as diagnosis and treatment. Different organizations employ different types of assessment formats, such as standardized protocols and different types of candidate case assessments, in various combinations or alone. To explore whether differences in assessment formats influence the outcome of such assessments, I identified and characterized similarities and differences in measurement outcomes for formats that two certification organizations used.;I used a many-facet Rasch measurement approach to analyze archival rating data from the certification examinations that two organizations administered. In each data set, the same group of examiners rated the same group of candidates on the same two skills, diagnosis and treatment. I compared three assessment formats: standardized protocols, candidate cases assessed with candidates present, and candidate cases assessed without candidates present.;There were significant differences in measurement outcomes across the three assessment formats, providing evidence of method effects within the certification organizations' examinations. The design of this study suggests a means of exploring where and how to look for evidence of method effects. For psychometricians working with medical specialty certification organizations, the study offers ideas about the different ways the assessment formats that such organizations use may unknowingly introduce method variance into the measurement process. The findings may help the psychometricians that advise certification organizations make recommendations about which assessment formats to use, or how to combine them most effectively.