A many-facet Rasch measurement analysis to explore rater effects and rater training in medical school admissions

For many undergraduate medical education (UME) programs, ratings of applicants' performance in interviews are integral to the decision making process. This study used many-facet Rasch measurement (MFRM) to explore the severity and consistency of 200 interviewers' ratings of 599 applicants to a UME program. The raters varied widely in their severity and many of the raters had poor fit to the MFRM model. Although raters at four stations rated both station-specific and across-station competencies, the highest correlations were within station; the correlations across stations for the common competencies were low. No significant relationships were found between raters' training mode(s) (online and in-person) and rating behaviours. The findings from this work demonstrate the need to make explicit how differences in rater judgement affect fairness of applicant assessment to UME programs.