The reliability of ratings of student teachers in the Teacher Education Department at Brigham Young University

The main purpose of this study was to estimate the reliability of ratings of student teachers and to find the optimal design for rating student teachers in Brigham Young University's Teacher Education Department. Six student teachers participated and were videotaped while teaching an elementary school class in a typical student teaching situation. After this taping, 12 raters rated both videotaped lessons taught by each of the six student teachers. The design for the collection of the ratings was a fully crossed teacher-by-rater-by-lesson design. A generalizability study was carried out to estimate the variance components and standard errors for each source of variability in the ratings. The largest sources of variation came from the raters and the teacher-by-rater interaction. A series of D Studies were performed to assess the relative effectiveness of using different designs for collecting the ratings in the future, and to identify the effects of varying the number of raters and lessons on the reliability on the ratings.; The results of this study indicate that the raters who participated in this study to evaluate student teachers in the Teacher Education Department at Brigham Young University can produce reliable ratings of student teachers' abilities even though these ratings do show at least two sources of variation that need to be minimized. The results indicate that estimates of a student teacher's teaching ability are likely be more dependable if based on average ratings of more than one rater.; The results of this study also indicate that the individual items in the rating scale are highly intercorrelated with each other, that the first principal component accounts for more than three-fourths of the total variability, and that the within-rater variability across items for a given ratee tends to be small. These three findings support each other and confirm that halo error is present in the ratings.