| Performance assessment has been frequently adopted in large scale and high-stakestests. And then different human raters are invited to assess candidates’ performance.Since the results are extremely important for candidates, it is vital to assure thatdifferent raters can attain comparable reliability and validity in their judgment,which is the aim of this research.Set in the context of NMET GD, two rater groups,10teacher raters (universityteachers) and10non-teacher raters (graduate students majoring in subjects relevantto English), were invited to make assessments on30candidates’ performance onstory-retelling. Besides awarding scores to the candidates, raters were also requiredto provide written comments to6of the candidates. Then comparison was conductedto explore whether there existed difference between the two rater groups in eithertheir ratings or their comments.Data analysis was approached from two complementary directions.Quantitative results, including reliability analysis, t-test and FACETS analysis,suggest that the two rater groups are comparable in terms of internal consistency. Ineach group,9of the10raters fall within the acceptable upper or lower limits(1.40--0.60). As to severity, though data shows that teacher raters was a little bitsevere than non-teacher raters, it was demonstrated that there existed no significantdifference in severity between the two rater groups. Qualitative results show thatboth raters groups followed a similar pattern in focusing on different criteriacategories when rating. Difference occurred only in how the two rater groupscommented: teacher raters’ comments tended to be more specific while non-teacherraters’ comments were more general.Based on the findings, it can be concluded that non-teacher raters’ ratings are as reliable and valid as teacher raters’, suggesting the suitability and practicality ofincluding non-teacher raters in the rating of some large-scale high-stakes tests likestory-retelling. |
PDF Full Text Request