Reliability, validity, user acceptance, and bias in peer evaluations of self -directed interdependent work team

Teamwork education has become increasingly important over the last decade. Peer evaluations are being used as a source of information to provide feedback to team members as well as a mechanism to determine an individual grade from a group grade. Peer evaluations have an appeal because the team members are in the best position to observe the team skills of their fellow team members. Despite this advantage, peer evaluations can be abused and may have undesirable effects on individuals in the group. Because of these concerns and the corresponding use of peer evaluations, there is a need for a better understanding of the reliability, validity, and bias in peer assessments of students working on self-directed interdependent problem-solving teams.;Generalizability theory techniques, structured interviews, and survey data were used to answer research questions related to the reliability, validity, user acceptance, and bias in peer evaluations. Seniors from a multidisciplinary team focused engineering design course were selected for the study. The level of consensus in the peer evaluations was relatively low. Expected behavior is believed to account for a significant proportion of the rater effects but further research is needed to confirm this hypothesis. Friendship bias was a factor in the peer evaluations but was not significant in reducing the reliability or validity of the measurements. In the case of multiple measurements of the same construct, residual variance accounted for approximately 60% of the total variance in the peer ratings. The relatively low level of consistency between multiple measures is believed to reflect the contextual nature of teamwork. Ratings at any given point in time more closely reflect the current stage of team development and the current activities of the team. Convergent validity for "effort applied to the task" and "technical knowledge applied to the task" was high. User reaction to the peer evaluations was generally positive with only limited targeting of specific individuals by team members. Although common practice is to use peer evaluations for evaluative purposes, the emphasis should be placed on developmental uses of the ratings rather than evaluative.