Font Size: a A A

The effects of consensus process expectations and rater training strategies on rater accuracy, interrater agreement, and behavior recall in an assessment center simulation

Posted on:2003-09-16Degree:Ph.DType:Dissertation
University:University of Missouri - Saint LouisCandidate:Lewis, Cornelius FreemanFull Text:PDF
GTID:1466390011989843Subject:Occupational psychology
Abstract/Summary:
The study investigated the effects of consensus process expectations (no consensus process expectation versus consensus process expectation) and rater training strategies (behavior-driven training, frame-of-reference (FOR) training, and behavior-driven training + FOR (hybrid) training) on accuracy, interrater agreement, and behavior recall (the dependent variables) in an assessment center simulation. Differential elevation was the accuracy measure. The two measures of interrater agreement were: (a) James, Demaree, & Wolf's (1984) interrater agreement index and (b) Schmidt & Hunter's (1989) standard deviation as a measure of interrater agreement. Behavior recall was a free recall measure. 149 assessors (students) participated in the study. Two role-play work-sample exercises were taken from a commercial assessment center. The assessor's task was to watch the videotaped performance and rate the candidate's performance on five dimensions in each of the two exercises. The two exercises were a fact-finding exercise and a sales call exercise. Analysis of variances and regressions were performed on the data.;First, a main effect was found for consensus process expectation's affect on accuracy, but the results were in the opposite direction. The study found assessors in the no consensus process expectation group had greater accuracy than assessors in the consensus process expectation group. Second, the assessors in the no consensus process expectation group had higher interrater agreement than assessors expecting a consensus process discussion. This means the assessors in the no consensus process expectation group agreed more on the assessment center ratings than the assessors in the consensus process expectation group. Third, the FOR training strategy produced the most accurate scores, the hybrid training strategy produced the second most accurate scores, and the behavior-driven training strategy produced the least accurate scores. Fourth, the assessors in the FOR training strategy had greater interrater agreement than the assessors in the hybrid and behavior-driven training strategies. Fifth, the consensus process expectation and rater training strategy main effects were not statistically significant for behavior recall. Sixth, accountability did not mediate the consensus process expectation---accuracy relationship. A discussion is provided on why hypotheses were not confirmed, suggestions for future research, and study limitations.
Keywords/Search Tags:Process expectation, Consensus process, Training, Interrater agreement, Accuracy, FOR, Assessment center, Behavior recall
Related items