| Purpose:In this study, Researchers tried to use the Semi-structured Thematic Apperception Test (STAT) to measure achievement motivation which is correlated well with the job performance to study the rater bias and bad using of the rating scale. The Multi-faceted Rasch Model (MFRM) was applied in the score results analysis to look forward to:(1) Estimate the value of Potential ability of candidates to replace the raw score in original scoring system to improve scientific decision-making; (2) Through the analysis of inter-rater consistency and intra-rater inconsistency, identify the unqualified rater and distinguish their error type in order to provide targeted training to improve reliability and validity of STAT (3) Explore the application of STAT in personnel assessment.Methods:In this study,8 pictures was selected from the third edition of the Thematic Apperception Test through the Delphi method to develop STAT, the pictures ware made semi-structured by adding a theme, time, person, place and other clues. We applied STAT in reserve cadre selection for a district government of Dalian and collected data of candidates who apply for legal departments.6 graduate students in psychology and education who understand Thematic Apperception Test as well as content analysis and coding was selected as raters, they ware offered coding training by researchers. Data sifting based on the following conditions: candidates meet entry conditions of reserve cadre selection; the story of every picture covering at least 3 points; the story of every picture is not less than 200 words. After Filtering data, we selected 48 candidates as research subject. Scoring system used in this study is achievement motivation scoring system revised by Blankship (2006), this system was proved to be validity. Using the revised scoring system, raters coded story for each picture by paragraph, then they summarized the original data coded in each dimension and set the standard of each rating grade in all dimensions. Reference to the 7 point rating scale, raters rated the story imagined by every candidate independently. Using the multi-faceted Rasch model analyzes score results by FACETS in three aspects:(1) Rater facet analysis: inter-rater consistency, intra-rater inconsistency, rater-subject interaction, rate-dimension bias; (2) Scoring system analysis:difficulty of dimension, inter-rater consistency of each dimension, rating-scale analysis. (3) Candidate facet analysis:potential ability estimate, difference of potential ability.Results:(1) Rater facet analysis:there is significant difference in rater severity, rater C is the most severe and rater B is the most lenient. Infit-value which stands for rater's self-inconsistency ranged from 0.76 to 1.26, rater E and rater C had less self-inconsistency and rater B was considered too conservative. There are a variety of rater biases model in this study. (2) Scoring system analysis:the inter-rater consistency of each dimension is acceptable, rater D made biases in four dimensions (Z>2.0 or Z<-2.0) and should be coached in rating skill. The dimension of "satisfaction" is less difficult compared to other dimensions. In contrast, The dimension of "Plan to succeed" is more difficult compared to other dimensions. The study of the rating scale showed that it was not an equal interval one, middle grade had small interval and both ends grade had broader interval. (3) Candidate facet analysis:there is significant difference in "motivation of success" of all candidates, this result refered that "motivation of success" can be a well indicator in reserve cadre selection.Conclusion:(1) The inter-rater consistency, intra-rater inconsistency, rate-subject interaction, rate-dimension bias and the use of rating scale can introduce biases into scoring result of Semi-structured Thematic Apperception Test。Using the MFRM corrected score instead of raw score can improve scientific decision-making. (2) Through the study of rater biases, we can develop more directional rater training.
|
PDF Full Text Request