Item Selection Methods In Computerized Adaptive Testing For Forced-Choice Items And Its Application In Personality Assessment

Computerized adaptive testing with forced-choice items(FC-CAT)combines features of item response theory(IRT),forced-choice questionnaire and computerized adaptive testing(CAT),which is a relatively new study at present.Compared with the traditional testing,it can not only greatly reduce the test time due to the actual measurement of respondents,but also eliminate response biases caused by Likert-type scales such as social desirability as much as possible,so as to improve the test efficiency and measurement accuracy.Therefore,it is a more appropriate measurement form for personality test,which is helpful to improve the measurement mode of talent evaluation,further promote the use of FC-CAT in the process of talent evaluation and personnel selection,and improve the effectiveness and authenticity of the test results.Then,it has a good development and application prospect,and is worth referring to and popularizing.As a crucial component of CAT,item selection methods are used to choose items as the test progresses,which exerts a significant influence on measurement accuracy,validity,and safety of tests,and thus has important significance and value for the further theoretical research and practical application of FC-CAT.However,at present,the relevant research on FC-CAT is still in the preliminary stage,and only a few theoretical studies and practical applications are carried out based on the ideal point model.It is found that the Thurstonian IRT(TIRT)model,as a dominant response model,can be used to model a variaty of forced-choice scales and has demonstrated efficacy in accommodating many combinations of traits and block sizes.This makes it widely applicable to many existing forced-choice questionnaires such as the Survey of Interpersonal Values,the Customer Contact Styles Questionnaire,Occupational Personality Questionnaire and useful in designing questionnaires in the future.Therefore,this paper focuses on the TIRT model in this study,which is conducive to the promotion and application of FC-CAT.So far,there are hardly any discussions and studies on the item selection methods of FC-CAT at home and abroad,and only a few researchers have used the FC-A-optimality method based on the Fisher information(FI)to directly conduct FC-CAT research without discussing its applicability and feasibility for forced selection tests.It is well known that the item selection methods based on the FI(FI-based)has defects such as attenuation paradox and overexposure,thus affecting the measurement accuracy and test security.This study intends to develop new FC-CAT item selection methods to improve the trait estimation accuracy or improving the utilization rate of the item pool and test security as much as possible.At present,there is no comprehensive comparison on FC-CAT item selection methods,which is not conducive to the choice and use of item selection methods by practical users.The research of FC-CAT is weak at present,so there are still great prospects for research and development in the future,and there are too few item selection methods to choose,and there are no opinions and suggestions for reference.Therefore,it is necessary to make a comprehensive comparison of FC-CAT item selection methods from the aspects of accuracy,item pool utilization,test security and so on.To sum up,in order to make up for the deficiencies of the current research on FC-CAT and its item selection methods at home and abroad,four studies have been carried out in this paper in the following.Study 1:The Development of Item Selection Methods based on the KL Information for FC-CAT.In study 1,based on the Thurston IRT model,three new item selection methods are proposed for FC-CAT based on the KL information,namely the KL information index(KI)method for FC items(FC-KI),the posterior expected KL information method(K~B)for FC items(FC-K~B)and the KL distance between posteriors(KLP)method for FC items(FC-KLP).Based on the simulated item pool,this study verified the performance of the new methods,FC-KI,FC-K~Band FC-KLP in the three-and five-dimensional scenarios through Monte Carlo simulation studies,and compared them with the existing FC-A-optimality methods and FC-D-optimality methods based on the Fisher information.At the same time,this study also explored the influence of the test length,inter-trait correlation and other factors on the effect of the new methods.The results showed that:(1)The trait estimation accuracy and test security of all FC-CAT proposed item selection methods based on KL information,FC-KI,FC-K~Band FC-KLP,were better under various experimental conditions.Then,these new methods are all applicable to FC-CAT.Except for the FC-KI method to be further discussed and improved,other new methods perform better than the traditional existing FC-A-optimality methods and FC-D-optimality methods based on the FI.(2)The accuracy of trait estimation and item pool security of the five FC-CAT item selection methods increased with the increase of the test length.(3)The accuracy of trait estimation and item pool security of the five FC-CAT item selection methods decreased with the increase of inter-trait correlation.Study 2:The Development of Item Selection Methods based on the Bayesian Theory.In study 2,based on the Thurston IRT model,two new item selection methods are proposed for FC-CAT based on the Bayesian theory,namely the mutual information method for FC items(FC-MUI)and the continuous entropy method for FC items(FC-CEM).Based on the simulated item pool,this study verified the performance of the new methods,FC-MUI and FC-CEM,in the three-and five-dimensional scenarios through Monte Carlo simulation studies,and compared them with the existing FC-A-optimality methods and FC-D-optimality methods based on the FI.At the same time,this study also explored the influence of the test length,inter-trait correlation and other factors on the effect of the new methods.The results showed that:The results show that:(1)The trait estimation accuracy and test security of all FC-CAT proposed item selection methods based on the Bayesian theory,FC-MUI and FC-CEM,were better under various experimental conditions.Then,these new methods are all applicable to FC-CAT.Moreover,these new methods perform better than the traditional existing FC-A-optimality methods and FC-D-optimality methods based on the FI.(2)The accuracy of trait estimation and item pool security of the four FC-CAT item selection methods increased with the increase of the test length.(3)The accuracy of trait estimation and item pool security of the four FC-CAT item selection methods decreased with the increase of inter-trait correlation.Study 3:The Comprehensive Comparison of FC-CAT Item Selection Methods.The study 3 further verified the performance of five FC-CAT item selection methods newly developed in Study 1 and Study 2 under the high-dimensional situations and real item pool respectively,and makes a comprehensive comparison between them with the existing FC-A-optimality methods and FC-D-optimality methods based on the FI.This paper comprehensively compared the performance of these seven FC-CAT item selection methods in terms of the estimation accuracy,the item pool utilization and the test security,and explored the influence of some factors such as the dimensions,test length and inter-trait correlation on the effect of them under the high-dimensional situations and real item pool.The results show that:(1)The estimation accuracy and item pool security of the three kinds of FC-CAT item selection methods respectively based on the Fisher information,the KL information and the Bayesian theory are still acceptable under the high-dimensional situations and real item pool.Except for the FC-KI method,other selection methods are superior to the existing FC-A-optimality method and FC-D-optimality method with the trait estimation accuracy and the item pool security,which once again verified the feasibility of the new methods.(2)The estimation accuracy of FC-KI method based on the KL information and the item pool security are still the lowest,while the estimation accuracy of FC-MUI method and FC-CEM method based on the perspective of Bayesian theory is the highest and the item pool security is the best.Moreover,the performance trends of the two methods are also very similar.(3)As the number of test trait dimensions increases,the estimation error of each item selection method is greater than that of the three-dimensional and five-dimensional situations,and the estimation accuracy is significantly reduced.The item exposure rate basically decreases with the increase of the number of test trait dimensions.(4)With the increase of test length,the estimation accuracy and exposure rate of seven item selection methods are improved.(5)The inter-trait correlation still has a great influence on seven FC-CAT item selection methods.With the increase of inter-trait correlation,the estimation accuracy decreases obviously,and the item pool security also decreases.In short,the choice of various FC-CAT item selection methods varies with the changes of experimental conditions and specific measurement requirements.Study 4:The Application of FC-CAT in Personality Assessment.The study 4mainly applied FC-CAT and its new item selection methods to the personality assessment of college students on the basis of the above three studies.Specifically,in the case of using real item pool and real data,on the one hand,this study tested the effectiveness of FC-CAT,and discussed the feasibility and effect of the new methods in real application.On the other hand,it conducted an in-depth analysis of the results of personality evaluation of college students.The results show that:(1)Based on the real data,the applicability and effectiveness of the FC-CAT item selection methos proposed in this paper are verified again.(2)The measurement accuracy of the personality assessment of college students can be guaranteed by FC-CAT,while saving nearly three quarters of the administrated items,greatly reducing the length of the test,improving the efficiency of the test,and verifying the feasibility and advantages of FC-CAT in practical application.In conclusion,this paper proposed five FC-CAT item selection methods based on the KL information and the perspective of Bayesian theory.The results of three simulation studies and one empirical study based on real data showed that the new methods,such as FC-KI,FC-K~B,FC-KLP,FC-MUI and FC-CEM,all have high estimation accuracy and item pool security.When considering the measurement accuracy and test security comprehensively,except for the FC-KI method which needs further discussion and improvement,the other four new methods perform better than the existing FC-A-optimality method and FC-D-optimality methods on the whole based the FI.In personality assessment,FC-CAT can ensure high measurement accuracy while also saving nearly three quarters of the number of items,greatly reducing the testlength and improving the measurement accuracy.In short,this paper makes up for the shortcomings of the existing FC-CAT item selection methods,provides several more accurate and effective methods for future FC-CAT research,and provides important methods and technical support for further expanding the application of FC-CAT in talent evaluation.