Crowdsourcing Quality Evaluation And Control Method For Test Question Labeling Task

Test questions are the traditional carriers to examine students whether master the knowledge points they have learned.The examined knowledge points and the corresponding cognitive verbs are the most fundamental data resources in teaching activities.With them,the subsequent educational data mining and learning analysis research can be further carried out.Therefore,it is particularly important to label the knowledge points and the corresponding cognitive verbs examined in test questions.Crowdsourcing,as an open knowledge production process,combines the joint efforts of a large number of independent individuals to generate solutions superior to individuals.It has achieved remarkable results in solving many complex practical problems.Therefore,this paper proposes to use crowdsourcing mode to complete the labeling task of knowledge points and cognitive verbs.Only when the quality of crowdsourcing results is qualified can the labeling work be meaningful.Therefore,this paper studies the quality evaluation and quality control of crowdsourcing results:(1)In order to effectively evaluate the quality of crowdsourcing results,a crowdsourcing quality evaluation method considering worker credibility is proposed.Firstly,qualified workers are selected by gold standard data to improve the quality evaluation process.Then,the historical data of workers are quantified and the worker confidence index is introduced.The worker attribute is judged according to the worker's labeling results.The worker confidence and the worker attribute are used to measure the worker credibility.Finally,the worker credibility is introduced into the traditional EM algorithm in order to improve the initial value and thus improve the quality of crowdsourcing labeling products.(2)In order to control the quality of crowdsourcing results and motivate workers to complete crowdsourcing tasks with high quality,a crowdsourcing quality control method based on workers' ability and classified incentives is proposed.Firstly,the labels of low-quality workers are filtered out,two indexes of difficulty and ability of labeling test questions are defined,and a model of workers' ability is put forward,in which workers' ability is taken as the weight.Then,the support ratings of each knowledge point(or cognitive verb)are weighted,so that knowledge points(or cognitive verbs)with high support ratings are as close as possible to the correct labeling answers of the test questions,and higher quality labeling results are integrated iteratively.Secondly,in order to encourage high-quality workers to provide more high-quality labels,workers are encouraged respectively.For high-quality workers,the labeling output of workers is defined as a function of workers' credibility,workers' quality and random factors.Based on the principal-agent theory,an incentive mechanism model is constructed,and the optimal solution is found.Finally,a bidding incentive mechanism suitable for high-quality workers is obtained.While ordinary workers can get basic salary.In this study,high school mathematics test question labeling data is used to verify that the improved EM evaluation method based on worker credibility can effectively evaluate crowdsourcing results.At the same time,it is also verified that crowdsourcing quality control algorithm based on workers' ability can improve the quality of crowdsourcing results.Finally,the classification incentive mechanism proposed in this study can encourage high-quality workers to submit better annotation results,thus effectively controlling the quality of crowdsourcing results.