Research On Answer Aggregation For Open Task In Crowdsourcing

In real life,it is often difficult for computer to handle some problems very well,such as labeling the images and judging whether the two records are the same entity,etc.Crowdsourcing directly publishes these problems on the Internet,and solves these problems by assembling unknown people on the Internet.There are certain limitations in existing answer aggregation methods.On the one hand,the worker's quality model is fixed,and it is only suitable for the case where the number of candidate answers is fixed.On the other hand,the types of questions on the crowdsourcing platform include blank-filling questions,single-choice questions,multiple-choice questions,and a mixture of choices and blanks.The previous answer decision algorithms are mainly based on single-choice questions and not compatible with multiple types of tasks,especially those that contain open answers.Aiming at existing problems,this paper considers a variety of types of tasks,and proposes an open answer aggregation algorithm based on Bayesian.First,we use the past performance of workers to establish a worker quality model,which is not affected by the number of candidate answers.Second,we use the Jaro-Winkler Distance method to calculate the similarity between fill-in answers,thereby expanding candidate options.When considering the interaction between fill-in answers,we believe that a group of answers with high similarity is more likely to be the same answer.In addition,in the previous decision,the prior probability of the candidate answer is not taken into account.We pre-processes the prior probability of the candidate answers according to the prior knowledge given by the machine algorithm and the situation of the extended answers.Extensive experiments show that our answer aggregation algorithm can deal with multiple types of tasks effectively and efficiently.The processing of open answers and prior probabilities improves the accuracy of the algorithm.Because the decision process of multiple-choice questions is an NP-hard problem,we presents an optimization algorithm based on pruning strategy for the multiple-choice answer aggregation,which reduces the number of candidate answers that need to be calculated.Experiments show that the optimization strategy reduces the time cost of the algorithm while keeping the accuracy of decision algorithm.