Study On Evaulation And Optimization Strategies For Crowdsourcing Quality

Recently, crowdsourcing has become a very popular business model. As a employer, enterprise distribute their jobs to solve problems by the Internet. By the control of Internet, they could make full use of the volunteer's creativity and ability. And these volunteers could fulfill these jobs in their spare time and earn a little money. Crowdsourcing provides a new way to organize the labor force for the software and service companies.The working mode provided by the crowdsourcing could help employer obtain sufficient free workers and solve problems by their ability and intelligence. However, some workers don't take these jobs seriously and just want to realize the maximum of interests. They provide some low quality results and go against the employer's original intension. It's a great challenge to evaluate the workers' precision and guarantee the result quality. So far the study on the crowdsourcing quality management is still in an initial exploratory phase at home and abroad.This dissertation studies some evaluation and optimization strategies for the crowdsourcing quality and proposes some key methods to improve the task quality. The main contributions are listed as follows:(1) A worker precision evaluation strategy based on Bayes decision theory is proposed. The worker precision is regarded as a random variable. The Bayes decision theory is adopted to verify the worker's precision. The evaluation process take into account the worker's history record, current performance and risk factor. Compared with the simple evaluation strategy based on golden standard data, this method is much more efficient and reliable.(2) A dynamic worker replacement strategy based on voting consistency is introduced. Based on simple majority voting, the working group is updated dynamically. The task set is divided into multiple task phases. When every task phase ends, we detect unqualified workers by hypothesis testing technique and replace them according to the accuracy requirement of the crowdsourcing tasks.(3) A novel weighted aggregation rule based on agreement among workers is given to improve the result accuracy. We improved traditional simple majority voting strategy. According to the agreement of answers given by the workers, we classify all the tasks into the high-agreement tasks and low-agreement tasks. For the high-agreement tasks, we use simple majority voting to select the correct answer while ensuring the result accuracy. For the low-agreement tasks, we adopt weighted majority voting strategy, which assigns more weight to the worker who has a higher precision. The worker precision is evaluated by his performance in the high-agreement tasks, which offers the weight information for the low-agreement tasks.(4) The optimal group selection algorithm based on two commission model is proposed. In the passive commission model and active commission model, certain workers are selected to construct the group. The algorithm could obtain the minimum group commission while the result accuracy satisfies the precision requirement.In summary, this dissertation dedicates to study the evaluation and optimization strategies for the crowdsourcing quality. Experimental evaluations show that these methods could improve the result quality for the crowdsourcing tasks efficiently. We hope that these approaches and techniques could make some referential values for the crowdsourcing quality management in the future.