| Crowdsourcing has become a powerful tool for generating large numbers of labeled examples for use in machine learning, but its inability to work on complex or specialized problems has prevented it from meeting its true potential. In order to overcome the difficulties associated with these problems, we must consider the workers to be complex and specialized as well. By taking advantage of prior knowledge about the workers, such as their resume, forum posts, purchase history, direct testing, or the prior performance on other crowdsourcing tasks, we can generate a model of such a complex worker. This dissertation provides a framework for considering the different types of prior knowledge about workers, identifies specific conditions that cause crowdsourcing to fail, and then shows how that prior information can be used to overcome those failure conditions with a method called Crowdsourcing with Complex Workers. Furthermore, we show how the prior knowledge about workers can be used with active learning to reduce the cost of our method. |
