Research On Key Technologies Of Truth Discovery On Dirty Data

With the rapid development of the information age, the Internet has become themain way for people to obtain the data. Even some of today’s data managementapplications has been extended to integrate data from the network data sources. Sofor the same real world entity, different data sources may provide a conflictdescription of it, how to find the truth description of this entity among all of theseconflicts informations is called the issue of conflict resolution. We mainly focus onsemantic conflicts resolution, also referred to as truth discovery.With the proportion of the confusion dataset increasing, the accuracy of theexisted methods also decrease rapidly. In case of this situation, we propose adata-division method based on information entropy, the original data is divided intonormal dataset and confusing dataset. For normal datase, we use the method basedon probability model,we apply multi-properties to calculate the accurary of thedata sources,and also take the existence of dependencies between the data sourcesinto consideration, so as to eliminate the influence brought by copied data.But for confusing dataset, existing methods can not effectively find thetruth.So we propose a network-based conflict resolution that obtain thecorresponding text information via Google API, then construct a candidate set basedon the entity information provide on web,we identify the truth by an efficientevaluation model from the candidate set. Besides,we identified the size of candidateusing a dynamic way to reduce the amount of computation.In addition to considering the conflict resolution of structured data, weexpanded the background to the topic discussing of social networking, because thedata is no longer provided by traditional data sources, but the person. We get relatedtopic description from the web, then we construct the candidate set by the fivecharacteristics from the description provided by users. People should have theunique characteristics，so we divide users into different fields,and use it to evaluatethe elements in candidate set，then find the truth.The experiments on real-worlddataset proved our method more efficient and accurate.