The Study And Application Of Data Cleaning And Incremental Extraction For Flight Task Date

In order to guarantee the operational efficiency of the airport apron and enhances the quality of fli ght task data,it is necessary to do data cleaning on the dataset.This thesis analyzed the characteristics of flight task data and cleared the main object,and designed the corresponding cleaning rules respectively,and determined the extraction mechanism when flight task data updating,so as to achieved the purpose of efficient display and real-time refresh,and developed the apron operation software for flight task data.The main work of this article includes the following aspects:Firstly,a sorted-neighborhood algorithm based on clustering index is proposed for similar duplicate record.The clustered index is created on the gate field in the flight task dataset before sorting;during the sorting process,through the combination of multiple keywor-ds,and utilizing the gate use frequency to drive the sliding window’s size which is real-time changes,to imp roved the detection efficiency of similar duplicate records,as well as to ensured the loading speed of the flight task data after data cleaning.Secondly,the variation of flight task data varies with different periods,so it is crucial to choose reasonable data extraction mechanism.By analyzing the data extraction method,the attributes are extracted according to the weight of each attribute,and the change records are extracted by the whole table matching method which is combined with MD5 code.By comparing the execution time of the flight task data obtained after the full extraction and incremental extraction,the strategy of the extraction is that when the airport is busy with the full extraction,and the airport is idle with the full table comparison method based on decision attribute,and the airport is closed do not extract.Thirdly,by analyzing the lack of the airport’s current work method,the actual demands of the apron operation software were investigated,and the software needed to om nidirectional display the task progress was cleared.The airport operation processes were combed,and the software function module diagram were designed,and the abstract layout of the apron physical were determined,and divided the five kinds of gate use states,which is idle,reserve,occupy,normal and delay.Finally,it achieved the apron operation software for flight task data,and applied it to a domestic civil aviation airport.