Research On Three Types Of Experimental Data Preprocessing Methods

With the continuous development of technology and economic level,a large number of experimental data will be used,but because of the acquisition of different environment,different time and different state of the experimenta data,so these data can not be used directly.It is necessary to pre-process the experimenta data first,so the research of experimental data preprocessing method is very meaningful.In this thesis,preprocessing methods are researched on three types of experimental data,time series experimental data,small sample experimental data,and non-time series experimental data.It focuses on the method of eliminating abnormal data,cluster analysis and outlier detection,and analyzes the performance of the proposed algorithm through corresponding examples.For the time series experimental data,this thesis constructs the recurrence model by constructing the initial fitting data,using the method of B-spline curve to construct the recurrence model,using the judgment threshold estimated by the spline smoothing method to determine whether the abnormal value of the result data of the bidirectional test is abnormal,and fitting and 'repairing the data that meet the repair conditions.When the results of the bidirectional test are different,the extrapolation model is constructed to further verify.The example analysis shows that the method proposed in this thes can eliminate abnormal data more effectively than other methods,and the data that may produce stage jump can be tested more effectively by data segmentation processing,so that the model has better stability.It has higher application value and wider applicability and higher removal rate of abnormal data,and it is a set of feasible methods to eliminate and repair abnormal data.For the non-time series experimental data,this thesis first introduces the improved process of the two clustering algorithms,and tests the clustering effect of the algorithm on the experimental data set and compares it with other algorithms.Then,the two improved clustering algorithms are compared and analyzed from the complexity of algorithm,the index value of clustering evaluation and the running time of the algorithm.Finally,the outlier detection framework based on clustering is proposed.The performance of the experimental data set is compared and analyzed.For small sample experimental data,this thesis uses the maximum likelihood estimation method to first perform a goodness-of-fit test on the original data to determine its optimal distribution,and then adopts different outlier rejection criteria for different distributions.Finally,an example is used to show the abnormal point detection process of small sample data.