Design And Implementation Of Data Analysis Task Operating Environment For Heterogeneous Models

In recent years,more and more enterprises have used data analysis applications as an important boost to promote enterprise development.The data analysis framework can help enterprises to train data analysis models;the data analysis task operating environment provides the operating environment for the data analysis model and is the basic support for data analysis applications.The data analysis models supported by different data analysis frameworks are different in file extension,format and usage,which are called "heterogeneous models" in this thesis.Configuring multiple sets of data analysis task operating environments for heterogeneous models will not only cause great waste of resources,but also greatly increase the complexity and workload of system operation and maintenance.This thesis aims to design and implement a data analysis task operating environment for heterogeneous models and simplify system operation and maintenance through functions such as data analysis task anomaly detection.The data analysis task operating environment for heterogeneous models provides functions such as data access,data analysis,and result output.Among them,data access realizes the functions of reading real-time data sources and preprocessing data.Data analysis realizes the functions of storing,loading and model inference for heterogeneous models.Result output realizes the functions of encapsulating and sending inference results.This thesis also proposes an anomaly detection method based on continual trusted transfer(Anomaly Detection Via contInual truSted transfER,ADVISER for short).The method first proposes a distance and density based instance transfer decision method to continually judge which knowledge should be transferred to the target domain.Secondly,the method maintains a set of trusted classifiers and a set of untrusted classifiers in the target domain.The knowledge transferred to the target domain will be trained as a new classifier and put into the set of untrusted classifiers.Finally,the method proposes a method to continually update the weights of the classifiers so that the two classifier sets change with the data distribution of the target domain.Compared with the existing methods,this method solves the problem that the model training cycle is long in the anomaly detection scenario;in the transfer learning scenario,it solves the problem that the traditional transfer learning cannot consider the continual update of the source domain and target domain data distribution.A series of experiments show that the accuracy of this method is at least 12%higher than that of traditional anomaly detection and transfer learning methods.This thesis first introduces the research background of the data analysis task operating environment for heterogeneous models.Then combines the research and analysis of existing data analysis task operating environment products to propose the requirements of data analysis task operating environment for heterogeneous models.The anomaly detection method based on continual trusted transfer is introduced.The design and implementation of the data analysis task operating environment for heterogeneous models are introduced in detail.Finally,the module and integration test of the system is carried out to verify the availability of the system.