Research On Quality Evaluation Of Machine Learning Models Based On ADS-B Dataset

In the past decade,with the rapid development of wireless communication technology,the categories of electromagnetic signal transmitting devices have become diverse and complex,and traditional electromagnetic signal recognition technologies have been unable to meet the needs of situational awareness.Machine learning technology has the advantage of high recognition rate and has been widely used in the field of electromagnetic signal recognition.However,in the process of application,there are also threats to robustness and security,and there is a lack of relevant technical research on evaluating model quality.This article conducts research on traditional machine learning models and deep learning model quality evaluation techniques based on the ADS-B signal dataset.The main research contents are as follows:Firstly,this paper studies the traditional machine learning algorithms to be evaluated,and introduces the sources and collection methods of the dataset required for this evaluation.The principle of traditional machine learning algorithms for modeling data is explained in principle,and the model performance under different signal-to-noise ratios,different feature dimensions,and different signal category numbers is explored.Experiments have found that traditional machine learning algorithms cannot effectively identify individual signals from radiation sources,and are sensitive to the number of categories.Secondly,this article trained ten different deep learning models to identify individual signals from radiation sources.Then,the performance of the model was evaluated from the perspectives of classification performance,robustness performance,and security performance.The classification performance indicators include accuracy,recall,and other indicators.The robust performance of the model was studied under conditions such as carrier frequency offset and multipath channel damage,and a comparison of the performance of each model under different indicators was provided.Finally,the security performance of model was evaluated under adversarial sample attacks,the performance of different models under white and black box adversarial sample attacks was presented.Evaluation provides important references for the construction of indicator evaluation systems and evaluation of model quality.Finally,based on the data obtained from the above evaluation,and according to the principles of systematicness,typicality,scientificity,and effectiveness of the evaluation index system,this paper first establishes an initial index system for model evaluation.Next,through qualitative analysis and screening of indicators,a scientific and complete quality evaluation indicator system for the in-depth learning model was finally established.According to this system,the relationship between indicators is analyzed by gray correlation analysis,the importance of indicators is ranked,and then the weight of indicators is determined by combining the analytic hierarchy process.Based on the above work,a membership function and a fuzzy relational matrix are constructed,and a comprehensive fuzzy evaluation method is used to evaluate the quality of the deep learning model.This paper proposes a complete set of deep learning model quality evaluation index system,as well as a new index weight allocation method.It is the first time to introduce the model comprehensive evaluation method into the field of deep learning model quality evaluation,making up for the gaps and shortcomings of deep learning model quality evaluation in the field of electromagnetic signal recognition.