Research On Large-scale Data Production System Based On Automatic Dialing And Measuring Of Mobile Applications

With the rapid development of 5G technology and mobile Internet,smart phones are rapidly popularized,and mobile phones have become an indispensable part of our daily life.In order to comply with the development of the times,major operators need to recognize the service types in the pipeline by identifying the mobile phone traffic,so as to guide the network operation optimization,tariff package promotion and network security.At present,traffic identification in industry mainly adopts regular matching based on message,popular behavior analysis and deep learning methods.These methods all need a large amount of data as support,but the traditional method of manually collecting data can no longer meet the current network development,and the traffic resolution rate of the existing network is gradually decreasing.Based on this,this thesis proposes a set of large-scale data production system,which is used to automatically obtain the traffic after service update,so as to guide the update of characteristic rules or models,provide data support for traffic identification,and improve the identification rate of current network traffic monitoring.The main research contents and conclusions are as follows:(1)By monitoring the application market,the system senses the release and update of new applications in real time,and downloads relevant information and installation files of applications to HDFS file system and Mongo DB database in time.The back-end obtains the path of applications in HDFS through the information stored in the database and downloads it to the local area for dial testing.This system can timely obtain the dial test data of the latest application in the current network,greatly reducing the cost of data acquisition,and providing data support for model training to obtain high accuracy.(2)The system encapsulates a series of mobile applications’ operations(such as sliding click,etc.)by Uiautomator2 tool,and uses the directed graph to traverse the page elements in depth to realize the automatic dialing test mode.At the same time,the manual dialing test function is added to improve some applications with unsatisfactory automatic dialing test results.Statistics show that the number of effective APPs dialed and tested by the system has reached 54740 since its operation,which greatly reduces the labor cost and improves the degree of production automation.(3)The distributed deployment of different nodes(crawler node,master node,dial-and-measure node and analysis node)is divided by functional modules in the system to deal with the problem that regional factors may lead to differences in dialand-measure data of the same application.The crawler node is responsible for monitoring each APP portal and downloading APK files and information to HDFS file system and database for storage;The master node is responsible for task distribution and node management;The dial node is responsible for the dial test and packet capture of the application;The analysis node is responsible for parsing and cleaning Pcap packets.The reliability of data is ensured by deploying dial-andmeasure nodes in different test areas,and the public traffic information in the data can be effectively eliminated by filtering the original data with analysis nodes.The unfiltered and filtered data are used as data sets for model training comparison.The results show that the application accuracy of unfiltered model identification is80.70%,the recall rate is 76.31%,and the F1 value is 77.46%.The application accuracy of model identification by filtering is 93.44%,the recall rate is 92.91%,and the F1 value is 93.05%.After data cleaning and filtering,all indexes reach about 93%,which is at least 13% higher than that of unfiltered data.