Performance Analysis System For Big Data Analysis Application Platfor

Currently,data is growing faster and faster,and it is difficult for enterprises to analyze valuable information from massive data to provide information support for the development of business strategies;a large number of enterprises have started to use big data analytics application platforms to make decisions based on data analysis,so as to improve the quality of enterprise products and services.Some enterprises have a wide variety of data,and analysts need to use the platform to deal with a large number of data analysis tasks,and the high concurrency of these tasks easily leads to various performance problems of the platform,due to the complex functional architecture of the big data analysis application platform,resulting in the difficulty of performance analysis,which requires a lot of time to find and locate performance problems.If the cause cannot be located and solution measures taken in time,there will be a risk of platform collapse,which may cause serious consequences,so there is an urgent need to design an efficient performance analysis system.In this paper,we focus on the key technologies for performance analysis of big data platform and further build a performance analysis system for big data analysis application platform.This paper mainly carries out the following work.(1)To solve the problem of automatic generation of performance data for different scenarios,a test benchmark Micro Big Bench for big data analytic application platform oriented to enterprise decision making application scenarios is designed.three scenarios of workloads are generated based on Micro Big Bench to simulate the high concurrent environment under different scenarios,and non-intrusive acquisition is used to realize platform multi-task scenarios under。(2)In order to solve the problem of low accuracy of anomaly detection,an anomaly detection process is designed.Based on the collected performance indicators,the LSTM model is used to predict the CPU utilization of the platform,and the difference between the predicted CPU utilization and the actual value is calculated.It provides an important basis for analyzing performance bottlenecks by providing alerts and locating the code block that caused the anomaly.(3)Based on the aforementioned research,we design and implement a performance analysis system for big data analysis application platform.The system integrates test management,performance data collection and storage,performance index data prediction,anomaly detection and performance bottleneck location,and verifies the practicality of the system through testing.The performance analysis system realizes the determination and location of abnormal performance indicators under multiple tasks of the big data application platform,improves operation and maintenance efficiency,and reduces manual operation and maintenance costs.