Research On Fast Storage And Simulation Technology Of Electromagnetic Big Data

With the development of networking radar,electromagnetic space network,wireless communication,and other technologies,electromagnetic big data is becoming a new type of big data.The number of radio stations has shown a significant growing trend in recent years.Specially,the ability of electromagnetic space detection has been greatly enhanced by high-speed radars with a data generating rate of 10 Gbps.A radar network consists of many types of radar that can generate large-flow electromagnetic data even up to 100 Gbps.Such a high speed of radar data generation calls for new efficient big data storage technologies.On the other hand,due to the complexity of the electromagnetic space network,it is difficult to obtain real electromagnetic big data by constructing a real electromagnetic space network.To this end,simulation seems to be a more reasonable way to generate electromagnetic big data streams.However,effective methods are needed to simulate electromagnetic space network as well as the generation of electromagnetic big data streams.Based on the above background,in this dissertation we focus on two issues of electromagnetic big data management:the fast storage technology for electromagnetic big data and the simulation-based electromagnetic big data benchmark platform.We first describe the research background and current situations of big data storage technologies,and then analyze the research progress of storage models and big data storage platforms.Aiming at offering high-throughput storage for electromagnetic big data,we propose rotation model based fast storage layer for electromagnetic big data.Regarding the issue of electromagnetic big data simulation,we design an electromagnetic big data benchmark testbed based on the simulation of electromagnetic space network.Overall,the main work and contributions of this dissertation can be summarized as follows:(1)In order to provide high-throughput storage for electromagnetic big data,we propose a fast storage model named RotaryDS,which is based on a rotation model built in the memory.RotaryDS acts as an intermediate layer between high-speed data streams and underlying storage nodes.We set up multiple data buckets in the memory to construct the rotation model to disperse the I/O pressure of high speed data streams on underlying storage nodes,and thereby to improve the write throughput of the entire system.The experimental results show that this design can effectively improve the overall write throughput and is easy to be horizontally expanded to support larger data streams.In practical applications,a large-scale rotational storage model can be constructed by increasing the number of nodes to support larger electromagnetic data streams.(2)As it is difficult to construct a real electromagnetic space network to carry out performance evaluation of electromagnetic big data studies,we propose and implement an electromagnetic big data testbed named STEM which provides flexible and configurable simulations with user-friendly interfaces.The STEM system can simulate the generation of continuous electromagnetic big data streams with multiple data labels and devices based on point targets and linear FM radar signals.Thus,it can effectively support the testing and further studies of electromagnetic big data related algorithms.Currently,we apply STEM on MongoDB and provide storage performance tests to verify the availability and reusability of STEM.