Distributed Database Storage Technology Research For Next Generation Tokamak Experiments Data

Tokamak is a huge complicated experimental device for magnetic confinement fusion,composed of different subsystems.The experiment will produce plenty of data,which could be analyzed by researchers to further fulfil the theory justification and realize experiment's improvement.With the development of long-pulse tokamak experiments and the improvement of data acquisition,the data generated more rapidly,which requires data storage and management systems' quickly saving,efficient searching and accessing function offering to scientists.According to the background information mentioned above,this research makes a progress in J-TEXT data storage and acquisition system,achieving the new design and improvements in MDSplus data management,backup and D-TACQ acquisition,eventually develops a cloud database,named JCDB(J-TEXT Cloud Database),for massive data processing.In the first place,because of the oversimplified function of MDSPlus in management,this research designs and successfully develops the MDSplus data management system,achieving the data batch management,offering an efficient searching method.Combined MDSPlus with SQL Server,the system is able to import all metadata of MDSplus into SQL server directly.In this background,the JTEXT-Traverser is developed for data visualization,which could make it easy for users to process and synchronize the results to both MDSPlus and SQL Server.Apart from that,on the basis of network transformation,the research utilizes Rsync open source software to realize incremental backup,overcoming the weakness of small capacity and long recovery time of the original tape-based backup system.At the same time,the researcher finds that D-TACQ acquisition device is prone to fail when uploading in a complex experimental environment.To solve this problem,the D-TACQ runtime infrastructure software is developed,realizing a series of functions,such as status monitoring,data checking,problems discovering and re-uploading automatically for all DTACQ in J-TEXT,and improving the availability of acquisition card at the same time.Moreover,to respond the big data challenge in future long pulse fusion experiment,this paper creatively adopts a cluster database technology.It deeply studies the modern distributed file system and distributed database technology,designing a data storage management system,the J-TEXT Cloud Database,for large scientific experimental devices,and creating the thought of data division into metadata and scientific data,taking a lead in proposing matadata managing solution based on MongoDB.The solution is a practical application of the technology that ITERDB plan to adopt.By designing the data model,interface and plugin,the metadata could be managed and accessed efficiently,making the system add and delete functions more flexibly and modularly.By iterate improvement and continuously optimization of the storage engine,the index-based model,CassandraIndex,is proposed.In the test of cluster with 4 nodes,the read/write performance could respectively reach 337MB/s and 280MB/s.With the comparison of GlusterFS and MDSPlus,the great potential of storage engine on the basis of NoSQL database is verified,also the probability of providing storage service to long pulse fusion experiment.In addition,the CassandraIndex Model could also be a reference for CFETR storage management design.Eventually,as for the cluster deployment difficulties,the automatic deployment scheme of the cluster is given,achieving the automatic deployment of MongoDB cluster and Cassandra cluster.