Design And Implementation Of Steelmaking And Continuous Casting Information System Based On Hadoop In Steel Works

With the continuous advancement of information technology and the rapid development of computer technology,the era of "big data" in iron and steel enterprises has come.How to effectively develop and utilize the potential value of big data is an important way for iron and steel enterprises to achieve transformation,upgrading and strategic breakthrough.However,due to the single data structure,small storage capacity,slow computing speed and poor fault-tolerant recovery ability of traditional storage mode,the storage and calculation of large data become bottlenecks,which seriously restricts the application of large data.The emergence of Hadoop provides a way to solve this problem.Hadoop provides powerful distributed storage and parallel computing,which can solve the problem of PB-level data storage and computing.Therefore,how to use Hadoop to build a large data platform for iron and steel enterprises has become a hot research direction of in-depth reform of information technology in iron and steel enterprises.Based on the research background of the key project of the National Natural Fund of China "Modular Method and Application of Coordination and Optimization of Production Scheduling and Process Control in Steelmaking-Continuous Casting Process",this thesis takes the actual construction of Fujian Sangang Steelmaking-Continuous Casting Information Platform as the research foundation,and takes the practical application of Hadoop data in iron and steel enterprises as the research direction.The design and implementation of information platform are completed in four aspects:overall architecture,modular design,platform performance optimization and platform-based data analysis.The storage and calculation of large data are mainly studied and solved,and the application of Hadoop technology in iron and steel information construction is realized.The main works of this thesis is showed as follows:(1)Based on the coordination optimization principle and the analysis of steelmaking-continuous casting production process,the basic requirements of the information platform are determined,and the overall framework of the information platform is designed.Aiming at the requirement of real-time interaction and mass storage of information platform,the advantages and disadvantages of traditional database and Hadoop solutions are analyzed.A hybrid architecture scheme based on MYSQL and Hadoop is proposed to solve the defect of single storage scheme.(2)Detailed analysis and design of the two core functional modules of data exchange and data storage of information platform.According to the principle of optimization and technological process,the data model and interface scheme of data interaction are designed.According to the application characteristics of steelmaking-continuous casting production data,a storage scheme based on HBase is proposed.Aiming at the problem of low performance of HBase in non-primary key query,a secondary index scheme based on Coprocessor is proposed.After establishing the two level index,the query performance of non primary key columns is increased by more than 80%.(3)Defect analysis and optimization design of information platform.Aiming at the single point fault problem of information platform,a solution based on "hot backup" is proposed,and the effectiveness of the solution is verified.Aiming at the problem of high write latency in information platform,the limitations of Hadoop default storage strategy are analyzed,and an improved copy placement strategy is proposed.It is proved that the improved storage strategy reduces the write latency by about 41.2%.(4)Based on the information platform,the data mining algorithm for massive data is studied,taking the refinery monthly average power consumption statistics and the segregation prediction of continuous casting center as an example.Aiming at the problem of time-consuming and low precision of traditional algorithms,the parallel optimization of the algorithm is carried out using Hadoop's MapReduce framework.After verification,the accuracy of the optimized algorithm is increased by about 4.7%on the standard set.In the practical application of segregation prediction in continuous casting center,the prediction accuracy is improved by 11.4%,the running time is shortened by 58%,and the accuracy and efficiency are greatly improved.(5)Finally,based on the overall architecture,the design scheme of each functional module and the optimization scheme of the related data analysis algorithm proposed in this thesis,the system implementation and functional testing of the information platform are carried out.The results show that the information platform built in this thesis can meet the needs and run well.At last,the performance of the information platform is tested from three aspects:data storage,data reading and writing and data analysis.After testing,the performance of the information platform in data storage capacity,data access speed and data computing ability has been greatly improved due to the introduction of Hadoop.