Design And Implementation Of Big Data Processing Platform For Digital Manufacturing Workshop Based On Kafka

With the rise of Industrial Big Data,Intelligent Manufacturing and other similar concepts,equipment data as a ”first-hand information” that can directly reflect the manufacturing process,is increasingly valued.How to efficiently use the massive amount of heterogeneous data generated from Digital manufacturing workshops,which is the basic unit of the modern manufacturing,in the production process,is a problem that must be solved to make the manufacturing industry more modernized and high-end.The traditional manufacturing data acquisition and processing system are often problemoriented,which means they're only focusing on a few points and the system isn't open and flexible enough.Modern manufacturing has put forward higher requirements for data utilization and it needs a data platform with good integration,openness and flexibility.In response to such needs,this paper firstly designed and built a four-layer network topology for digital workshops and developed a unified data acquisition system.At the same time,based on the Lambda Architecture,a data processing system provided with batch and stream computing capabilities for massive heterogeneous production processes in parallel is developed.Then using this platform,a big data visualization system based on B/S architecture is developed.Finally,this paper builds a development platform for the experimental platform verification.The main contents of the full text are summarized as follows:(1)Heterogeneous device connecting network design and unified data acquisition system development.A four-layer network architecture was designed for the digital manufacturing workshop and corresponding guidance experience was given for hardware selection.The unified data acquisition system of the equipment was researched and developed,which overcome the drawbacks of traditional data acquisition software that can only collect data of a single type of equipment.And the experimental verification of the Huazhong 8 CNC system and ABB industrial robot was carried out.(2)Design and implementation of manufacturing data transmission systems.Based on the high-performance distributed message middleware Kafka,the communication system co-operation module and components were developed.The Kafka Topic strategy and data exchange formats were designed and optimized.The performance of the data transmission system was improved through a series of performance tuning experiments based on existing hardware conditions.The system can broke through the performance bottlenecks existing in the traditional data transmission system to deal with massive heterogeneous manufacturing data and provided the basis for the subsequent realization of manufacturing big data platform.(3)Manufacturing workshop big data processing platform and real-time visualization system development.The mathematical modeling and algorithm implementation of full-batch calculation and streaming calculation were analyzed and extended to the manufacturing process data processing.Based on the Lambda architecture,a processing architecture that can parallel the manufacturing data batch and streaming computing was implemented.Based on the architecture,a big data real-time visualization system based on the B/S architecture was developed,which can presented the platform data processing results to the user.(4)System automation deployment realization and performance,stability assessment.Based on Docker containerization technology,the software can be automatically deployed to solve the problem of complex software deployment.The hardware experiment platform was designed and built.The data acquisition system performance,transmission system throughput and delay,visualization system adaptability,overall system stability was tested to demonstrate the effectiveness of the design and development system in this paper.