Research And Implementation Of Information Collection Technology Based On Message Queue

With the introduction of the "Internet+" power operation mode of the State Grid Corporation,the data scale of the power marketing business is getting larger and larger.Due to the limitations of early design,the existing power information collection system has limited expansion capability,and it has become difficult to adapt to the rapid business.Growth requirements for system performance.On the other hand,the use of electricity information collection system as the most important data support and data source in the entire marketing business,will inevitably put forward new demands on the processing capacity of massive power consumption data.This paper adopts a design scheme that combines the traditional electricity information collection system research and development technology and message queue technology to reconstruct the electricity information collection system.Firstly,according to the characteristics of power consumption information collection,the common message queue is studied.According to the high throughput of Apache Kafka and the features supporting large-scale data analysis and processing,Apache Kafka distributed message queue is selected as the system design component.Secondly,combined with Apache Kafka to reconstruct the relevant modules of the electricity information collection system,based on the in-depth study of the working principle,architecture and message data structure of Apache Kafka,the message production service is designed in the process of electricity data reporting.Handling message queue related function modules such as services to improve the performance bottleneck of existing power information collection systems in coping with massive data concurrent requests,to meet the large-scale growth of traffic volume of other services such as electric information collection and marketing The demand presented.Finally,the system performance of the electricity information collection system combined with Apache Kafka is tested,and the function panel of the main system station and the system realization of the subordinate function modules are given,which will be used for the analysis and mining of large-scale users' electricity data information.Provide strong technical support.