Design And Implementation Of Real-time Processing And Monitoring System For Big Data Of Internet Of Vehicles

With the new round of technological innovation in the field of information technology,the Internet,the Internet of Things and even the Internet of Vehicles（IoV）are developing rapidly under the background of the Internet of Everything.The combination of IoV application technology and big data technology,through the analysis and research of the data generated during commercial vehicle driving,is able to obtain all-round vehicle information in the driving process,so as to improve the driving experience of the vehicle and improve the safety and transportation efficiency in the transportation process.Faced with the massive amount of data of IoV and the timeliness requirements in some scenarios,the system needs to process important onboard sensor data in real time and present the data intuitively to business personnel.The traditional vehicle management system relies heavily on relational databases.Faced with the massive amount of IoV data,the pressure of database reading and writing in the system intensifies and the delay increases,which affects the overall performance of the system.In addition,the storage capacity of traditional databases is difficult to expand horizontally,and it is difficult to support the persistent storage of massive amounts of IoV data.The processing mode of the traditional vehicle data system is mostly based on batch processing of offline data,which is difficult to meet the scenarios with high real-time requirements.Large vehicle companies have accumulated a large amount of vehicle-related data.Under the traditional mode of data storage and calculation methods,they cannot meet the requirements of efficient storage,real-time processing,and fast calculation of original data of IoV.This article describes the research background and significance of the application of the real-time processing and monitoring system for big data of IoV.Combined with the actual requirements of a large commercial vehicle manufacturer,the principles and characteristics required by the real-time processing and monitoring system for big data of IoV are explored.Then the functional architecture,technical architecture,network architecture and technology selection of the system are introduced.Finally,combined with the overall architecture and detailed design,a real-time processing and monitoring system for big data of IoV based on Flink is realized.The system can be divided into real-time data synchronization module,real-time data processing module,data storage module and dynamic data visualization module.The data to be processed by this system can be divided into dynamic real-time data and static dimensional data.The dynamic real-time data is the Protobuf format source data sent to IoV platform in real time by the data terminal of the commercial vehicle in motion,and the static data is the vehicle dimensional data stored in MySQL.The realtime data synchronization module synchronizes the data to be processed from IoV platform to the cluster deployed by the system in real time through Kafka Mirrormaker and MySQL master/slave deployment.The real-time data processing module is based on Flink to complete real-time parsing of stream data,real-time dimensional data associating and real-time statistical calculation.The data storage module selects the key attributes of the parsed data and stores them in HDFS.The dynamic data visualization module will dynamically display the processed data on the front end through Grafana,which will show some important vehicle index data more intuitively and in real time to the business personnel.In this way,it is convenient for business personnel to monitor the quality and safety of commercial vehicle in motion.The main work of this paper is to design and implement a real-time processing and monitoring system for big data of IoV.The system is based on the streaming processing framework Flink,which realizes the high concurrent real-time processing of the massive data of IoV with terabyte growth every day.The system uses Kafka,Flink,Hadoop,Zookeeper,Elasticsearch and other big data components to build the overall framework.The relational database MySQL is used to store static dimensional data,while Redis and asynchronous interaction are used to alleviate the performance problems of the relational database due to disk I/O.This technical architecture can effectively solve the problems such as limited storage space of massive data and limited performance in the previous processing system for big data of IoV,which can not achieve low-latency processing,and meet the requirements of the processing system for big data of IoV for low latency and high concurrency.After being tested in a cluster environment,the real-time processing and monitoring system for big data of IoV has achieved the expected design goals.At the same time,it was deployed on the production cluster of a large commercial vehicle manufacturer,and the actual effectiveness was relatively satisfactory,which improved the availability of IoV data and enhanced the ability of commercial vehicle manufacturers to analyze massive amounts of IoV data.