Research On Framework Design And Business Model Scheduling Optimization For Traffic Management Big Data Analysis

With the rapid development of internet technology and the modern information industry,traffic management departments collect more and more channels of data,larger and larger amounts of data,and a variety of data types.How to dig out valuable potential information by analyzing,judging,and correlating data in massive data,and assisting the traffic management department to make accurate decisions on urban transportation is an urgent problem.Traffic management data is usually related to spatial attributes,while most traditional analysis databases lack spatial analysis capabilities.Although traditional GIS platforms have powerful spatial analysis functions,they are limited by the stand-alone architecture,and stand-alone processing performance becomes a bottleneck,and spatiotemporal big data analysis is inefficient.Therefore,after investigating the characteristics of traditional big data analysis tools and traffic management data,we selected the distributed column database Clickhouse and calculation engine Spark to build a big data analysis framework,and provided business analysis capabilities to the traffic management department by constructing a business model.After analyzing the advantages and disadvantages of Clickhouse and Spark,this article builds a hybrid computing framework of Clickhouse and Spark with virtual tables as bridges.It provides rich analysis functions,efficient reading and writing capabilities through Clickhouse,and provides spatial analysis capabilities through Spark RDD,the structure and design of the business operator in the traffic management business model are designed in detail.Based on the hybrid computing framework,combined with the characteristics of traffic management data and the actual analysis requirements,the scheduling optimization design for the creation of the business model and the timing execution were respectively carried out.The shared application pool optimizes the creation of business models,and serves more analysts on the basis of ensuring the user experience.For the timing execution requirements of the business model,from the overall and operator levels of the business model,the result set timing analysis optimization method and DAG-based branch optimization method are designed respectively,aiming to reduce redundant calculations and improve the efficiency of business model analysis.Based on the hybrid computing framework,the characteristics of traffic management dynamic data are analyzed,and data tables that are incrementally added for synchronization methods such as trajectory data and bayonet photo data are optimized by incremental computing techniques.The concept and model of incremental calculation are proposed,and the incremental calculation design is performed at the database level.According to the difference of the incremental calculation merge function,the business operators are divided into Append type operator and Update type operator,and the calculation principle of the incremental calculation operator is introduced in detail;at the end of the incremental calculation framework,the related design of the SQL logic merge and cache rules is introduced.The experimental results show that when the data size is large,the computing efficiency of the hybrid computing framework is significantly improved compared to the traditional database;the DAG-based branch optimization method has a certain improvement effect on the timing execution of the business model;incremental calculation fundamentally reduces the amount of data involved in the calculation.When the data size is large,the calculation time can be significantly reduced,and the calculation efficiency is significantly improved.