Research And Implementation Of Highway OD Analysis System Based On Hadoop

OD data of expressway is a kind of important data supporting the operation and management of expressway.It can be used to grasp the vehicle travel distribution and traffic efficiency of each section of expressway network in detail,so as to help the traffic induction,congestion relief and utilization efficiency improvement of expressway network.With the improvement of China’s expressway network construction,the demand for OD analysis is getting higher and higher.Due to the increase in the scale of expressway related data,the traditional processing method not only USES the data in a low efficiency,but also has a high error rate.Therefore,how to obtain and effectively analyze and manage the existing expressway OD data by using the existing expressway operation data(especially the traffic data of toll stations)is an important issue in the current expressway intelligent construction.To this end,aiming at the generation and analysis of expressway OD data,this paper designed and implemented a Hadoop-based expressway OD analysis system,the specific work of which is as follows:1.A data preprocessing method was proposed to generate the basic data of expressway OD analysis for the data quality defects of expressway toll collection big data.The method ensures the validity of the data by judging and filtering the original data of the vehicles passing through the expressway toll station.In the case of incomplete data,the method adopts the method of completion or elimination based on historical data to obtain the subsequent basic data of OD analysis used to generate THE OD matrix.Meanwhile,in order to reduce the time of reading data when OD matrix is generated,the basic data storage model and auxiliary index table for OD analysis are designed to facilitate data storage and subsequent queries.2.In highway OD on the basis of analyzing the basic data,gives the highway OD matrix data related definition,and then designed based on the Key-Value of highway OD matrix data storage model,and based on the algorithm generated graphs of highway OD matrix series,to support the different types,different time periods anddifferent efficient OD matrix calculation and effective storage of objects.In addition,in order to reduce the calculation time of OD matrix generation and improve the calculation efficiency,a multi-Map Reduce job execution scheduling method with priority in short jobs of HIGHWAY OD matrix generation is presented.3.A highway OD data analysis system is designed and implemented.The system provides functions such as expressway OD data generation and management,expressway OD data query and visual display based on spatial and temporal dimension,and provides customizable query service and spatio-temporal aggregation query support.The system has been applied to the expressway network operation analysis application project in a province,and through the experiment based on the1-year real expressway toll station vehicle data,it shows that the system has a good effect in the expressway OD data generation computing performance and OD matrix data storage consumption.