Research On Optimization And Improvement Of Feature Extraction And Traffic Flow Prediction Algorithms For Intelligent Transportation Big Data

As the development direction of the future transportation system,the intelligent transportation system has achieved rapid and in-depth development in recent years and has presented the characteristics of big data.The massive multi-source heterogeneous and real-time data in the intelligent transportation big data environment increases the complexity and difficulty of data mining.Feature extraction and traffic prediction are two types of data mining methods widely used in this field.And the feature extraction method includes two frequently used and representative algorithms for extraction of association feature and event feature.With the continuous evolution of the intelligent transportation data environment,traditional models and algorithms for association feature extraction,event feature extraction and traffic flow prediction have gradually shown problems such as reduced accuracy,low performance and efficiency.These problems have become the bottleneck of various high-reliability,high-efficiency,high-precision system applications.Therefore,it is of great practical significance to study how to optimize and improve these three algorithms to adapt to the development trend and meet the higher standard data requirements.This thesis aims to further studies the characteristics of parallelization and feature extraction of big data mining,summarizes the unique characteristics of machine learning applied to traffic data,analyzes the shortcomings of existing work,and combines with relevant experiments to propose three new optimization algorithms.The new algorithms can improve the accuracy and efficiency of existing related algorithms and improve the application effect in intelligent transportation systems.The main innovations contents of the thesis are as follows:(1)The purpose of the association feature extraction algorithm in the transportation field is association rule mining.However,a large number of disk I/O operations in the iterative calculation process of the mining algorithm can make the running platform inefficient,and the constant algorithm iterative strategy generates a large number of intermediate candidate sets,resulting in high space and time costs.Aiming at this problem,this thesis proposes an association feature extraction algorithm based on distributed parallel computing and adaptive strategy,which utilizes Spark's memory storage features and adopts an improved method to remove the generation steps of the traditional Apriori algorithm intermediate candidate set,and proposes an adaptive strategy based on the nature of the data set to find frequent patterns with higher precision and efficiency to achieve minimized time and space complexity.(2)In the data mining process of massive data and multiple attributes in the transportation field,there is a large amount of noisy and redundant data,resulting in unclear training data.Direct use of artificial neural networks increases the system complexity and computation cost.The overly complex network structure will lead to excessive learning time and face local minimization and over-fitting problems,and the mining efficiency is low.Aiming at this problem,by studying the advantages and existing problems of fuzzy sets,rough sets theory and neural networks,a granular neural network based on the newly defined fuzzy rough set concept is proposed to extract the domain knowledge of the data in the form of a dependency factor.This method used a granular structure to define the input vector and target value of the network and assigned dependency factors as the initial connection weights for the entire granular neural network,and then trained in an unsupervised manner using the newly proposed feature evaluation index minimization.After the training is completed,the importance of each feature is obtained from the weight update between the hidden layer and the output layer.(3)Traffic flow forecasting is a fundamental problem in traffic modeling and management.Most of the existing traffic flow forecasting methodologies and frameworks are based on shallow network models which adopt discrete independent learning and prediction modes for different roads.Some deep architecture models exist,such as deep belief networks(DBN).But it is difficult to provide the most favorable prediction results due to the congenital defects of the back-propagation method,such as slow convergence and local optimization.In order to solve these problems,a traffic flow prediction optimization algorithm based on multi-layer neural network architecture and multi-task learning is proposed.The model structure of the algorithm is a two-part multilayer network structure,including the underlying DBN and the top-level multitasking regression layer.DBN performs feature learning in an unsupervised manner,creating a multi-tasking regression layer on top of the DBN,embedding an echo state learning mechanism rather than a traditional backpropagation method for monitoring prediction.The model combines the advantages of DBN and echo state networks,and comprehensively considers the interaction of multiple roads through multi-task learning mechanism to improve the prediction accuracy.The thesis further studies the influence of different task grouping strategies on forecasting effects,the application of homogeneous and heterogeneous multi-task learning in traffic flow forecasting and proposes a grouping method based on top-level weights to make multi-task learning more effective.