Research On Key Technologies Of Massive Heterogeneous Data Processing In Smart Highway

Highway transportation artery as econom y,undertake traffic volume growth with the economic and social needs.In current highway construction management,operation management,maintenance management and service,etc,also presents some problems.Such as system of dispersi on,lack of unified coordinatio n,the inform ation isolated island,the lack of the comprehensive utilization of traffic information.With the full use of Internet of things,cloud com puting,artificial intelligence,automatic control,mobile Internet technology in the fiel d of transportation,the sm art highway also cam es into being.Smart highway rem ains fully aware of the highw ay to realize the depth of information fusion,data analysis and data m ining and uses the accu rate model and control system for m anagers and travelers to provide collaborative m anagement and intelligence services.But in the p rocess of building smart highway,perception of the data has th e characteristic variety,large amount of data and distributed storage.Therefore,data s torage,efficient data retr ieval,the depth of da ta fusion is cur rently smart highway one of the important research contents.This paper takes sm art highway as the ob ject of study,combines with th e characteristics of the constr uction and data of sm art highway,to res earch massive heterogeneous data processing relevant key technology basing on H adoop big data platform.It mainly has the following several aspects of research work.(1)We analyze cloud computing and Hadoop related technologies,m ainly including HDFS distributed file,MapReduce distribute d computing framework and HBase distributed database.(2)On the basis of the analysis of the sm art highway construction and data characteristics of the region,we build the overall structure of ma ssive heterogeneous data processing in smart highway,and realize the process of massive heterogeneous data processing.At the sam e time,according to the needs of users,we determ ine key technologies of m assive heterogeneous data processing in sm art highway,including data migration,data storage,data retrieval,etc.(3)The data storage,data m igration and da ta retrieval are re alized in th e key technologies of massive heterogeneous data in smart highway.First,we design the data storage structure,relational data table and HBase table mapping rules.Then,we propose a structured data m igration based on disk read-write scheduling,and an unstructured data migration scheme based on V FS Apache.Finally,we design a data retrieval strategy based on HBase two level index,which is combined with the data cache layer to improve the efficiency of data retrieval.(4)On the whole architecture of heterogene ous data processing in sm art highway,the key technologies of heterogeneous data processing in smart highway are verified.The experimental results show th at compared with the structured data disk read-write scheduler and default scheduler based on migration,with increase of the amount of data migration,migration efficiency improved significantly;design of unstructured data migration scheme supports m ulti thread con current operation,with the in crease of thread,migration velocity faster;the retrieval efficiency of the improved data retrieval method is higher than that of the original HBase,and the data cache retrieval efficiency is much higher than the direct from established index of HBase retrieval efficiency.