Design And Implementation Of Open Processing Service Based On Flink Stream Processing Framework

With the development of Internet of things and information technology,the number of sensors is increasing rapidly.The sampling frequency and accuracy of sensors are gradually improving,and a large number of streaming GIS data are continuously generated,which poses a great challenge to the real-time processing performance of GIS system.GIS has gradually developed from traditional offline static GIS to real-time dynamic GIS.Apache Flink,as a streaming engine with high throughput and low latency,is widely used in real-time streaming programs.However,the purchase and configuration of cluster server,the deployment of distributed stream processing engine and the management of distributed computing resources have caused some difficulties for small GIS system developers.The isomerization of real-time stream processing service resources is not conducive to the sharing and interoperability of services.Open Geographic Consortium(OGC)proposed Web Processing Service(WPS)standard in 2007.WPS defines a description specification for geospatial data processing,which is used to discover and share web processing service resources.However,the WPS Standard does not support streaming data processing,and the existing WPS services do not support streaming data real-time processing.The main research contents and achievements are as follows(1)Combined with WPS 2.0 Standard and Google Dataflow streaming programming model,the Streaming WPS model is proposed.The model defines how the streaming service provider publishes the sharable services through WPS Standard in the real-time streaming data processing scenario.WPS users can quickly build streaming data processing applications by invoking WPS Standard operation interface without deploying and considering the details of stream processing framework and algorithm.(2)Based on the service description and service interface model in OGC WPS 2.0Standard,it is extended to support streaming operation.This paper constructs a description model of WPS stream processing operation by mapping dataflow stream programming model and WPS processing model,and implements the model with XML schema.(3)Based on the WPS flow processing service model proposed in this paper,a WPS flow processing service system based on Flink framework is designed and implemented.Meanwhile,based on the "two passengers and one danger" vehicle realtime monitoring application scenario,the vehicle real-time monitoring service and client application are developed,and the performance of streaming service is tested.Aiming at the interoperability problem of streaming data processing service,this paper proposes a WPS streaming service model based on WPS 2.0 Standard,and implements the corresponding WPS service system.The results show that the system based on WPS stream processing service model can support the discovery,sharing and interoperability of stream data processing services,and GIS users can quickly build real-time GIS system through WPS operation interface.Compared with the traditional static GIS service or spark program,the real-time data processing performance of WPS stream processing service based on Flink framework is improved obviously,and the stream processing advantage is more significant with the increase of data volume.It shows that the model has positive significance for the sharing of heterogeneous stream processing service resources.