Design And Implementation Of A Traffic Stream Data Real-time Processing System Based On STORM

Internet of things activates various industries now. Because of internet of things, dynamic data stream is extracted from behaviors of real life by IT industry which strongly improves the technology of data processing. Many vocabularies such as massive data, cloud computing are popular quickly. The not real and static data can’t satisfy traditional industry which wants to control real and dynamic data and increase ability to compete in their fields. For example, combining the IT service and traditional industry produces some new industries such as intelligent city, intelligent house, intelligent traffic and intelligent car. All of them benefit the long improvement of the application of internet of things and technology of cloud computing. To deal with stream data from various sensors, the technology of data processing is the key point, which traditional technology can’t meet the need of managing data because of the real-time, continuity, and limitless of data. Recently, the technology is still developing rapidly at home and board, some frameworks of stream computing and platforms of real-time computing still need to be improved.Under this situation, some relative state-of-the-art stream data processing technologies are studied in this paper, based on the research center named traffic real-time data stream computing platform, this paper proposes a suit of real-time data processing system combining distributed message system’kafka’and’mongodb’which is a document type No Sql database, and improves the web monitoring of storm to meet the need of personality like checking assignment running list of one computer. After that, to deal with the drawbacks in the above mentioned real-time platform, this paper optimize the performance of computing and allocation of computing resources. In order to do improve the performance, this paper defines a regime of task dispatching in storm which allocates different computing resources for different tasks. The method implements a’lightScheduler’which is a dispatcher in the default dispatcher of storm. It changes the strategy of allocating computing resources of default dispatcher, divides the resources on logic layer. Towards a computing task’s real demand, allocates the task to the assigned physical computing node, and if there is no special demand for tasks, the method apply the default strategy in storm, allocating them to physical node cluster in storm, at the same time, the resources already allocated by’lightScheduler’may not be occupied by other tasks, and reduces the network delay. The research and contribution in this paper are the following three points:1) Design and implement a real-time stream data processing system, which is different from the original platform and modifies its drawbacks, such as can’t meet the need of storage for heterogeneous data, can’t extend message queue, allocate resources orderly only.2) Towards the small computing cluster, proposes a special task dispatching algorithm for lightweight computing assignment, reaching the high performance level and logically allocation of computing resources.3) Improves the function of physical computer monitoring and the ability of submitting computing assignment by pages, and also keeps the original function of web monitoring towards storm.Experiment shows that this paper’s algorithm can allocate computing resources according demand when there are many computing tasks on the platform, and the performance of the task allocated by’LightScheduler’ is gone up by10%roughly.