Design And Implement An Edge Streaming Data Processing Framework For Autonomous Driving

In recent years,autonomous driving been considered as the best solution for future urban traffic and received a lot of attention.As a complex system,autonomous driving vehicles use a variety of sensors to sense the surrounding environment.A large number of sensors generate huge amounts of data every second,and real-time processing of these data is critical to the reliability and safety of autonomous driving vehicles.Streaming processing has become the preferred solution for processing these sensors data due to its high throughput and low latency.However,traditional streaming processing frameworks deployed in cloud-based data centers inevitably suffer from two problems: first,limited bandwidth resources can hardly support the transmission of a large amount of data at the same time;second,the data transmission process will cause large latency.The instability and high latency of the transmission process are unacceptable for latency-sensitive applications such as autonomous driving.To solve the above problems,this thesis proposed an edge streaming data processing framework for autonomous driving,which off-load computation and storage capabilities from remote cloud-based data centers to neared edge data centers.Based on a thorough analysis of the sensor data flow characteristics and task processing requirements of autonomous driving,we implemented our framework based on Spark Streaming.We proposed a fuzzy control based system load dynamic balancing mechanism,which can monitor and predict the traffic flow in a specific area,adjust the running state of our framework according to the changes in traffic flow and system workload to achieve lower latency while meeting the requirement of system throughput.We proposed a differentiated autonomous driving task scheduling algorithm that enables the identification of priority and urgency of tasks,which can meet the processing needs of different priority tasks.Experiments show that we proposed framework is able to accurately predict data flow changes by predicting short-term road traffic changes,and the daily average percentage error lower than 4%.We proposed fuzzy control based system load dynamic balancing mechanism can quickly handle the mismatch between data arrival rate and system processing capacity,converge them to an equilibrium state within minutes.Compared to the vanilla Spark Streaming,we proposed framework is able to reduce end-to-end latency by 35% when data arrival rate at a low level,can increase 21% throughput when data arrival rate at a high level with the cost of increase end-to-end latency by 25%.We proposed differentiated autonomous driving task scheduling algorithm can accurately distinguish different priority tasks and accelerate the execution of high-priority tasks,resulting in a 46% reduction in end-to-end latency for highpriority tasks compared to low-priority tasks.