Multi-task Learning Based Intelligent Vehicle Environmental Perception Method

Autonomous driving has become a popular research direction in recent years,and its demand for perception,decision-making,control and other algorithms has promoted the development and implementation of related technologies.As one of the core technologies of autonomous driving,perception is the premise for planning,decision-making and control to be effective,and also the basis for the implementation of autonomous driving technology.The real-time and accuracy of environmental perception largely determine the safety of automatic driving,which often requires processing data from various sensors and executing multiple perception tasks at the same time,which brings huge pressure on the hardware computing power of the on-board computing platform.Multitask learning can use a single neural network to realize the function of multiple perceptual tasks.Compared with designing neural networks separately for each task,multitask learning can reduce the total number of neural network parameters,the deployment complexity,and reduce hardware resource occupation.In addition,the task of autonomous driving requires the use of low-power embedded on-board computing platform.How to efficiently deploy neural network on the embedded platform has become a problem that needs to be solved in the landing of neural network in autonomous driving.Therefore,it is of great significance and value to study the multitask learning based perception algorithm and the embedded deployment acceleration for the perception task in the autonomous driving scenes.This dissertation studies the application of multitask learning and embedded deployment of neural network in autonomous driving scenes,and improves and innovates on the basis of the existing methods in these fields.The idea of multitask learning is to integrate two or more perceptual tasks into a single neural network by using parameter sharing.Compared with a single task neural network,multitask learning can reduce the total number of network parameters,and parameter sharing can implicitly learn the feature correlation between different tasks,and achieve the effect of mutual promotion among tasks.Therefore,multitask learning is more suitable for autonomous driving.In this dissertation,multitask learning is taken as the main research content,the mechanism of influence among different tasks is explored,and some efficient implementation methods of multiple perceptual tasks in autonomous driving scenes are studied.According to the application of multitask learning in different perceptual scenes,this dissertation studies multitask learning from three aspects: intertask application,single-task application and inter-modal data application.In addition,the perception algorithm will eventually be deployed on the embedded chip,so this dissertation studies the acceleration of the deployment of perception algorithms on the embedded chip to ensure that the algorithms can be effectively applied to the real scenes.At present,most researches in the field of multitask learning focus on two aspects,one is the form of parameter sharing among tasks,and the other is the training balance of multitask network.These two issues are the key to the effectiveness of multitask learning and the basis of the research,but there are still some problems in the existing methods.First of all,in terms of the form of parameter sharing,most of the mainstream studies simply share part or all parameters of the encoder,and realize feature sharing by eavesdropping mechanism,while ignoring the correlation among tasks,which requires additional guidance to be learned by the neural network.For the balance of multitask training,most of the existing methods are to manually or use additional modules to design weights for the loss function of each task,which requires trial and error,and increases the complexity of neural network training.In addition,existing methods only design multitask learning network for multitask under single modal data,and few application studies on multi-modal data or single task scenes limit the application of multitask learning.For the research of neural network deployment on embedded devices,due to the limitation of hardware resources of embedded chip,such as memory,bandwidth and computing power,there is a lack of efficient and flexible neural network embedded deployment architecture.In regard of the above analysis,this disseration is studied from the following contents:Firstly,the application of multitask learning in dealing with multiple tasks is studied to explore the interaction mechanism between different tasks,and a multitask learning based multi-object tracking and segmentation network is proposed.In order to improve the interaction between the two tasks,not only the strategy of sharing encoder is applied,but also the information transmission channel is established between the instance segmentation decoder and the multi-object tracking decoder.The influence of background information in the multi-object tracking task is effectively reduced by using the results of instance segmentation.Aiming at the training imbalance problem of multitask network,an optimal geometric mean loss strategy was proposed to reduce the impact of training imbalance without introducing additional parameters and training complexity.Secondly,the application of multitask learning in dealing with a single task is studied to find new ideas of multitask learning application,and a multi-task distillation based endto-end lane detection network is proposed.The detction of the two representation forms of lane,segmentation mask and polynomial model,is regarded as two different tasks,and a multitask network composed of a single purpose is established.The ability of sharing encoder to extract lane features is improved by taking segmentation mask based lane detection as auxiliary supervision.Aiming at the semantic relationship between two different representation forms of lane lines,an in-model knowledge distillation strategy is proposed to transfer clear lane geometry semantic information from the segmentation decoder to the polynomial decoder,so as to improve the prediction accuracy of the polynomial model of lanes.In addition,aiming at the application of multitask learning in multi-modal data scenes,a 3D semantic segmentation network based on multitask learning and multi-modal data fusion is proposed.In order to solve the problem of missing or inconsistent labels of different modal data during training,a weak supervision loss is proposed to propagate the labeled data information to the unlabeled data.In order to guide the data feature sharing of different modals,an attention-based feature fusion algorithm is proposed to screen out the features that have positive effects on the results.To solve the problem of reliability inconsistency of different modal data in different scenes,a self-confidence based late fusion strategy is proposed to improve the robustness of 3D semantic segmentation results.Finally,aiming at the deployment of neural network on embedded devices,a mixedpruning based framework for embedded neural network acceleration is proposed on FPGA.In order to solve the problems of limit on-chip memory and low data bandwidth,a model compression strategy based on mixed-pruning is proposed,which can reduce the number of parameters and hardly affect the network performance.In addition,a row tile feature map access approach and a sparse matrix storage strategy are designed for the compressed network to further reduce the memory and bandwidth pressure.In order to solve the problem of lack of open source available neural network library in FPGA,some computational processing engines with flexible hardware configuration are designed to deploy the common structure of neural network on FPGA.An efficient neural network acceleration framework is designed to realize the deployment of neural network on FPGA,and the deployment of some common neural networks and neural networks in this disseration are evaluated.The above research contents are equipped with experimental evaluation and result analysis to prove the feasibility and effectiveness of the researches of this disseration.In addition,the proposed algorithms are also verified by experiments on real autonomous driving scenes or embedded devices,which shows that the research contents of this disseration has the ability of landing application.