Real Time Multi-Objects Detection And Events Analysis By UAV-Human Collaboration

Reconnaissance is an important task for the Unmanned Aerial Vehicle Systems,however,kinds of the automatic objects detection algorithms in the Unmanned Aerial Vehicle Systems is not able to achieve the goal of no human intervention and how to make the human-in-the-loop system more intelligent and effective remains a key problem for researches.By analyzing the details in the objects detection task in the process of UAV real time reconnaissance,We designed the supporting methods between operator and the automatic detection algorithms,considered the possibility of executing the same task by different collaboration methods to improve the flexibility of the whole system,which made the automatic algorithms more like teammates to probeto carry out the task with the operator,maximizing the effectiveness of the whole system on current technical level.On the one hand,the auxiliary object detection system is able to reduce the operator's burden.On the other hand,putting human's attention mechanism and cognitive ability into the real time reconnaissance loop by some effective interactive operations such as the eye tracker,touch panel,mouse,displayer and so on,we can take advantage of the operator's heuristic information to highly improve the ability of detecting and tracking.On the base of the results of detection and tracking,we analyzed the key and unexpected events in the task and improve adaption of the system in different actual combat environments(1)Designing four collaboration methods corresponding to different human-computer participating ratio based on the coactive design principles and modeling the whole process of multi-objects detection.We analyzed the details,the contemporary ability and interdependence between the operator and the automatic detection algorithms in the task;designed four collaboration ways: H,HR,HOR and R;modeled the whole multi-objects detection process including the four collaboration methods especially,and simplified the model based on the signal detection theory,regarding the operator and the automatic detection algorithm as two detectors.(2)Aimed at the unknown complex environment in the UAV reconnaissance task,we designed the methods of switching the collaborating ways dynamically based on the state of the process of UAV reconnaissance and the objects detection and tracking algorithms in combination with the operator's visual attention mechanism.We evaluated the condition of the environment,the operator and the adaption of the automatic algorithms at real time and chose the best collaboration methods based on the model and achieve the goal of dynamic switching,which was validated by simulation;built a complex dataset including kinds of scenes to simulate the complex unknown environment where the UAV executed reconnaissance task and carried out the human factor experiments on this dataset,which validated the good performance of the switching methods: compared with HOG-SVM method trained in the initial dataset,we had a about 30% increase in the hit ratio,a 7 decrease in the mistake ratio and a 21% decrease in the miss ratio and compared with complete human operator detection,although a 1% increase in the mistake ratio,we had a 9% increase in the hit ratio and a 10 decrease in the miss ratio;We retrained the detector based on the operator's detection results and successfully detected the missed object before.We brought in the human's visual attention mechanism by the eye tracker to make up the lack of information for the two-dimension images using the heuristic information of the operator,which improved the speed and accuracy of the system in a low cost.(3)Considering the circumstances those were hard to model and expect in the process of UAV reconnaissance,we analyzed specific and unexpected events.We analyzed the specific events such as the objects' appearance,disappearance and the failure of the track algorithms,which would alarm when accident occurred and stimulate the switch of collaborating methods at proper time;built a hardware-in-the-loop simulation system combining the state of the process of UAV reconnaissance and specific event and validated our conclusion on it.