Research On Capture Target For Multi-dimensional Data Fusion Of Human-computer Interaction

In the aviation field,touch screens or control sticks are commonly used for operating multifunctional displays(MFD).However,due to the limited hand actions of operators during flight,gaze interaction technology is considered a forward-looking input method.However,only using gaze interaction has the "Midas touch" problem of determining whether the gaze is intentional or unintentional.Therefore,multiple interaction methods are generally used in combination.Among them,the combination of EEG generated by motor imagination(MI)and eye movement data is more common.However,the simple combination has some limitations and can not reflect the user’s intention well.Therefore,this thesis takes the interaction of fighter jet MFD as the research background,and based on multimodal interaction,multi-dimensional data fusion target capture research will be carried out on the physiological information generated by the operator,so that the computer can understand the user’s intention more comprehensively and capture the target efficiently.This thesis is mainly divided into three aspects:1)Feature extraction and analysis of eye movementTo study eye movement target capture,the attention mechanism ECANet was introduced based on Res Net50 to establish the regression model of Point of Gaze(Po G)estimation.The average error accuracy of this model in the self-collected data set can reach 22.84 pixels.The position of the MFD button was matched with the Po G for target acquisition.Extracting features and using SVM for preliminary target capture analysis,the average accuracy on the test set can reach 77.29%.2)Feature extraction and analysis of EEG dataA proposal for a research technique for capturing target through motor imagery electroencephalogram signals(MI-EEG)is presented.At the outset,the data synchronization acquisition experiment employed the picture-guided method to acquire EEG signals of motor imagery tasks,which were then preprocessed and visual features analyzed.Finally,using sequential EEG as input,the classical convolutional neural network Shallow Conv Net model was used to classify the operation intention of "confirmed capture" on the button target on the MFD,and the average accuracy rate on the test set could reach 90.95%.3)Multidimensional data fusion target capture modelBased on the preliminary analysis of the data in the previous section,this thesis discusses the synchronization of eye movement and EEG data in the synchronous collection experiment through three different methods of data fusion,the data layer,feature layer,and decision layer.Fusion experiments have been done mainly at the feature and decision levels.Using the feature concatenate method on the feature layer,the average capture accuracy rate can reach 78.82%.At the decision level,D-S evidence theory is used for fusion,and the average capture accuracy rate can reach 93.15%.Finally,the target capture scheme with decision-level fusion can achieve a higher target capture accuracy rate,and the target capture speed is at the top of the existing MFD target capture research.Providing a design idea for human-computer interaction research,thus achieving efficient collaboration between humans and computers.