Research On Dexterous Prosthesis Control Method Based On Eye-tracking And Augmented Reality

With the advancement of technology,dexterous prostheses have made great progress in dexterity.Modern dexterous prostheses can reproduce multiple grasping modes of human hands.However,current control methods of dexterous prostheses can hardly satisfy the control of multi degrees of freedoms,which leads to low user acceptance.In order to improve the acceptance of users and achieve closed-loop control of dexterous prostheses,a new control method that is intuitive,robust,and interactive is urgently needed.In this paper,a method of using human visual channels based on eye tracking technology is studied,and the experimental verification of the method is completed,the results of experiment show that the new control method have more advantages than traditional control methods.Additionally,the augmented reality technology is used to realize intuitive control with active sensory feedback,which expands the bandwidth between the user and the dexterous prosthesis.Firstly,an eye-control human-computer interface(HCI)is designed based on the classification of grasp mode,and Fitts' Law evaluation index system is used to analyze the impact of the width factor of the grasp mode menu on the performance of the eye-control graphic user interface(GUI).The appropriate size of the grasp mode menu is selected and the eye-control GUI is compiled using Open CV library.Then,a 15-degree-of-freedom virtual prosthetic hand model was designed according to the structure of the human hand,and simulations of four grasping modes of the virtual prosthetic hand(cylindrical grasping,spherical grasping,tripod grasping,and lateral grasping)were realized.To achieve the active feedback from the dexterous prostheses to the user,a new eye-control HCI is made in Unity3 D using augement reality technology.The feedback information includes eye movement data,grasping mode,s EMG signal,multi-finger graping force and contact status.With the active feedback,the closed-loop control of dexterous prostheses is achieved.Finally,a dexterous prostheses control system based on eye tracking is built.The eye movement is used to select the prosthetic hand grasping mode,and s EMG signals is used to control the opening and closing movement of the prosthetic hand.The multi-objects grasping and transferring experiment verified the effectiveness of this new control method.The experimental results show that the selection time to complete the grasping mode is about 1s on average,which is far less than the electromyographical coding control method(larger than 4s).The grasping success rate is 100%,and the single grasping time is less than 16 s.