Research On Wheelchair Motion Control System Based On Eye Movement Information

Approximately 132 million people worldwide with disabilities need to use wheelchairs in their daily lives.Nearly one million of these people suffer from a movement disorder and have lost the ability to control their wheelchair with both hands.Intelligent products not only offer a new way of operating wheelchairs but also improve the comfort of using them.Existing smart wheelchairs are mainly voice-activated smart wheelchairs and EEG smart wheelchairs,which are unable to provide services to patients who have lost the ability to speak,and EEG smart wheelchairs require patients to wear other external devices that increase the burden on the patient.Therefore,there is an urgent need to provide an eye-movement controlled wheelchair for the above-mentioned people to solve the travel problem.In this paper,we design an eye-movement intelligent wheelchair control system,build a human eye gaze dataset and construct a convolutional neural network model to realise the motion control of the wheelchair by the patient’s eyes.Firstly,a virtual acquisition scene and a real acquisition scene are designed and built,and videos of 100 volunteers gazing in different directions are collected.The videos are processed frame by frame and turned into picture data with annotation information,from which 4200 face images are filtered.After pre-processing operations on these 4200 images,only the human eye images and their labels are retained.The human eye gaze dataset consisting of these 4200 eye gaze pictures in different directions is named the multi-environment attention gaze dataset.Next,the eye movement information is divided into two types: gaze in different directions and blink.A convolutional neural network（Gaze Network,Gaze Net）is built to determine the direction of gaze and a blink detection algorithm is used to determine whether a blink has occurred.The two action states,gazing in different directions and blinking,are associated with commands to control the wheelchair,i.e.gazing in different directions to control the wheelchair to the left,forward and right,and blinking to control the start and stop of the wheelchair.The Gaze Net network is built to meet both the lightweight requirements of the network and the recognition accuracy requirements.Once again,the human eye gaze direction estimation algorithm program and blink detection algorithm program are ported into the embedded device Nvidia Jetson TX2 to design an eye-movement intelligent wheelchair control system,which collects eye movement information through a camera,outputs wheelchair control commands,parses the commands through an Arduino,and uses 3D printing technology to create a mechanical device to achieve control of the wheelchair rocker direction.The upper computer application software is designed based on Py Qt5 to visualise the wheelchair movement status.Finally,a real wheelchair test environment is built to plan the wheelchair movement trajectory.Through training volunteers,the performance of the eye-movement wheelchair control system is tested in the real scenario built to verify the effectiveness of Gaze Net model and blink detection algorithm on the modified wheelchair system and to prove the practicality of this eye-movement control system.