Research On Object Context Information Recognition Method For Semantic Laser Interaction

In order to help the older and disabled persons with impaired speech and fine motor skills to perform unstructured tasks more efficiently using the Wheelchair Mounted Robotic Arms(WMRA),in this paper,a method of inferring action intention by selecting objects with laser spots and a method of semantic laser intuitive interactive operation method adapting to an unstructured environment was proposed.In this paper,we focus on the rapid and reliable identification technology of the "context" information of objects,such as the laser semantics,the category of the object and the subsequent action involved in the laser locking method.The key research contents include the design of semantic laser interaction framework,the recognition of laser spot context information,the design and detection of laser semantics,and the identification of reliable categories of any household objects being selected,etc.,Specific research contents are as follows:Design of laser intuitive interaction frame and recognition of li ght spot context information.Firstly,based on the International Classification of Functionality,Disability and Health(ICF),the household tasks of the elderly and the disabled were analyzed to obtain the set of objects and actions.Then,the feasibilit y of action intention recognition based on context clues is analyzed based on the idea of object function affordance.On this basis,combined with YOLO V3 algorithm and the alignment strategy of depth camera and color camera,the spatial position detection method of laser spots at any position in complex daily home environment was researched.Based on Point Cloud Library(PCL),the segmentation and extraction of arbitrary objects in the field of view of the depth camera and the calculation of centroid coordinates of point cloud are completed.The laser spots coordinates are compared with the centroid coordinates of each object through coordinate transformation to lock the object picked by laser spots.Laser semantic design and anti-shaking detection.Based on the principle of easy-to-use,two basic semantics of laser spot selection and always-on lock are designed,and the two semantics are combined to map common household task commands.Based on the algorithm of Support Vector Machines(SVM)the laser semantic was designed and in order to improve the robustness of laser semantic detection,hand-shaking data were randomly added into the training set to solve the hand-shaking problem of special users when the training set was established.Reliable category identification of any household object selected.Combined with the public data set and the data set made of a self-made household object,the dataset covering about 130,000 pieces and 42 types of objects covering the ICF home task were established,the recognition model based on YOLO v3 was trained and the research on the identification methods of everyday household objects was carried out.In addition,in order to ensure the fast and reliable identification of any additional object,an arbitrary new unknown object detection method based on a multi-weight model loading is proposed.Under the condition that only the original model and the new object data set are used,the new and old objects can be detected,and the catastrophic forgetting of the convolutional neural network can be overcome.The new object detection method based on the incremental learning idea is compared and analyzed.Integration and experimental research of laser semantic interactive operating system platform.Firstly,the hardware platform of WMRA was built based on the JACO manipulator arm,Weimeiheng electric wheelchair,Lenovo laptop and ordinary laser pointer,and the software and hardware integration of the laser semantic interaction System were integrated based on the Robot Operation Syste m(ROS).On this platform,the experiment test of reliable category recognition of arbitrary new unknown objects and the experiment of laser semantic interactive control for unstructured daily household tasks are carried out.the result of the experiment shows that the method based on multi-weight model loading can successfully detect any new object class and overcome catastrophic forgetting.On the other hand,simple interaction based on laser semantics can be used to realize action intention reasoning and assembly and execution of unstructured daily household tasks,and the accuracy can reach more than 92%.The experiments show that the method proposed in this paper effectively meets the needs of disabled users for ease of interactive operation,participation and adaptability to the environment.