Research On Deep Learning Image Matching Algorithm For GUI Control Recognitio

The rise of agile development has led to a shortening of software development cycles,an acceleration in the frequency of iteration and delivery,and higher requirements for software product quality and efficiency.UI(User Interface)automated testing is the key to ensure software quality and user interaction,the first of which is GUI(Graphical User Interface)control recognition technology.The traditional recognition methods have typical problems such as poor adaptability and weak script migration ability,but based on the image matching method,the GUI control can be identified by calculating the similarity between the GUI control images,which has excellent performance in detecting GUI control deformation,and the script has strong cross-platform and high reusability.Compared with the traditional image matching method based on manual features,the deep learning method can extract deeper features in the image,which is robust to noise and deformation,and has achieved good results in aerial photography,remote sensing and other fields.Therefore,this paper studies the deep learning image matching algorithm in the field of GUI control recognition,focusing on the optimization and improvement of control image detection and control image matching algorithm,so as to improve the detection ability and matching accuracy of GUI control.The main work is as follows.(1)Aiming at the problem that traditional NMS(Non-Maximum Suppression)lacks position reliability factors,and the control needs more accurate center coordinates,it is proposed to apply Io U-guided NMS to YOLOv5,and add Io U(Intersection over Union)prediction branch to the predictor part of the network,so as to improve the positioning accuracy of the prediction box when the initial category and coordinates of the prediction box are not affected.(2)In this paper,the GUI screenshots of the mobile application are used as the data source,and the GUI control image matching dataset is constructed through the extraction and annotation of the images.Aiming at the problem that the fully connected layer in Siamese network needs to have a fixed input image size,a method of replacing the fully connected layer with a convolutional layer is proposed,and an attention mechanism module is introduced to improve the accuracy of the network.The training is completed on the custom dataset,and the matching network’s ability to distinguish similar control images is improved.(3)For typical cases where GUI control images are mixed with text,a deep learning text recognition model is introduced to separate the detection of text and the detection of control image,so as to improve the detection ability of text.Based on the above models,a UI automated test system is implemented and the application practice is carried out.The experimental results show that AP90 of the improved YOLOv5 is improved from22.8 to 24.0,and the accuracy of the improved Siamese network is improved from 98.24% to99.02%,and it is verified that the algorithm studied in this paper has a good application effect in the UI automated test system.