Research On Testing Method Of Multi-platform Deep Learning Frameworks

Deep learning framework(DL framework)is the foundational infrastructure for the development and industrialization of artificial intelligence(AI)technology,which facilitates the deep learning(DL)techniques rapidly applying in many fields and extending to many platforms.Due to the complex structure in design and the large implementation differences under various platforms,there are inevitably some problems in DL frameworks,such as the implementation defects and execution insufficiency,which affect the actual performance of the DL models.Therefore,a quality assurance and testing research on multi-platform DL frameworks is of great significance,including the capability analysis to support DL tasks by the frameworks,the quality assurance of the framework implementations,and the framework testing methods that adapt to the characteristics of different platforms.However,there are still many challenges in these attempts.Specifically,during the quality analysis,there lacks a comprehensive understanding of the supportive capability of model construction(i.e.,the entire process of model development and deployment)by the frameworks;during the testing process,it still needs to address several bottleneck problems,such as the test case generation,the testing oracle selection,and test cost reduction.Aiming at above challenges,this thesis conducts a quality analysis and testing research on DL frameworks under multi-platforms from three aspects: First,it comprehensively analyzes the supportive capability and influence on model construction from the mainstream DL frameworks on PC,mobile and Web platforms.On this basis,according to the key problems found under different platforms,this thesis then selects the DL training frameworks on the PC platform,and the DL inference frameworks on the mobile platform to carry out further testing,respectively.The main innovative research work of this thesis are summarized as follows:? Perform an empirical study towards understanding how DL frameworks and platforms affect the DL model construction process.The differences in design and implementation of various DL frameworks under heterogeneous platforms have brought new challenges to the construction of DL models(i.e.,model development and deployment).To this end,this thesis conducts an empirical study on how the mainstream DL frameworks and different platforms affect the model construction,by analyzing and exactly comparing the execution performance and problems when DL models are developed using multiple frameworks and deployed to end-side platforms,based on several widely used neural networks,various dataset that are publicly available,and multiple real devices under different platforms.The results reveal the model accuracy degradation(i.e.,an accuracy decline over 5%)and robustness differences caused by the implementation discrepancies and defects of DL frameworks,as well as the compatibility issues,reliability issues,and the inefficiency problems during the models multi-platform migration and quantization on top of end-side DL frameworks.This study not only provides usage suggestions for the DL developers,clarifies the improvement direction for the framework vendors,but also lays the foundation for the subsequent testing techniques for DL frameworks on different platforms in this thesis.? Propose new methods for automatically detecting and localizing detects of PC-side DL frameworks.The DL frameworks on PC platforms play an fundamental role in catalyzing the process of AI software.However,it will inevitably bring unknown defects and even potential security risks due to the high implementation complexity and frequent evolution.To this end,aiming at verifying the logical correctness and implementation robustness of the DL frameworks,this thesis proposes an automated defect detection method based on multi-level mutation techniques and heuristic algorithms,as well as a fine-grained defect localization method based on the causal testing theory.Compared to previous work,the detecting method covers more types of defects with the help of a series of testing oracles,including the framework crashes,the not-a-number(Na N)exceptions,and the inconsistencies between different frameworks.The localization method narrows the defect range within the source code of frameworks,and provides a more comprehensive root cause analysis for the detected defects.In the real-world testing,above methods have achieved promising results and received positive feedback from the industry.In total,26 unique unknown defects across 5 mainstream DL frameworks(i.e.,Tensor Flow,Py Torch,CNTK,Theano,and Keras)are detected,7 of them have been confirmed or fixed by the framework development team,which further verifies the effectiveness of the proposed methods.? Propose a new method for analyzing and testing the mobile DL frameworks.The DL frameworks on mobile platform are typically encapsulated in the DLbased apps as binary libraries,which forms a black-box execution environment and further increases the testing difficulty in a direct manner.Additionally,the computational and resource limitation of mobile platform also put forward higher requirements for the testing cost.To this end,this thesis first collects 369 popular DL-based apps in the wild,and comprehensively analyzes the basic characteristics of these apps,as well as the mobile DL frameworks and models integrated inside them.On this basis,this thesis further propose an automated method for testing mobile DL frameworks on top of aforementioned DL-based apps,aiming at investigating how the embedded DL frameworks affect the robustness of DLbased apps.This method solves the bottleneck of locating testing entries of mobile DL frameworks in such a black-box environment,by combining the static DL API analysis of mobile DL libraries and the dynamic instrumentation on targeted DL-based apps.Through the interaction with PC devices on top of the aforementioned testing entries,this method further designs a variety of mutation strategies to automatically generate adversarial inputs for DL-based apps,which realizes the automatic testing pipeline and effectively improves the testing efficiency under the mobile platform.Finally,it verifies the effectiveness of the proposed method by conducting real-world testing on 45 popular DL-based apps.The results not only reveal the robustness issues that are widely existed in current DL-based apps,but also confirm that the mobile DL frameworks with frequent evolutions can reduce the robustness of DL-based apps.To sum up,this thesis focuses on the basic problem of the quality analysis and testing of DL frameworks.Taking DL frameworks that are widely used under multiple platforms as the research objects,this thesis designs targeted analysis and testing methods combined with the characteristics of the corresponding platform,and solves many testing challenges.The results reveal a variety of problems persisting in current DL frameworks,which provide useful guidance for the further quality evaluation and performance opti-mization of these frameworks.