Research On DNS Tunnel Detection Scheme With Open Set Recognition

With the continuous development of network communication technology,network scenarios and services are becoming more and more complex,and various network security incidents are emerging in endlessly.Lots of network protocols that are designed only for availability at the beginning provide attackers with more opportunities to carry out network attacks.As the key infrastructure DNS ecosystem in the Internet,its protocol lacks data confidentiality and integrity protection,and has become the preferred covert communication channel for attackers.Many large-scale APTs exposed in recent years tend to build covert C&C channels with DNS.Bypassing the security control policy through the DNS tunnel,realizing the transmission of remote-control commands or stealing related sensitive data,it brings a serious threat to the network security environment.Aiming at the problem of DNS tunnel detection,existing detection technologies usually analyze the relevant rule features by reconstructing the captured DNS data packets into a complete data flow.Obviously,these post-event detection methods cannot meet the requirements of intrusion detection systems to respond to attack threats in a timely manner.In addition,existing detection techniques only use feature engineering to build recognition models based on existing knowledge sets.Even if a specific DNS communication method or signature is cracked,it will be ineffective against new malware and new data formats that emerge in the future.The existing detection technology obviously cannot deal with the emergent types,which leads to serious consequences of security incidents.In order to solve these shortcomings of existing works and effectively deal with the main network security threats brought by DNS tunnels,combining deep learning network and open set recognition model,this thesis proposes an effective DNS tunnel detection scheme.Aiming at the problem of real-time detection and timely data collection,the scheme in this thesis proposes to use DNS query domain name for detection in a single DNS request.The whole detection scheme mainly includes two parts: feature extraction and open set recognition.In the feature extraction part,this thesis designs different network structures based on neural network for the feature vector representation of DNS data,so as to avoid the limitation of manual feature extraction and improve the detection accuracy.In the part of open set identification,a new open set identification model OSEForest for DNS tunnel detection is proposed based on the relevant definitions of open set identification and the compact probability decay model used to limit the risk of open space.The model effectively limits the open space risk and can be applied to the problem of DNS tunnel open set identification.By setting open set data sets with different openness,this thesis establishes the index performance to evaluate the recognition effect of the open set recognition model,and evaluates the recognition effect of the proposed model and the existing models W-SVM and OpenMax.The model effectively solves the problem of DNS tunnel identification of unknown classes.In the research of model incremental update detection method based on deep feature extraction,in order to extract the deep semantics of DNS query domain names,this thesis constructs an improved Transformer network as a feature extraction network,which effectively solves the long dependence between characters and achieves a more efficient representation of DNS data feature vectors.And combined with the requirements of the intrusion detection system deployed in the existing network environment,this thesis further expands the definition of open set recognition,and proposes that the open set recognition model suitable for online systems should meet four functions: detecting unknown classes,marking unknown classes,selecting labeled samples added to the model and updating the classifier.In this thesis,an incremental update mechanism is added to the proposed open set recognition model OSEForest.The experiments show that the overall scheme has further improved the identification performance of DNS tunnels.