Research And Application Of High-Performance One-Way Data Acquisition Technology

With the development of Internet technology,the exchange of application data between non-equivalent security level domains has become increasingly frequent.Direct data interaction between non-equivalent security domains has led to security problems such as data leakage and hacker attacks,which have become important challenges for ensuring the internal trust network security of many government and enterprise units.Therefore,achieving highperformance and secure data exchange has become a major research hotspot at present.One-way transmission technology,as one of the important technologies to solve the problem of data exchange between trusted and untrusted networks,has been widely applied in many government and enterprise units,effectively protecting the security of internal trust network data.However,in the face of complex,heterogeneous,and massive data scenarios,the current one-way transmission technology has problems such as low data acquisition efficiency and low transmission performance.Therefore,this paper combines Remote Direct Memory Access(RDMA)technology with one-way transmission technology,and integrates the message middleware Kafka,firstly studying the high-performance one-way data acquisition technology based on RDMA,and designing the ODAR(One-way Data Acquisition by RDMA)architecture.Furthermore,based on this,the transmission optimization technology based on the ODAR architecture was studied.Finally,a highperformance one-way data acquisition system was built on the basis of both,aiming to improve data acquisition efficiency and transmission performance while taking into account the reliability of data transmission.Specifically,the main work and achievements of this paper are as follows:1)First of all,to address the issue of low efficiency in traditional data acquisition techniques and performance bottlenecks such as low throughput,high latency,and high CPU overhead in traditional one-way transmission systems,this paper combines RDMA technology with one-way data transmission techniques and integrates the Kafka message middleware to design a new architecture called ODAR(One-way Data Acquisition by RDMA).Furthermore,to overcome the problem of inaccurate distribution at the receiving end during the transmission of raw data,this paper introduces a reliability-based data encapsulation strategy.In response to the excessive delay in memory allocation and synchronous processing of data transmission,a dynamic memory optimization strategy is proposed.Additionally,to tackle the problem of underutilized network card bandwidth during data transmission,a novel priority-based data transmission scheduling algorithm is investigated.The experiments show that compared to one-way data acquisition technology based on UDP implementation,the ODAR architecture proposed in this paper has better transmission performance,improves data transmission throughput,reduces latency and CPU overhead,and the reliability-based data encapsulation proposed in this paper ensures the correct distribution of data to the receiver and greatly improves the accuracy of data transmission.The dynamic memory optimization strategy performs asynchronous processing of memory allocation and data transmission,reducing the latency in data processing,while the priority-based data transmission scheduling reduces the impact of fixed data packet header length on small data packets and improves the actual utilization of network card bandwidth.2)Secondly,due to the occurrence of packet loss and the inability to perform traditional retransmission operations during one-way transmission,this paper addresses this issue by designing an adaptive error correction technique based on the ODAR architecture.This technique utilizes erasure coding mechanisms to provide certain recovery capabilities for incomplete data and adaptively adjusts the ratio of redundant segments based on the current transmission rate.Furthermore,when multiple data nodes are being collected by the sending host,limited host cache resources can lead to head-of-line blocking,resulting in high latency for real-time data.To tackle this problem,this paper introduces a priority queue scheduling technique based on the ODAR architecture.It constructs multiple virtual queues and processes different types of data separately during data acquisition,giving priority to real-time data.It also takes into account high and low priority data,thus avoiding a "starvation state".Experimental results demonstrate that the adoption of adaptive error correction techniques effectively reduces data packet loss rate when the actual transmission rate does not exceed the threshold of the tested transmission rate.The priority queue scheduling technique based on the ODAR architecture reduces the latency of real-time priority data while considering both high and low priority data.3)Finally,based on the research work in 1)and 2),this paper designs and implements a high-performance unidirectional data acquisition system.The system is based on RDMA technology and integrates the message middleware Kafka,which can monitor the data acquisition,transmission,reception,and distribution rates,and persistently store the collected and transmitted data according to the needs.The current data operations are also logged for future review needs,improving the comprehensiveness of the system and facilitating user monitoring and management.