Research And Implementation Of Multi-application Communication Optimization Method In Low-latency Data Center Network Environment

Since the performance of traditional data center networks can no longer meet the growing network needs of distributed memory applications,new low-latency data center network emerges as the times require.The core technology of low-latency data center network is RDMA(Remote Direct Memory Access).RDMA provides low-latency and high-bandwidth benefits for data center applications by offloading the protocol stack to the hardware and implementing kernel-bypass,which can avoid the overhead of copying data from user mode to kernel mode in traditional network and eliminates the overhead of kernel processing.The traditional TCP/IP-based communication framework is no longer applicable for data center applications in RDMA environment and is replaced by the RDMA communication framework which provides a new set of communication architectures and mechanisms,bringing great changes to the traditional data center network communication architecture.In detail,RDMA is a gray box that offloads the control functions and network resources of the protocol stack to the network card,and exposes a variety of hardware interfaces to upper-layer applications to customize network transmission capabilities.However,the mechanism of this gray box brings two challenges to the deployment of RDMA in the data center: 1)application performance degradation caused by difficult parameter selection;2)scalability and fairness problem caused by difficult resource sharing.In order to solve the above problems,this paper conducts research work from the following three aspects:Firstly,in view of the problem of application performance degradation caused by the fixed parameter transmission mode of the current research work,a dynamic selection mechanism of communication parameters based on decision trees is studied.We have to analyze and collect different application features and resource states related to RDMA parameter selection in the data center and find out some relationship results between different RDMA parameters and application features,resource states based on theoretical analysis and comparative experiments.Furthermore,to attain the performance of native RDMA,we convert these relationship results into a lightweight decision tree,and set an appropriate parameter adjustment frequency.Then the decision tree can be used to dynamically select the best combination of RDMA parameters for the application during application operation,thereby optimizing RDMA application performance.Secondly,aiming at the fairness problem of resource competition between applications due to the existing work that does not implement a specific RNIC resource scheduling mechanism,the RNIC resource scheduling mechanism based on max-min fairness is studied.We construct a RDMA resource scheduling model based on max-min fairness based on the traditional NUM problem,considering the limited number of RDMA network connections and the linear relationship between RDMA resources to ensure the utility fairness between applications while optimizing the total utility of system resources.Finally,we have to design and implement a multi-application communication optimization system in the low-latency data center network which can provide a more abstract programming interface in response to the slow application innovation caused by difficult framework programming.Based on the real environment of infiniband cluster in southeast university data center,this paper combines theoretical research results with practice,designs and implements a prototype system,and conducts deployment and experiments.The experimental results in the actual environment of the southeast university data center show that the multi-application communication optimization method proposed in this paper can dynamically select the best RDMA parameters for applications and ensure the utility fairness between applications,achieving high-performance,high-scalability multi-application communication optimization.