Research On Resource Management Algorithm Based On Reinforcement Learning In Heterogeneous Multi-connection Networks

In the context of information globalization,the development of mobile communications is advancing by leaps and bounds,and innovative technologies such as cloud computing,blockchain,and big data have greatly changed the way people live,work and communicate.And applications continue to emerge,such as: automatic driving,smart home,telemedicine,etc.People put forward higher requirements for network transmission rate,network reliability and other performance.Therefore,it is necessary to integrate multiple standard networks to promote the "interconnection" system of the network.In order to achieve seamless roaming between heterogeneous networks,5G multi-connection technology has become a research hotspot at home and abroad.It enables user equipment to maintain connections with multiple access base stations at the same time,which can increase network coverage,especially for network edge areas,which can effectively improve communication.stability and increase the reliability of the network.However,users connecting multiple base stations will greatly increase the number of communication links in the network,making the network structure more complicated,especially causing ping-pong handover.Existing single-connection network resource management algorithms are no longer suitable for multi-connection networks,and new resource management schemes need to be explored for multi-connection networks.Aiming at the problem that mobile users frequently switch cells in a multi-connection network,the set of multiple base stations accessed by the user at the same time is regarded as the cooperative set of the user,and the cell handover problem is transformed into a cooperative set update problem,and a cooperative set management scheme is proposed.The innovations are as follows:First,this paper proposes a cooperating set update scheme with thresholds added in a multi-connection network.The cooperating set update in the scheme is divided into an initial stage,a decision stage and an execution stage.In the initial stage,the macro base station generates an initial cooperating set according to the signal reception strength of the small base station at the user.In the decision-making stage,the macro base station chooses whether to trigger the cooperating set update mechanism according to the signal to interference plus noise ratio（SINR）value of the user side.When the SINR of the user side is lower than the set threshold,it means that the cooperating set accessed by the user at this time cannot meet the communication requirements at all,then the update mechanism is triggered,and multiple base stations with higher current received signal strength are continuously selected to access,and the update of the cooperating set is completed.If the current cooperating set can meet the communication requirements,the original connection is maintained.In the execution stage,the user updates or does not update the collaboration set according to the decision stage.Second,this paper introduces reinforcement learning algorithm,uses the adaptive nature of reinforcement learning,and combines the researched collaborative set update scheme to propose a collaborative set update algorithm based on Deep Q-learning Network（DQN）.Experimental results show that the DQN-based algorithm also reduces the switching frequency while maintaining a low computational complexity.Third,DQN is only suitable for dealing with discrete low-dimensional action spaces.In order to make the algorithm more practical,this paper further introduces the Actor-Critic（AC）framework.Since the AC framework uses a deterministic policy function,it can also directly output actions for high-dimensional action spaces.This paper proposes a time slot allocation algorithm based on the AC framework to solve the base station time slot resources accessed by users in a multi-connection network.Simulating the system,the results prove that the time slot allocation algorithm based on the AC framework can achieve the performance close to the optimal solution with the computational complexity of the polynomial level.