Research On Offloading Strategy In Maritime Edge Networks Based On Reinforcement Learning

With the continuous development of our country’s marine industry,Unmanned Surface Vehicle are widely used in environmental monitoring,marine surveys,security and rescue in unknown and dangerous marine environment.There are more and more resource-intensive tasks such as surface rescue and salvage have emerged in the business requires of Unmanned Surface Vehicle.These changes have increased the energy consumption of Unmanned Surface Vehicle and put forward higher requirements for the time delay to complete the tasks.Mobile Edge Computing makes it possible for users to reduce latency by deploying computing nodes or servers at the edge of the network.Reinforcement Learning can maximize returns and achieve specific goals through learning strategies in the process of interaction between the agent and the environment.Reinforcement Learning can effectively solve the problem of offloading decision-making in mobile edge computing networks.Aiming at the maritime edge computing network,this paper uses the Reinforcement Learning technology to design the task offloading decision-making schemes in the two scenarios of single Unmanned Surface Vehicle and multiple Unmanned Surface Vehicle formation to optimize the execution efficiency of Unmanned Surface Vehicle computing tasks in the maritime mobile edge computing network.The work in this paper is summarized as follows:(1)For the single Unmanned Surface Vehicle offloading computing scenario,the surrounding large ships serve as the computing nodes of the maritime mobile edge computing network to provide low-latency offloading services for the single Unmanned Surface Vehicle.The Unmanned Surface Vehicle effectively reduces the task completion delay by offloading delay-sensitive tasks to nearby edge computing nodes,and faces changing of wireless channel conditions,computing resources of edge computing nodes when offloading as well as the energy constraints.Consider whether the task should be calculated locally or offloaded,and which edge computing node should be selected,the channel and communication model,task model,time delay model and energy consumption model of the single Unmanned Surface Vehicle offloading scenario are first introduced.Then,we propose an adaptive offloading decision-making algorithm UCB-AOD based on Multi-armed bandit to minimize the time delay.The simulation results show that the proposed algorithm can adaptively determine the offloading strategy according to the channel conditions and the computing resources of the edge computing nodes,and then obtain a lower delay compared with other edge computing offloading strategies under different positions and tasks.(2)The multiple Unmanned Surface Vehicle formation offloading decision-making scenario is managed by the host ship under the maritime edge network.As an edge node,the host ship is responsible for completing the offloading decision,making the multiple Unmanned Surface Vehicle formation offloading decision-making scenario is managed by the host ship under the maritime edge network.As an edge node,the host ship is responsible for completing the offloading decision,making the next judgment and providing computing resources for the offloading by allocating bandwidth to the multiple Unmanned Surface Vehicle formation at each time frame.However,affected by the channel environment and Unmanned Surface Vehicle task queue conditions,it is difficult to offload all computing tasks,and queue congestion will discard tasks and cause losses.Aiming at this problem,this paper establishes the channel model as well as task queue model,and then establishes an optimization problem aiming at maximizing the offloading reward variable of the entire multiple Unmanned Surface Vehicle formation.In addition,we propose a deep reinforcement learning based offloading decision algorithm DDPG-OD with Deep Deterministic Policy Gradient.Simulation results show that the proposed algorithm performs well under different task arrival rates and channel conditions.