Study On Earthwork Allocation Based On Discrete Q-learning Algorithm

In the construction process of water conservancy and hydropower engineering,the balance of earthwork allocation is directly related to the quality,cost and schedule,and it is one of important problems in construction organization design and construction management of water conservancy and hydropower engineering.For large earth-rock dam projects,the amount of excavation and filling works is often huge.Many construction links need to be considered during the earthwork allocation,such as excavation,filling,transportation,material storage and so on.Traditional earthwork allocation problem is mostly solved by building linear programming,large-scale system decomposition and coordination,dynamic programming and multi-objective programming models,but there are some limitations.With the development of artificial intelligence and machine learning theories,reinforcement learning algorithms have been used effectively in many fields.Aiming at the conventional earthwork allocation problem in hydraulic and hydropower projects,this paper tries to use discrete Q-learning algorithm to construct and solve the earthwork allocation model,and compares it with the conventional linear programming method to prove the feasibility of the algorithm.Then,for the dynamic earthwork allocation problem,an earthwork allocation based on the combination of neural network and discrete Q-learning algorithm is proposed,which provides a basis for solving the coordination problem of excavation and filling.The contents and structure of this paper are as follows:(1)Based on the analysis of the current situation of the domestic and foreign researches on earthwork allocation and the existing problems in the construction of water conservancy and hydropower engineering in China,the key contents and research methods of this paper are put forward.(2)On the basis of the systematic analysis of the earthwork allocation,the paper focuses on the conventional earthwork allocation problem with determined excavation and filling time and the dynamic earthwork allocation problem with uncertain quantities of excavation and filling activities of each period,which provides the basis for the subsequent earthwork allocation research with Q-learning algorithm.(3)Aiming at the problem of earthwork allocation with known quantities of excavation and filling works in each period,the state,action and reward matrix of Q-learning algorithm is constructed,and the method of earthwork allocation based on Q-learning algorithm is discussed.The feasibility of the proposed algorithm and the reasonableness of the Q-learning model are proved by comparing with the integer programming algorithm of earthwork allocation.It provides a theoretical basis for the reinforcement learning algorithm to solve the problem of conventional earthwork allocation.(4)Aiming at the problem of dynamic earthwork allocation problem with certain total construction time,total quantity and uncertain quantities of excavation and filling activities in each period,an earthwork allocation method based on the combination of neural network algorithm and discrete Q-learning algorithm is proposed.The method predicts the reachable strength of excavation and filling in each stage with neural network,then takes the allocation quantity,excavation and filling pressure coefficient as the state,the excavation(or filling)quantity and the coordination rules of excavation and filling in each stage as the action and constructs a compensation matrix based on immediate and long-term returns.(5)Aiming at the conventional earthwork allocation problem and the dynamic allocation problem,combined with two water conservancy and hydropower engineering examples,the paper calculates and analzes earthwork allocation problem with discrete Q-learning algorithm,which proves the validity of the model.