| Deep reinforcement learning has achieved great success in many fields.At present,most deep reinforcement learning methods are based on historical information for planning.These methods don’t consider that future information may play a guiding role in the decision-making process.In addition,the current method of planning using future information is usually to establish an environmental model,and then use the data generated by the model to train the planning algorithm,but it is difficult to obtain an accurate model of the system in reality.In order to reasonably incorporate future information into the planning process,this paper studies how to use future information to assist decision-making.The main work and contributions are divided into the following two aspects:(1)The real environment is often partly observable,and the planning algorithms in the partly observable environment usually use methods such as POMDP and DRQN.The belief state representation in these kinds of method does not use the future information part.The future information usually contains relevant information to assist decision-making.The Contrast Predictive Coding|Action algorithm incorporates future information into the belief state representation,so that the representation can contain the key information of the global state.But the algorithm only uses it as a state representation,and does not apply it to some observable environmental planning problems.This paper uses Contrast Predictive Coding|Action to express belief states to solve part of the observable environment planning problem.At the same time,the next step predictive feature is also used as a decision-making basis to assist decision-making,which improves the planning effect of the algorithm,and demonstrates the effectiveness of the algorithm through experiments.(2)Existing methods that use future information to assist decision-making,such as the imagination-based algorithm proposed by deepmind,build an environment model through training data,and then generate a predicted trajectory based on the model.The predicted trajectory is combined with a model-free training strategy.But under normal circumstances,it is very difficult to obtain an accurate model of the environment in reality,and the inaccuracy of the model will lead to deviations in the forecast information,making the forecast information misleading for planning.Considering that the current state may have been experienced in the past,the method of extracting future relevant information from the training data becomes feasible.The existing methods of using the future information in the training data to assist decision-making usually adopt the difference method to find and compare,and the algorithm efficiency is low.This article studies how to quickly find future relevant information from training data to assist decision-making.The algorithm in this paper uses Gaussian random projection to increase the index,quickly search for similar past states in the training data,and find subsequent relevant information to assist decision-making,which improves the efficiency of the algorithm and avoids the problems caused by inaccurate modeling. |
