| Vision-and-Language Navigation(VLN)is a task that allows the agent to navigate in accordance with language instructions and based on the current visual information within a limited range at different times to reach the target point in unknown environments.This task is of great significance to the realization from perceptive intelligence to cognitive intelligence.However,the existing research methods for processing perceptive information still have the problems of incomplete utilization,and lack practical significance for the realization of cognitive intelligence.Therefore,this paper proposes the VLN algorithm of agent based on LSTM(Long Short-Term Memory)network to solve the problems of information fusion,efficiency optimization and model generalization improvement,so as to make up for the shortcomings of existing methods and improve the potential application value of this task.First,aiming at the problem of incomplete encoding of multi-modal input information,a VLN algorithm based on LSTM network and combined with fusion of visual and language features is proposed.A cross-modal feature fusion method is constructed based on the structure of intra-modal-inter-modal-intra-modal.The pre-trained Res Net-152 modal and Bi-LSTM(Bi-directional LSTM)network combined with Bag-of-words model are used to extract the visual features and language features separately.With LSTM network as the core,combined with the monitoring and evaluation module of instruction following and the backtracking module of agent action selection,a complete navigation algorithm is constructed.Under Matterport3 D simulator,using R2R(Room-to-Room)dataset,the model is end-to-end trained.The effectiveness of the proposed algorithm and the performance of VLN task are tested in known and unknown environments to realize the agent VLN task.Secondly,aiming at the problem of success rate and trajectory length of navigation,a cross-modal VLN algorithm based on A2C(Advantage Actor Critic)deep reinforcement learning is proposed.Based on A2 C reinforcement learning algorithm,the Actor network is constructed,and the Critic network is designed using the hidden layer state of LSTM network.The reward function is designed according to the distance between the final position of agent and the target position.The difference of the distance between the agent position and the target position at time points apart is used as the standard for reward shaping.And then a deep reinforcement learning training method is constructed.The experiment results show that the VLN algorithm based on deep reinforcement learning can shorten the trajectory length of the navigation while ensuring the success rate of the task.Finally,in order to improve the generalization ability of the model in unknown environments,a VLN method based on data augmentation is proposed.Using the cross-modal feature fusion method,the LSTM network is used as the core to encode the navigation trajectory and visual information in the navigation task.Using the cross-modal fusion method and LSTM network which are different from the encoder,the instruction output is obtained by decoding.And then the data augmentation method Cro-Speaker is proposed.Reversing the direction of the navigation trajectory in R2 R dataset and applying environmental dropout on visual feature,the new data is synthesized using the proposed data augmentation method,and three training methods combined with data augmentation SD,CSD and SCSD are proposed.The experiment results show that using the proposed data augmentation method can obtain more accurate language instruction,and using the training method combined with data augmentation can obtain more diverse data,so that the agent can effectively complete the navigation task in unknown environments. |
