The Study And Improvement Of Q-Learning Algorithm Based On Agent System

Reinforcement learning is a kind of unsupervised learning method for agent to acquire optimal behavior sequence to adapt to unknown environments with a clue of reward. Now reinforcement learning is widely used in agent system, among which Q-learning algorithm is widely used reinforcement learning algorithm. But when the standard Q-learning algorithm applies to agent system, it has some problems.Firstly, consider reinforcement learning interacting with environment, agents learn its policy only through trial-and-error method. And agent adjusts its actions merely depends on these external signals, so the system learning speed is relatively slow. Secondly, Q-learning usually deals with discrete states and actions, but it is important to deal with continuous states and actions in order to adapt to real complex environments. Therefore how to solve that multi-agent learning in continuous space is a very important problem. So in this paper, owing to these problems we improved and extended the standard Q-leaning in order to improve the algorithm's performance in agent system. The main works are as follows:The first problem approached in this paper is minimizing the reinforcement learning's main disadvantage: the learning time. This paper presents a new algorithm, called heuristic Q-learning that allows the use of heuristics to speed up the well-known Reinforcement Learning Q-learning. A heuristic function influences the choice of the agent actions. Compared with the standard Q-learning, the new algorithm has a faster speed to converge and has better performance.Second, we proposed a kind of multi-agent parallel heuristic Q-learning algorithm. There are multiple agents in the learning system. Each agent exists in an independent environment. In a learning episode, each agent based on heuristic Q-learning algorithm learns independently. After a learning episode, the results of all agents are fused so as to achieve common result, which are shared by all agents in turn as the learning basis for the next learning episode. Experiments show the feasibility and validity of the given algorithm.Finally, in this paper, a new approach based on fuzzy inference system is proposed to improve the ability of the multi-agent reinforcement learning in continuous domain. We improve and extend Q-learning algorithm by combining with fuzzy inference system. Furthermore, we divide the complex problem into many sub problems in order to resolve the learning problem in complicated environment. Besides, we applied one kind of non- symmetrical expression structure to design the strengthen function, give the different reward and the penalty to the different movement. We tested the new algorithm in multi-agent hunting system; experiments show the step number of hunting agent become stable gradually, proved the feasibility and validity of the new algorithm.