Multi-objective Optimization For Fed-batch Fermentation Process Based On Reinforcement Learning

The features of the Fed-batch fermentation process contain strong nonlinearity, time-varying parameters, large time delays, and complicated real-time measurement of biological state variables. Thus, online controlling the yield, substrate and time consumption directly is difficult. Operating offline optimization becomes a main method to improve various objectives of production. Further, the complex optimization involves multiple, incommensurable and conflicting objectives. Such problems typically allow numerous solutions to exist, which means constructing Pareto-based optimization.On the other hand, in fed-batch fermentation process, there are 3-5 separate loops for controlling pH, temperature, dissolved oxygen and other state variables, which also face the complexity of the fermentation reaction. In this case, traditional control algorithm cannot achieve good result.Intelligent algorithm used in control and multi-objective optimization of complex system is in rapid development. As one of the most widely used reinforcement learning algorithms, Q-learning has the advantages of simple structure, learning without priori knowledge, and with fewer tuning parameters, which is suitable for complex system optimization and model-free control.In this paper, a design of a Pareto-based distributed Q-learning optimization strategy (PDQL) is presented to solve Pareto optimal flow rate trajectories for the lysine fed-batch fermentation process, in order to obtain optimal production targets. The Q-learning algorithm and Pareto sorting method were combined to generate the nondominated solution set and to make this set approximate the actual Pareto front. The strategy in this paper enhances parallel-searching capability, with the help of multiple randomly initialized groups of agents. These agents share the experience to improve the performance of Q-learning optimization. The result of PDQL optimization was compared to PSO with the aggregated function method. It generates much larger solution set which is in better distribution characteristics and is more close to the Pareto front directions. In addition, for neutralization control and temperature control in fermentation reaction, A multi-step Q-learning controller is designed. With the re-design of the error state and the advantage of Q-learning able to achieve model-free control, this controller overcomes the complexity of fermentation process. The results are compared with the PID controller, and prove the effectiveness of the new controller.