Discrete-Continuous Action Reinforcement Learning With Hybrid Action Representation

In recent years,deep reinforcement learning has made rapid progress in solving sequential decision problems.Discrete-continuous hybrid action space is a natural setting in many practical problems,such as robot control and game AI.However,most previous Reinforcement Learning(RL)works only demonstrate the success in controlling with either discrete or continuous action space,while seldom take into account the discrete-continuous hybrid action space.One naive way to address hybrid action RL is to convert the hybrid action space into a unifified homogeneous action space by discretization or continualization,so that conventional RL algorithms can be applied.However,this ignores the underlying structure of discrete-continuous hybrid action space and also induces the scalability issue and additional approximation diffificulties,thus leading to degenerated results.In this thesis,we propose Hybrid Action Representation(Hy AR)to learn a compact and decodable latent representation space for the original hybrid action space.Hy AR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder(VAE).To further improve the effectiveness,the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction.Finally,the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original discrete-continuous hybrid action space.We evaluate Hy AR in a variety of environments with discretecontinuous hybrid action space.The results demonstrate the superiority of Hy AR when compared with previous baselines,especially for high-dimensional action spaces.