| Recognizing the existence of humans,estimating the human pose and attributes,and further reconstructing humans from images and videos are necessary abilities for the machine to achieve artificial intelligence.As one of the foundational tasks,human pose estimation(HPE)is also an important research problem in the field of computer vision.Recently,with the development of deep learning,human pose estimation models based on deep neural networks have achieved great process and breakthroughs,and are gradually deployed into various practical scenes and applications,including human-machine interaction,action recognization,social security,autonomous driving,video special effects,virtual reality and etc.The human body is an articulated object with a specific connected structure,but current HPE models based on deep neural networks don’t explicitly exploit or model such inherently structural relationships of the human body.They generally adopt a scheme of implicit modeling.It is expected the model can learn implicit relationships of human joints from large-scale data.As such,it’s hard for current models to acquire what image clues the model relies on to make predictions.There is still internal redundancy in the architecture design,resulting in low inference efficiency and poor real-time performance.And the model lacks certain interpretability and stability.To address these research problems and difficulties,this paper incorporates the concept of explicit modeling into the design of human pose estimation models with the support of deep learning related techniques to solve important problems in human pose estimation.The core is to explicitly take into account the structural relationships of the body joints and the priori knowledge of the huamn body structure in the model design.This mainly involves how to explicitly utilize and characterize the human body structure,and how to explicitly model the spatial relationships between different joints of the human body,the human body affiliation relationships between keypoints,and the temporal relationships of joint motions.Specifically,the granularity of modeling ranges from modeling the spatial relationships between single human parts,to the spatial relationships between single human joints,to the spatial relationships between multiple human joints,and finally to the 3D spatial relationships of single human joints and the temporal relationships.The research in this paper covers the following four aspects:(1)To solve this problem of modeling the relationships of parts in a single person,this paper proposes to divide the human body into different body parts guided by explicit human body structure knowledge,and then use different sub-networks to predict the human body joints on different parts.Since the local dependencies within different sub-parts are different but cannot be clearly described,the neural network architecture search can play the role of searching for specific differentiated neural network sub-architecture for each sub-part.Experiments show that,compared with the typical human pose estimation models,the finally searched neural architectures for specific parts are compact and lightweight,and can accurately predict the position of human joints.(2)To solve the problem of efficiently and explicitly modeling the dependencies between human joints,this paper introduces the Transformer model into the pose estimation task.It combines the advantages of the convolutional layer in extracting low-level feature and the advantages of the attention layer in modeling high-level relationships,and propose the Trans Pose model.Compared with the mainstream DCNN based human pose estimation model,this model achieves obvious advantages in accuracy,model parameters,and inference speed.Furthermore,the model’s attention maps explicitly reveal specific image-dependent regions that have significant explanations for the prediction.In this way,we can observe what image clues the model relies on for inference,and how the model deal with special cases such as occlusion and so on.(3)To solve the assignment of joints to different human instances,reduce the overall redundancy,and simplify the inference pipeline,we explore how to use the attention mechanism to measure the matching degree between pairs of keypoints for the association.Nevertheless,the naive attention does not fully meet the expectations.For accurate association,this paper proposes to use human instance masks to explicitly supervise self-attention to obtain instanceaware characteristics and designs a keypoint grouping algorithm based on pair-wise attention scores.At the same time,it solves the problem of human instance segmentation and simplifies the pipeline.(4)To solve the above problems,this paper proposes an explicit token representation scheme for human body,which abstracts the representation of human joint 3D rotation,human shape,and camera parameters from image features.They interact with image feature through Transformer models to learn prior knowledge and dependencies between joints from a large amount of data.Given an image,the updated tokens can estimate human pose and shape as well as camera parameters for 3D human reconstruction.Thanks to this explicit joint rotation representation,the temporal rotational motion of each joint can be captured separately through a temporal model,and then the information of the past frame and the future frame is used to regularize the estimation of the current frame,which can reduce local joint jitters,to improve the stability of video prediction. |
