Human Pose Estimation And Visual Synthesis With Wireless Signals

Wireless sensing can perceive human activities in a non-contact and non-line-of-sight manner and has the all-weather characteristic,which has various applications in smart homes,emergency rescue,and social security.Therefore,wireless sensing technology has drawn lots of attention in recent years.In particular,wireless-based human pose estimation can represent fine-grained human activities,which means more intuitionistic and more accurate human perception.Taking one step further,it will be a great innovation to realize the all-weather human activity visualization by combining wireless signals with visual information.However,wireless-based human pose sensing faces challenges such as low estimation precision and heavyweight model structure,which limits existing technologies from practical applications.Besides,the multimodal fusion of wireless signals and visual information also faces technical challenges.To tackle the above challenges,inspired by related theories and applications,this dis-sertation conducts comprehensive research: from high-precision to lightweight wireless-based human pose estimation to expand the application scenarios of the system,from wireless single-modal perception to multimodal fusion of wireless and vision to achieve all-weather human activity visualization,from the use of mm Wave radars to utilizing home WiFi devices to reduce hardware costs for technology implementation.Specifically,the main contributions of this dissertation can be summarized as follows:1.A mm Wave-based high-precision human pose estimation model is proposed.To address the problem that wireless-based human pose estimation is low-precision,which is mainly caused by the structure difference between the wireless signals and the human poses,a human pose estimation method based on the optimal transmission theory is intro-duced.By mapping the mm Wave signals to the human pose domain in the feature space first and then estimating the human poses,high-precision prediction can be achieved.2.A mm Wave-based lightweight human pose estimation model is proposed.To meet the demand for portable wireless-based human perception systems in practical applica-tion scenarios,an efficient mm Wave-based human pose estimation framework constructed from the whole to the local is designed.Firstly,human-reflected signals are cropped by lo-calizing human bodies,then pose information can be effectively extracted from the small-size cropped signal heatmap based on a pose feature selection mechanism,and finally,the human pose is estimated from the extracted pose information.Through the above frame-work,a lightweight model that can be deployed to the mobile terminal is achieved while ensuring the accuracy of pose estimation.3.A multimodal-based human pose visual synthesis model is proposed.To tackle the limitations of wireless single-modal sensing that lacks visual features,a multimodal model that fuses visual information is proposed.In an adversarial learning framework,the mm Wave-based human pose representation can be accurately extracted based on feature correlation matching,which is then fused with visual information by adjusting feature dis-tributions,and finally,corresponding human pose images can be synthesized.Through the above framework,the visualization of wireless-based human pose perception is achieved.4.A human pose perception model based on home wireless devices is proposed.To reduce the hardware costs caused by the mm Wave radar,a human pose perception system based on home WiFi devices is proposed.Firstly,a coarse-grained human pose heatmap is synthesized from the WiFi signals,which is then fused with visual information pixel by pixel based on an attention mechanism,and finally,corresponding human pose images can be synthesized.Through the above framework,the WiFi-based integrated human pose estimation and visual synthesis are achieved.In summary,this dissertation somewhat solves the problems existing in the previous technologies and also visualizes the wireless-based human pose perception through the combination of visual information.Furthermore,integrated human pose estimation and visual synthesis based on common WiFi devices are also achieved.This dissertation makes technical and methodological contributions and also provides new research perspectives for subsequent works.