Research On Panoramic Image Semantic Segmentation Based On Self-Supervised Learning

Panoramic images,as a type of image that can capture the surrounding physical space at 180° vertically and 360° horizontally,contain rich and complete environmental information,making them widely applicable in fields such as autonomous driving and virtual reality.Panoramic image semantic segmentation refers to the per-pixel classification on panoramic images,allowing for a comprehensive perception and understanding of the shape,category,and position of scenes or objects represented by pixel sets in the panorama.Currently,panoramic images are facing the problem of insufficient labeled data.Due to the wider view of panoramic images compared to the narrow view of flat images shot by regular pinhole cameras,the cost of semantic segmentation annotation for panoramic images has increased.Typically,panoramic images in datasets are represented in equirectangular format,which not only makes them convenient for storage but also records global information.However,the presence of distorted image content results in a sharp decline in the accuracy of existing planar image segmentation algorithms.Panoramic images contain latent structural context information,and designing models for panoramic features will help further improve the results of panoramic image semantic segmentation.Moreover,due to the large field of view and dense predictive nature of panoramic images,attention methods aimed at performance improvement entail high computational costs,necessitating a segmentation algorithm that balances accuracy and computational efficiency in real-time application scenarios.Furthermore,the process of intuitively viewing model-segmented panoramic images is cumbersome.Therefore,designing a system that combines model inference and panoramic image viewing functions will facilitate the broader application of panoramic image semantic segmentation.To address these issues,this paper conducts the following research:(1)In response to the issues of insufficient data annotation and image distortion in panoramic image semantic segmentation,this paper proposes a distortion-aware self-supervised method for panoramic images.By transforming planar images using distortion augmentation techniques composed of equirectangular projection and central cropping,existing large-scale unlabeled planar image datasets can be effectively utilized.Moreover,a distortion-decoupled self-supervised learning network is designed,which learns consistent features from distortion geometry and original appearance information,as well as distorted image features,to enhance the model’s backbone network’s perception of distortion.The backbone network generated by the training method can be transferred to the segmentation model,achieving state-of-the-art segmentation results on the Standford2D3D,SUN360E,and CVPG-Pano datasets.(2)To address the context problems and low segmentation efficiency in panoramic image semantic segmentation,this paper proposes a panoramic structure context-aware semantic segmentation method.Taking into account panorama distortion,gravity prior,and information characteristics,an inter-row attention module is designed to extract features from the height range and generate attention weights.An intra-row attention module assigns different feature extraction strategies to different rows,jointly improving segmentation accuracy.Based on the lightweight backbone network,it will segment images more effectively.Compared with the baseline method,our model on the SUN360E dataset has a 3.49 mIoU increase,and while maintaining effective computation and parameter volume,it has a 1.08 mIoU increase compared to the current real-time segmentation method for panoramic images.(3)To tackle the cumbersome process of viewing panoramic image segmentation results,this paper develops a real-time viewing and online segmentation system for panoramic image semantic segmentation.Based on spherical representation,the system allows users to interactively and intuitively view segmented images.The system incorporates the segmentation model presented in this paper and can quickly perform online inference for segmenting panoramic images.