Research On Adaptive Panoramic Video Streaming Based On Latent Variables Of GAN

Panoramic video can provide rich information about the full range of scenes,providing users with a sense of interaction and immersion.However,panoramic video requires a resolution of 4K and above to meet users’ viewing needs.As a result,panoramic video requires greater storage and transmission capacity than traditional video.In order to support interactive performance,the current method of adaptive transmission of panoramic video is to downsample or quantize the panoramic video in advance to obtain video content of different bit rates,and then select the appropriate video content for transmission based on the real-time network bandwidth value.However,this method does not consider the impact of different bit rates of video content on the reconstruction quality,resulting in the reconstruction of the image quality using/that uses this video content cannot obtain the optimal result of rate distortion.In recent years,video reconstruction based on deep learning has become increasingly popular,which provide new ideas to solve the above problem.The main research elements of this topic are as follows.(1)To address the problem that in the current adaptive transmission method of panoramic video,the video versions with different bit rates cannot obtain rate distortion-optimal reconstruction results,the thesis proposes a rate distortion optimisation algorithm based on GAN network latent variables.First,the thesis creates a time-space domain similarity map that can reflect the time-space domain similarity content of panoramic video,inspired by the pair of polar planes,starting from the time-space domain similarity content between video frames.GAN network has excellent generative capability,while the multilayer convolutional network can extract the more abstract features of the original image.Thus,the thesis constructs a joint network combining multilayer convolutional networks and GAN networks,and calls it the LVAS(latent variables of GAN)network.This network is trained by inputting a spatio-temporal similarity map,which can extract the low bit rate data of the panoramic video as well as learn the spatio-temporal similarity content to improve the reconstruction quality,and the extracted low bit rate data is called latent variables.Finally,in order to achieve a compromise optimization between the bit rate of latent variables and the image quality,a loss function based on rate distortion optimization in the LVAS network network is constructed to improve the coding efficiency.(2)To address the problem that in the current adaptive transmission methods for panoramic video,the transmitted data bit rate is likely to be much lower than the continuously varying network bandwidth,resulting in wasted bandwidth,the thesis proposes a code rate control method based on latent variable quantization of the GAN network.To solve the latent variable code rate and bandwidth adaptation problem,the size of the latent variable code rate needs to be controlled.In the constructed latent variable learning model,this thesis introduces quantization parameters to encode the latent variable for the purpose of controlling the latent variable code rate.Since adjusting the quantization parameters can control the code rate of the latent variable,this thesis can then achieve the adaptive bandwidth change of the latent variable code rate.Therefore,a relationship model between the quantization parameter and the latent variable code rate is constructed,and is applied to the rate distortion cost function of the latent variable,so that the latent variable code rate can be adapted to the bandwidth while maintaining the optimal reconstruction quality.In the experimental part,test experiments are conducted on six panoramic video sequences and the proposed GAN network-based panoramic video adaptive streaming method is compared with the latest two panoramic video adaptive streaming-based methods.The results show that the proposed method achieves an average bit rate saving of 56% on BDBR in terms of rate distortion performance and a 2d B improvement in the mean PSNR value compared to the two leading-edge methods.In terms of the code rate control algorithm,there is a 1.53 d B improvement in the mean V-PSNR for the same bandwidth.