Light Field Representation,Rendering And Display Technology For Virtual Reality

Representation,rendering and display technologies play an important role in virtual reality system.In order to achieve immersive and realistic user experience,virtual reality applications need to present high-resolution,high frame rate,large field of view and highly realistic images for users,which puts forward very high requirements for the quality and performance of related technologies.The scene reconstruction technology can reconstruct the 3D mesh and surface material of the scene from the images or point cloud data.However,the quality of reconstructed model is poor and requires huge effort on post-processing.Although modern rendering methods can achieve complex and realistic lighting effects,the rendering process needs a lot of computing resources so can not achieve real-time rendering performance.In addition,the images displayed in HMD(Head-Mounted Display)lacks of focus cue,which limits natural view experience and causes users to suffer from various degrees of dizziness,visual fatigue and other adverse reactions.In recent years,with the concept of light field,many novel graphics methods and technologies have been developed,providing new solutions to the above problems.The light field rendering technology can analyze and extract the light field of the scene from multiple photos,and then synthesize realistic images at novel views.At the same time,this process does not involve high-precision mesh model and complex and time-consuming rendering methods.The state-of-art method in this area uses neural network to encode the light field,achieving higher rendering quality than previous methods,and showing a broad application prospect as the content creation and consumption performance of virtual reality in the future.Multi-layers light field display technology can display complete light field information,which lets HMD to have the ability of displaying focus cue.However,the research work related to light field has not considered the characteristics and requirements of virtual reality applications.First,existed neueral light field models aimed at the object-centered viewing,so it is not proper for the egocentric 360° viewing;Second,the neueral light field rendering is computational expensive,which limits its application in virtual reality;Moreover,the core algorithm of the multi-layers light field display technology is the light field factorization algorithm,which also has the problems of time-consuming and can not meet the display frame rate required by virtual reality.This paper studied these problems and proposed corresponding solutions.The main achievement of this thesis is a series of optimization methods of light field technologies in a complete graphics workflow including representation,rendering and display for virtual reality applications and devices.Specifically,this thesis has the following contributions.(1)To solve the problem that the existing neural light field model is not applicable to the 360° egocentric scene,this thesis proposed the MFS-Ne RF(Multiple Fast Spherical Neural Radiance Field)model.In MFS-Ne RF model,spherical sampling strategy is introduced to replace the uniform sampling strategy based on Cartesian coordinate system in existing methods.The spherical sampling strategy is based on the egocentric spherical coordinate system.The experiment shows that the neural light field based on the spherical coordinate system and spherical sampling strategy can more effectively express the 360° egocentric scene,and achieve higher quality than the existing methods for both the near objects and far objects in the scene.On this basis,this thesis further proposed a concate- nated samples inference method and a multiple neural light field structure,which leads MFS-Ne RF model to achieve a significantly better rendering quality and an acceleration ratio of about 66 times than the existing methods in the 360° egocentric scene.(2)To solve the bad performance of the existing neural light field rendering method,this thesis proposed the foveated neural light field rendering method.considering the characteristics of HMD images with large field-of-view,this thesis introduced the idea of foveated rendering into the neural light field rendering by using the perceptual characteristics of human vision for monocular images and binocular stereoscopic images.Based on the distribution of monocular visual acuity and binocular stereoscopic visual acuity of human visual system,this thesis pruned the number of rendering pixels and the capacity of neural light field,which greatly reduces the computational cost of neural light field rendering.The experiment shown that the foveated neural light field rendering improves the rendering performance to about 50 FPS without sacrificing the perceptual quality of the rendered image,and achieves the performance required by the virtual reality application.(3)To solve the time-consuming problem of the light field factorization algorithm,this thesis proposed the real-time light field factorization algorithm.Firstly,the light field factorization problem is expressed in the rendering framework,and several key steps in the algorithm are transformed into operations that can be processed through the GPU rendering pipeline,so as to speed up the solving process by using the advantages of GPU parallel computing.Secondly,this thesis proposed a novel initialization strategy: depth-aware initialization strategy.This strategy assigns a more appropriate initial value to the display layers by using the scene depth information, thus accelerating the convergence.The experimental results shown that the light field factorization algorithm proposed in this thesis achieves a computational speed of about 60 FPS,which meets the performance requirements of virtual reality applications.In general,this thesis studied the application of state-of-art technologies in the three stages of light field representation,rendering and display in virtual reality systems,and proposed a series of optimization solutions for the characteristics and requirements of virtual reality,leading these technologies to meet the requirements of virtual reality systems,such as quality,performance and applicability,thus promoting the application of light field technology in virtual reality to a certain extent.