| Quantum phase transiton of two-dimensional strongly correlated systems is an important topic in condensed matter physics. Many novel states found in two-dimensional strongly correlated systems not only have significant theoretical value,but also have promising potential in applications. The most remarkable character of strongly correlated system is that the value of the interaction between elactrons reached a comparable or even larger magnitude than electron kinetic energy,which makes the celebrated solid states band theory disable in describing the strongly correlated systems. However, dynamical mean field theory and quantum simulation experiment via cold atoms in optical lattice have provide feasible theory and experimental platform for the investigation on the quantum phase transition in strongly correlated systems.In this work we adopt half-filled single band Hubbard model to describe the fermions in Square-octagon lattice, Ruby lattice and anisotropic triangular lattice. For Square-octagon lattice and Ruby lattice, we choose cluster dynamical mean field theory to map the lattice model onto the cluster impurity model and use continnous-time quantum Monte Carlo method as an impurity solver. The quantum magnetic phase diagrams of both Square-octagon lattice and Ruby lattice are presented based on the calculations of single particle’s density of states and double occupancy by combining with the definition of magnetic order parameter. The quantum magnetic phase diagram which composed of interaction and temperature of isotropic Square-octagon lattice shows that antiferromagnetic and paramagnetic order can be found in both metal state and insulating one. At T=0.17,the antiferromagnetic metal disappeared in the quantum phase diagram which composed of anisotropic parameter and on-site repulsive interaction while other states still exist. In addition, the relation between the energy gap and the on-site repulsive interaction in each state of Square-octagon lattice is also discussed in detail. For Ruby lattice, we have presented the quantum magnetic diagram which composed of the on-site repulsive interaction and the temperature,the relation between the energy gap and the on-site repulsive interaction in each state at different temperature. Results show that system favours Mott insulating state at low temperature and transforms to spin-density wave state to paramagnetic metallic state with the increase of temperature and the interaction.For triangular lattice system, we adopt dynamical cluster approximation to map the lattice model onto an impurity model and also use the continnous-time quantum Monte Carlo method as an impurity solver. The effect of interaction and temperature on the quantum phase transition in anisotropioc triangular lattice is shown based on the calculation of single particle’s density of states, double occupancy and the Fermi surface evolution by using the combination of dynamical cluster approximation and continuous time quantum Monte Carlo method. Results show that system undergoes a phase transformation from Fermi liquid to Mott insulating state. Based on cold atoms in optical lattice, we proposed a feasible quantum simulation scheme for the investigation on the anisotopic triangular lattice with fermions, in which the hopping terms are closely related to the lattice con?ning potential and atomic interaction can be adjusted via the Feshbach resonance.The whole thesis is composed of following chapters, including introduction, the quantum magnetic phase transitions in Square-octagon lattice, the quantum magnetic phase transitions in Ruby lattice and the quantum phase transitions in anisotropic triangular lattice and the chapter of conclusion and expectation. |