| In this thesis we use the quantum cluster theory to investigate the quantum phase transition of the frustrated spin systems, including the spin-1/2two dimensional frustrated J1-J2model and stacked J1-J2-Jc model.We apply the cluster density matrix embedding theory to spin lattice system with some modifications. The reduced density matrix of the impurity cluster is embedded in the bath states which are a set of block product states. The ground-state of the impurity model is formulated with variational wave function. We test this theory in a two-dimensional (2-D) spin-1/2J1-J2model for a square lattice. The ground-state energy (GSE) and the location of the phase boundaries agree well with the most accurate prior results from the quantum Monte-Carlo and the coupled cluster methods. Moreover, this cluster density matrix embedding theory is cost effective and convenient for calculating the von Neumann entropy, which has connections with the quantum phase transition.Base on the cluster mean-field approximation, we propose a cluster multiple product states method and a cluster nested method. Both the methods add the nonlocal corrections to the cluster mean-field theory. We use them to study the ground-state of a frustrated spin-1/2Heisenberg anti-ferromagnetic stacked square lattice at zero temperature. By analyzing the magnetic order parameter and the geometric measure of entanglement, we find that the stacked model produces a new ordered magnetic phase because of the competition between the interlayer coupling, Jc, and the frustrated interaction, J2. The interlayer coupling, Jc, strengthens the long-range Neel phase, the new phase and the collinear phase and destroys the disordered quantum paramagnetic phase. Increasing Jc results in the new phase gradually replacing the quantum paramagnetic phase. The quantum paramagnetic phase disappears at the critical point J2=0.5and J*c=0.12. |