Scanning Tunneling Microscopy Studies Of Surface States And Doping Effect Of Topological Insulator Thin Films

As a new state of quantum matter, three-dimensional topological insulatorhas recently attracted tremendous attention in the physics community. The bulkof a topological insulator is insulating, but there exist gapless metallic surfacestates, which are topologically protected by time-reversal symmetry. Bymolecular beam epitaxy technique, one can grow high-quality topologicalinsulator thin films with precisely controlled properties and can be used forexploring novel quantum phenomena in topological insulators and potentialapplications in spintronics and quantum computation.The surface states of the second-generation topological insulators Bi₂Te₃and Sb₂Te₃consist of a single Dirac cone structure on each surface. High qualitythin films of Bi₂Te₃and Sb₂Te₃have been successfully grown on graphenesubstrate by molecular beam epitaxy. In this thesis, we perform heteroepitaxialgrowth of Bi films and chemical doping on Bi₂Te₃and Sb₂Te₃substrates. Usingin-situ low temperature scanning tunneling microscopy/spectroscopy（STM/STS）, we systematically study the physical properties of the two systemsin nanoscale. The main conclusions are as follows:1. Confining topological insulator surface states with a closed boundary isfascinating and experimentally challenging. In this closed system,multiple-scattering of massless Dirac fermions at boundaries is predicted toinduce quantum interference. We have successfully confined the Dirac fermionsin Bi₂Te₃by bilayer Bi（111） films, which forms a new quantum resonatorsystem in analogy to optical cavity. The quantum interference nature isidentified by both spatial local density of state maps of STM and byenergy-dependent STS. The underlying physics of surface state resonantscattering is clarified, and the origin of surface state lifetime is discussed. Wealso investigate the growth kinetics of Bi film, its doping effect to Bi₂Te₃, andthe work functions of Bi and Bi₂Te₃films. We propose a schematic energy-banddiagram, which reveals band bending effect at the Bi/Bi₂Te₃interface.2. In Sb-doped Bi₂Te₃films, we observe a strong resonance peak at the impurity center. By probing its spatial and energy dependent STS, we demonstrate the coexistence of Dirac point and resonance peak at the impurity center. Coherent scattering of surface states around impurities is analyzed in real-and q-spaces. Based on the constant energy contour of Bi2Te3, we quantitatively discuss the origin of the wave vector involved in the scattering: nesting vectors q2connected to the electron vectors from next nearest Γ-K directions. We compare three kinds of samples prepared under different growth conditions, and determine the atomic structure of the impurity in films.3.（BixSb1-x）2Te3alloy films are a new promising class of topological insulator materials, by combining the electronic and structural properties of Bi2Te3and Sb2Te3. By fine-tuning composition ratio of Bi, we are able to control the position of the Dirac point with respect to the Fermi level. In the Bi-δ-doped Sb2Te3films, we analyze the dispersion change of and the evolution of Landau levels of surface states under magnetic field, as a function of doping level. Bi dopants suppress bound states induced by native Sb vacancies and make surface states approaching linear dispersions. Quasiparticle interference patterns related to surface impurities are also observed in the same samples. It is found that they are similar to the surface state scattering behavior in Bi2Te3.