Unconventional symmetry breaking in strongly correlated systems

New and interesting phases of matter can be stabilized in frustrated and strong correlated systems. In this thesis, we study a variety of exotic symmetry broken states that arise when conventional or natural ordering tendencies have been suppressed. We consider several diverse examples, drawing from frustrated magnets and Kondo lattice systems. We first explore this topic in the Z[Pd(dmit) 2] 2 quasi-two-dimensional organics salts, where a slave rotor approach is used to explain a variety of phases that are found in the vicinity of the metal-insulator transition; these include a spin liquid, valence bond solid and unconventional superconductor. Secondly, we propose a dotriacontapole density-wave order appearing out of the heavy Fermi liquid in URu2Si 2 as the so called hidden order phase of this material. This unconventional order parameter reproduces recent magnetic torque experiments and provides a simple explanation of the link between the hidden order phase and the pressure induced antiferromagnet. Thirdly, we study Kondo lattice systems where the local moments are frustrated. We show that if the frustrated moments realize a non-trivial spin liquid, then any hybridization with conduction electrons breaks some spatial symmetry. We apply this symmetry breaking mechanism to a model of the material Pr2Ir2O7, providing a theory for the anomalous Hall effect that appears at low temperature. Finally, we consider possible realizations of Kitaev's exactly solvable honeycomb model, a magnet that is frustrated not by geometry but by spin-orbit coupling. We map out the perturbations to this model that are relevant for the honeycomb iridates and microscopically derive the minimal spin model for these materials. We then study the diverse magnetic phases that are proximate to this spin liquid and discuss the experimental consequences for Na2IrO 3 and Li2IrO3.