Quantum anomalous Hall insulator stabilized by competing interactions

We study the quantum phases driven by interaction in a semimetal with a quadratic band touching at the Fermi level. By combining the density matrix renormalization group (DMRG), analytical power expanded Gibbs potential method, and the weak coupling renormalization group, we study a spinless fermion system on a checkerboard lattice at half-filling, which has a quadratic band touching in the absence of interaction. In the presence of strong nearest-neighbor (V1) and next-nearest-neighbor (V2) interactions, we identify a site nematic insulator phase, a stripe insulator phase, and a phase separation region, in agreement with the phase diagram obtained analytically in the strong coupling limit (i.e., in the absence of fermion hopping). In the intermediate interaction regime we establish a quantum anomalous Hall phase in the DMRG as evidenced by the spontaneous time-reversal symmetry breaking and the appearance of a quantized Chern number C=1. For weak interaction we utilize the power expanded Gibbs potential method that treats V1 and V2 on equal footing, as well as the weak coupling renormalization group. Our analytical results reveal that not only the repulsive V1 interaction, but also the V2 interaction (both repulsive and attractive), can drive the quantum anomalous Hall phase. We also determine the phase boundary in the V1-V2 plane that separates the semimetal from the quantum anomalous Hall state. Finally, we show that the nematic semimetal, which was proposed for |V2| V1 at weak coupling in a previous study, is absent, and the quantum anomalous Hall state is the only weak coupling instability of the spinless quadratic band touching semimetal.