Instabilities of the Fermi-liquid and staggered flux phases in the large-N t-J model

We study the stability of the Fermi-liquid and staggered flux phases in the t-J model for densities off half-filling using a large-N slave-boson approach. We examine the stability of the Fermi-liquid phase with respect to flux-density-wave (FDW) and bond-density-wave (BDW), or spin-Peierls, fluctuations at arbitrary wave vector as a function of doping and the ratio t/J. For densities near half-filling the Fermi liquid becomes unstable below a critical tc towards an FDW state with an incommensurate wave vector k near (,). The critical ratio tc/J for the Fermi-liquid FDW transition diverges rapidly as 0. For >18%, the FDW instability of the Fermi liquid is preempted by a BDW instability at incommensurate k. We examine the stability of the 2 × 2 staggered flux phase with respect to coupled FDW and BDW fluctuations and find that, upon lowering t/J at small, the staggered flux phase remains stable for t/J above a nonzero critical ratio and then develops an instability of primarily BDW character. At nonzero and still lower t/J, the staggered flux phase also develops an instability with respect to long-wavelength FDW fluctuations; tc/J for this instability is approximately independent of. We study the response function for such fluctuations analytically, showing the nontrivial k dependence induced by the presence of a small density of holes in the staggered phase. It is conjectured that this long-wavelength FDW instability is associated with the existence off half-filling of a commensurate flux phase with lower energy than the staggered flux phase.