In this paper, a dynamic subgrid scale (SGS) stress model is proposed by modifying the existing quadratic nonlinear model. Unlike the conventional eddy viscosity models, the nonlinear model suggests an anisotropic tensorial polynomial relation between the SGS stress and the resolved strain rate tensor. Examined in the rotating turbulent channel flow, the previous nonlinear model is found unable to successfully predict the turbulent kinetic energy and the Reynolds shear stress. In the previous nonlinear model, an excessive backward energy transfer from the SGS to the resolved scale is found, so it is removed in the new model to provide sufficient SGS dissipation. The dynamic method is reconsidered, based on the analysis of the transport equation of the resolved Reynolds shear stress. The new dynamic nonlinear model is examined in the rotating turbulent channel flow at Re = 7000 and various rotation numbers (Ro), ranging from 0.0 to 0.6. The new dynamic procedure determines a more proper model coefficient for the nonlinear term, so the modified nonlinear model predicts superior results to the previous nonlinear models. The Vreman's approach of the eddy viscosity is implemented into the nonlinear model, to compare with the conventional dynamic Smagorinsky type nonlinear model in the rotating turbulent channel flow.