For a self-gravitating viscoelastic compressible sphere we have shown that unstable modes can exist by means of the linear viscoelastic theory by both initial-value and normal-mode approaches. For a uniform sphere we have derived analytical expressions for the roots of the secular determinant based on the asymptotic expansion of the spherical Bessel functions. From the two expressions, both the destabilizing nature of gravitational forces and the stabilizing influences of increasing elastic strength are revealed. Fastest growth times on the order of ten thousand years are developed for the longest wavelength. In contrast, a self-gravitating incompressible viscoelastic model is found to be stable. This result of linear approximation suggests that a more general approach, e.g., non-Maxwellian rheology or a non-linear finite-amplitude theory, should be considered in global geodynamics.