A second order accurate finite difference scheme is proposed for multidimensional radiation hydrodynamical equations in a diffusion limit. The radiation hydrodynamical equations in the limit constitute a hyperbolic system of conservation laws plus radiative heat transfer. A Godunov scheme including linear and nonlinear Riemann solvers is proposed for the set of conservation laws. The scheme with the linear Riemann solver works well for relatively strong shocks. The nonlinear Riemann solver is specially designed for flows involving strong shocks. The radiative heat conduction is treated implicitly. The treatment possesses a number of advantages over typical implicit methods. The most notable are the second order accuracy in both space and time, quick damping of numerical errors when the size of time steps is large, iterative solver and the fast convergence, the accurate treatment for the nonlinearity, and the energy conservation. Numerical examples are given to show the features of the schemes.