The Focusing Optics X-ray Solar Imager (FOXSI) sounding rocket experiment conducts direct imaging and spectral observation of the Sun in hard X-rays, in the energy range 4 to 20 keV. These high-sensitivity observations are used to study particle acceleration and coronal heating. FOXSI is designed with seven grazing incidence optics modules that focus X-rays onto seven focal plane detectors kept at a 2m distance. FOXSI-1 was flown with seven Double-sided Si Strip Detectors (DSSD), and two of them were replaced with CdTe detectors for FOXSI-2. The upcoming FOXSI-3 flight will carry DSSD and CdTe detectors with upgraded optics for enhanced sensitivity. The detectors are calibrated using various radioactive sources. The detector's spectral response matrix was constructed with diagonal elements using a Gaussian approximation with a spread (sigma) that accounts for the energy resolution of the detector. Spectroscopic studies of past FOXSI flight data suggest that the inclusion of lower energy X-rays could better constrain the spectral modeling to yield a more precise temperature estimation of the hot plasma. This motivates us to carry out an improved calibration to better understand the finer-order effects on the spectral response, especially at lower energies. Here we report our improved calibration of FOXSI detectors using experiments and Monte-Carlo simulations.