Climatic conditions shift gradually over millennia, altering the rates at which carbon (C) is fixed from the atmosphere and stored in the soil. However, legacy impacts of past climates on current soil C stocks are poorly understood. We used data from more than 5000 terrestrial sites from three global and regional data sets to identify the relative importance of current and past (Last Glacial Maximum and mid-Holocene) climatic conditions in regulating soil C stocks in natural and agricultural areas. Paleoclimate always explained a greater amount of the variance in soil C stocks than current climate at regional and global scales. Our results indicate that climatic legacies help determine global soil C stocks in terrestrial ecosystems where agriculture is highly dependent on current climatic conditions. Our findings emphasize the importance of considering how climate legacies influence soil C content, allowing us to improve quantitative predictions of global C stocks under different climatic scenarios.