Accurate imputation and prediction of load data are important prerequisites for many tasks of power systems, especially as renewables and plug-in electric vehicles penetrate the grid. A low-rank and sparse matrix factorization model is considered for load inference tasks to capture spatial as well as temporal structures in multi-site load data. The low-rank structure captures periodic patterns, and sparse matrix factors explain localized and clustered signatures. In order to predict load values for future time instants (and possibly for unforeseen sites), prior knowledge on correlations is necessarily incorporated in a nonparametric kernel-based learning framework. An efficient learning algorithm is also derived. Tests with real load data verify the efficacy of the proposed approach.