Optimal Spectral Shrinkage and PCA with Heteroscedastic Noise

William Leeb, Elad Romanov

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

This paper studies the related problems of prediction, covariance estimation, and principal component analysis for the spiked covariance model with heteroscedastic noise. We consider an estimator of the principal components based on whitening the noise, and we derive optimal singular value and eigenvalue shrinkers for use with these estimated principal components. Underlying these methods are new asymptotic results for the high-dimensional spiked model with heteroscedastic noise, and consistent estimators for the relevant population parameters. We extend previous analysis on out-of-sample prediction to the setting of predictors with whitening. We demonstrate certain advantages of noise whitening. Specifically, we show that in a certain asymptotic regime, optimal singular value shrinkage with whitening converges to the best linear predictor, whereas without whitening it converges to a suboptimal linear predictor. We prove that for generic signals, whitening improves estimation of the principal components, and increases a natural signal-to-noise ratio of the observations. We also show that for rank one signals, our estimated principal components achieve the asymptotic minimax rate.

Original languageEnglish (US)
Article number9336680
Pages (from-to)3009-3037
Number of pages29
JournalIEEE Transactions on Information Theory
Volume67
Issue number5
DOIs
StatePublished - May 2021

Bibliographical note

Funding Information:
Manuscript received November 5, 2019; revised August 15, 2020; accepted December 26, 2020. Date of publication January 27, 2021; date of current version April 21, 2021. This work was supported in part by the Simons Foundation Collaboration on Algorithms and Geometry, in part by the NSF BIGDATA program under Grant IIS 1837992, in part by the BSF under Grant 2018230, in part by the ISF under Grant 1523/16 and Grant 1791/17, and in part by the Einstein-Kaye fellowship from the Hebrew University of Jerusalem. (Corresponding author: William Leeb.) William Leeb is with the School of Mathematics, University of Minnesota, Minneapolis, MN 55455 USA (e-mail: wleeb@umn.edu).

Publisher Copyright:
© 1963-2012 IEEE.

Keywords

  • Singular value shrinkage
  • covariance estimation
  • eigenvalue shrinkage
  • heteroscedastic noise
  • matrix denoising
  • principal component analysis

Fingerprint

Dive into the research topics of 'Optimal Spectral Shrinkage and PCA with Heteroscedastic Noise'. Together they form a unique fingerprint.

Cite this