Signal estimation and denoising using VC-theory

V. Cherkassky, X. Shao

Research output: Contribution to journalArticlepeer-review

28 Scopus citations


Signal denoising is closely related to function estimation from noisy samples. The same problem is also addressed in statistics (non-linear regression) and neural network learning. Vapnik-Chervonenkis (VC) theory has recently emerged as a general theory for estimation of dependencies from finite samples. This theory emphasizes model complexity control according to Structural Risk Minimization (SRM) inductive principle, which considers a nested set of models of increasing complexity (called a structure), and then selects an optimal model complexity providing minimum error for future samples. This paper applies the framework of VC-theory to signal estimation/denoising. There are three factors important for accurate signal estimation from finite samples: (1) the type of (orthogonal) basis functions used (i.e. Fourier basis, wavelets etc.). (2) The choice of a structure, i.e. ordering of the basis functions according to their 'importance' for accurate signal estimation. This corresponds to the choice of a 'structure' under SRM formulation. (3) Selecting an optimal number of terms (basis functions) from the ordered sequence of basis functions (2), aka model selection or complexity control (in statistics).We propose a methodology for specifying appropriate orderings (2) and an analytic expression for model selection (3) for signal processing applications. We also present empirical comparisons between the proposed methodology and current state-of-the-art wavelet thresholding methods for univariate signals. These comparisons suggest that the prudent choice of a structure (2) and the use of VC-based model selection (3) are critical for accurate signal estimation with finite samples.

Original languageEnglish (US)
Pages (from-to)37-52
Number of pages16
JournalNeural Networks
Issue number1
StatePublished - Jan 2001


  • Complexity control
  • Model selection
  • Orthonormal basis
  • Signal denoising
  • Sparse codes
  • Structural risk minimization
  • VC-theory
  • Wavelet thresholding
  • Wavelets


Dive into the research topics of 'Signal estimation and denoising using VC-theory'. Together they form a unique fingerprint.

Cite this