A consistent information criterion for support vector machines in diverging model spaces

Xiang Zhang, Yichao Wu, Lan Wang, Runze Li

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

Information criteria have been popularly used in model selection and proved to possess nice theoretical properties. For classification, Claeskens et al. (2008) proposed support vector machine information criterion for feature selection and provided encouraging numerical evidence. Yet no theoretical justification was given there. This work aims to fill the gap and to provide some theoretical justifications for support vector machine information criterion in both fixed and diverging model spaces. We first derive a uniform convergence rate for the support vector machine solution and then show that a modification of the support vector machine information criterion achieves model selection consistency even when the number of features diverges at an exponential rate of the sample size. This consistency result can be further applied to selecting the optimal tuning parameter for various penalized support vector machine methods. Finite-sample performance of the proposed information criterion is investigated using Monte Carlo studies and one real-world gene selection problem.

Original languageEnglish (US)
JournalJournal of Machine Learning Research
Volume17
StatePublished - Apr 1 2016

Fingerprint

Information Criterion
Support vector machines
Support Vector Machine
Model Selection
Justification
Gene Selection
Model
Parameter Tuning
Monte Carlo Study
Uniform convergence
Diverge
Feature Selection
Convergence Rate
Feature extraction
Sample Size
Tuning
Genes

Keywords

  • Bayesian information criterion
  • Diverging model spaces
  • Feature selection
  • Support vector machines

Cite this

A consistent information criterion for support vector machines in diverging model spaces. / Zhang, Xiang; Wu, Yichao; Wang, Lan; Li, Runze.

In: Journal of Machine Learning Research, Vol. 17, 01.04.2016.

Research output: Contribution to journalArticle

@article{1306ed4695a84a1a855445193a178427,
title = "A consistent information criterion for support vector machines in diverging model spaces",
abstract = "Information criteria have been popularly used in model selection and proved to possess nice theoretical properties. For classification, Claeskens et al. (2008) proposed support vector machine information criterion for feature selection and provided encouraging numerical evidence. Yet no theoretical justification was given there. This work aims to fill the gap and to provide some theoretical justifications for support vector machine information criterion in both fixed and diverging model spaces. We first derive a uniform convergence rate for the support vector machine solution and then show that a modification of the support vector machine information criterion achieves model selection consistency even when the number of features diverges at an exponential rate of the sample size. This consistency result can be further applied to selecting the optimal tuning parameter for various penalized support vector machine methods. Finite-sample performance of the proposed information criterion is investigated using Monte Carlo studies and one real-world gene selection problem.",
keywords = "Bayesian information criterion, Diverging model spaces, Feature selection, Support vector machines",
author = "Xiang Zhang and Yichao Wu and Lan Wang and Runze Li",
year = "2016",
month = "4",
day = "1",
language = "English (US)",
volume = "17",
journal = "Journal of Machine Learning Research",
issn = "1532-4435",
publisher = "Microtome Publishing",

}

TY - JOUR

T1 - A consistent information criterion for support vector machines in diverging model spaces

AU - Zhang, Xiang

AU - Wu, Yichao

AU - Wang, Lan

AU - Li, Runze

PY - 2016/4/1

Y1 - 2016/4/1

N2 - Information criteria have been popularly used in model selection and proved to possess nice theoretical properties. For classification, Claeskens et al. (2008) proposed support vector machine information criterion for feature selection and provided encouraging numerical evidence. Yet no theoretical justification was given there. This work aims to fill the gap and to provide some theoretical justifications for support vector machine information criterion in both fixed and diverging model spaces. We first derive a uniform convergence rate for the support vector machine solution and then show that a modification of the support vector machine information criterion achieves model selection consistency even when the number of features diverges at an exponential rate of the sample size. This consistency result can be further applied to selecting the optimal tuning parameter for various penalized support vector machine methods. Finite-sample performance of the proposed information criterion is investigated using Monte Carlo studies and one real-world gene selection problem.

AB - Information criteria have been popularly used in model selection and proved to possess nice theoretical properties. For classification, Claeskens et al. (2008) proposed support vector machine information criterion for feature selection and provided encouraging numerical evidence. Yet no theoretical justification was given there. This work aims to fill the gap and to provide some theoretical justifications for support vector machine information criterion in both fixed and diverging model spaces. We first derive a uniform convergence rate for the support vector machine solution and then show that a modification of the support vector machine information criterion achieves model selection consistency even when the number of features diverges at an exponential rate of the sample size. This consistency result can be further applied to selecting the optimal tuning parameter for various penalized support vector machine methods. Finite-sample performance of the proposed information criterion is investigated using Monte Carlo studies and one real-world gene selection problem.

KW - Bayesian information criterion

KW - Diverging model spaces

KW - Feature selection

KW - Support vector machines

UR - http://www.scopus.com/inward/record.url?scp=84979913354&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84979913354&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:84979913354

VL - 17

JO - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

SN - 1532-4435

ER -