TY - JOUR
T1 - A permutation test approach to the choice of size k for the nearest neighbors classifier
AU - Lai, Yinglei
AU - Wu, Baolin
AU - Zhao, Hongyu
PY - 2011/10/1
Y1 - 2011/10/1
N2 - The k nearest neighbors (k-NN) classifier is one of the most popular methods for statistical pattern recognition and machine learning. In practice, the size k, the number of neighbors used for classification, is usually arbitrarily set to one or some other small numbers, or based on the cross-validation procedure. In this study,we propose a novel alternative approach to decide the size k. Based on a k-NN-based multivariate multi-sample test, we assign each k a permutation test based Z-score. The number of NN is set to the k with the highest Z-score. This approach is computationally efficient since we have derived the formulas for the mean and variance of the test statistic under permutation distribution for multiple sample groups. Several simulation and real-world data sets are analyzed to investigate the performance of our approach. The usefulness of our approach is demonstrated through the evaluation of prediction accuracies using Zscore as a criterion to select the size k.We also compare our approach to the widely used cross-validation approaches. The results show that the size k selected by our approach yields high prediction accuracies when informative features are used for classification, whereas the cross-validation approach may fail in some cases.
AB - The k nearest neighbors (k-NN) classifier is one of the most popular methods for statistical pattern recognition and machine learning. In practice, the size k, the number of neighbors used for classification, is usually arbitrarily set to one or some other small numbers, or based on the cross-validation procedure. In this study,we propose a novel alternative approach to decide the size k. Based on a k-NN-based multivariate multi-sample test, we assign each k a permutation test based Z-score. The number of NN is set to the k with the highest Z-score. This approach is computationally efficient since we have derived the formulas for the mean and variance of the test statistic under permutation distribution for multiple sample groups. Several simulation and real-world data sets are analyzed to investigate the performance of our approach. The usefulness of our approach is demonstrated through the evaluation of prediction accuracies using Zscore as a criterion to select the size k.We also compare our approach to the widely used cross-validation approaches. The results show that the size k selected by our approach yields high prediction accuracies when informative features are used for classification, whereas the cross-validation approach may fail in some cases.
KW - Cross-validation
KW - Nearest neighbors classifier
KW - Number of neighbors
KW - Permutation test
KW - Prediction accuracy
UR - http://www.scopus.com/inward/record.url?scp=84860418931&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84860418931&partnerID=8YFLogxK
U2 - 10.1080/02664763.2010.547565
DO - 10.1080/02664763.2010.547565
M3 - Article
AN - SCOPUS:84860418931
VL - 38
SP - 2289
EP - 2302
JO - Journal of Applied Statistics
JF - Journal of Applied Statistics
SN - 0266-4763
IS - 10
ER -