Unsupervised learning based distributed detection of global anomalies

Junlin Zhou, Aleksandar Lazarevic, Kuo Wei Hsu, Jaideep Srivastava, Yan Fu, Yue Wu

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

Anomaly detection has recently become an important problem in many industrial and financial applications. Very often, the databases from which anomalies have to be found are located at multiple local sites and cannot be merged due to privacy reasons or communication overhead. In this paper, a novel general framework for distributed anomaly detection is proposed. The proposed method consists of three steps: (i) building local models for distributed data sources with unsupervised anomaly detection methods and computing quality measure of local models; (ii) transforming local unsupervised local models into sharing models; and (iii) reusing sharing models for new data and combining their results by considering both quality and diversity of them to detect anomalies in a global view. In experiments performed on synthetic and real-life large data set, the proposed distributed anomaly detection method achieved prediction performance comparable or even slightly better than the global anomaly detection algorithm applied on the data set obtained when all distributed data set were merged.

Original languageEnglish (US)
Pages (from-to)935-957
Number of pages23
JournalInternational Journal of Information Technology and Decision Making
Volume9
Issue number6
DOIs
StatePublished - Nov 2010

Bibliographical note

Funding Information:
The work was supported by NASA under award NNX08AC36A, by Natural Science Foundation of China (Nos. 60973120 and 60903073) and by the National High-Tech Research and Development Plan of China under Grant No. 2007AA01Z440.

Keywords

  • Distributed anomaly detection
  • combining models
  • global anomalies

Fingerprint

Dive into the research topics of 'Unsupervised learning based distributed detection of global anomalies'. Together they form a unique fingerprint.

Cite this