A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise

Xi C. Chen, Yuanshun Yao, Sichao Shi, Snigdhansu Chatterjee, Vipin Kumar, James H. Faghmous

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The autonomous identification of time-steps where the behavior of a time-series significantly deviates from a predefined model, or time-series change point detection, is an active field of research with notable applications in finance, health, and advertising. One family of time-series change detection algorithms, referred to as "model-based methods", although useful for many applications, performs poor when the data are noisy and have outliers. We introduce a new framework that enables existing model-based methods to be more robust to these data challenges. We demonstrate the effectiveness of our approach on remote sensing and mobile health data. Our method introduces two new concepts: (i) a random sampling procedure allows us to overcome outliers, and (ii) a matrix-based representation of anomaly scores provides a flexible and intuitive way to identify multiple types of changes and test their significance. We show that our method performs better than several baseline methods, including application-specific algorithms, and provide all data and open-source code.

Original languageEnglish (US)
Title of host publication16th SIAM International Conference on Data Mining 2016, SDM 2016
EditorsSanjay Chawla Venkatasubramanian, Wagner Meira
PublisherSociety for Industrial and Applied Mathematics Publications
Pages162-170
Number of pages9
ISBN (Electronic)9781510828117
StatePublished - Jan 1 2016
Event16th SIAM International Conference on Data Mining 2016, SDM 2016 - Miami, United States
Duration: May 5 2016May 7 2016

Publication series

Name16th SIAM International Conference on Data Mining 2016, SDM 2016

Other

Other16th SIAM International Conference on Data Mining 2016, SDM 2016
CountryUnited States
CityMiami
Period5/5/165/7/16

Fingerprint

Time series
Finance
Marketing
Remote sensing
Health
Sampling
mHealth

Cite this

Chen, X. C., Yao, Y., Shi, S., Chatterjee, S., Kumar, V., & Faghmous, J. H. (2016). A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise. In S. C. Venkatasubramanian, & W. Meira (Eds.), 16th SIAM International Conference on Data Mining 2016, SDM 2016 (pp. 162-170). (16th SIAM International Conference on Data Mining 2016, SDM 2016). Society for Industrial and Applied Mathematics Publications.

A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise. / Chen, Xi C.; Yao, Yuanshun; Shi, Sichao; Chatterjee, Snigdhansu; Kumar, Vipin; Faghmous, James H.

16th SIAM International Conference on Data Mining 2016, SDM 2016. ed. / Sanjay Chawla Venkatasubramanian; Wagner Meira. Society for Industrial and Applied Mathematics Publications, 2016. p. 162-170 (16th SIAM International Conference on Data Mining 2016, SDM 2016).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Chen, XC, Yao, Y, Shi, S, Chatterjee, S, Kumar, V & Faghmous, JH 2016, A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise. in SC Venkatasubramanian & W Meira (eds), 16th SIAM International Conference on Data Mining 2016, SDM 2016. 16th SIAM International Conference on Data Mining 2016, SDM 2016, Society for Industrial and Applied Mathematics Publications, pp. 162-170, 16th SIAM International Conference on Data Mining 2016, SDM 2016, Miami, United States, 5/5/16.
Chen XC, Yao Y, Shi S, Chatterjee S, Kumar V, Faghmous JH. A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise. In Venkatasubramanian SC, Meira W, editors, 16th SIAM International Conference on Data Mining 2016, SDM 2016. Society for Industrial and Applied Mathematics Publications. 2016. p. 162-170. (16th SIAM International Conference on Data Mining 2016, SDM 2016).
Chen, Xi C. ; Yao, Yuanshun ; Shi, Sichao ; Chatterjee, Snigdhansu ; Kumar, Vipin ; Faghmous, James H. / A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise. 16th SIAM International Conference on Data Mining 2016, SDM 2016. editor / Sanjay Chawla Venkatasubramanian ; Wagner Meira. Society for Industrial and Applied Mathematics Publications, 2016. pp. 162-170 (16th SIAM International Conference on Data Mining 2016, SDM 2016).
@inproceedings{e4d9312a10cb4242813523b955cd7cf8,
title = "A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise",
abstract = "The autonomous identification of time-steps where the behavior of a time-series significantly deviates from a predefined model, or time-series change point detection, is an active field of research with notable applications in finance, health, and advertising. One family of time-series change detection algorithms, referred to as {"}model-based methods{"}, although useful for many applications, performs poor when the data are noisy and have outliers. We introduce a new framework that enables existing model-based methods to be more robust to these data challenges. We demonstrate the effectiveness of our approach on remote sensing and mobile health data. Our method introduces two new concepts: (i) a random sampling procedure allows us to overcome outliers, and (ii) a matrix-based representation of anomaly scores provides a flexible and intuitive way to identify multiple types of changes and test their significance. We show that our method performs better than several baseline methods, including application-specific algorithms, and provide all data and open-source code.",
author = "Chen, {Xi C.} and Yuanshun Yao and Sichao Shi and Snigdhansu Chatterjee and Vipin Kumar and Faghmous, {James H.}",
year = "2016",
month = "1",
day = "1",
language = "English (US)",
series = "16th SIAM International Conference on Data Mining 2016, SDM 2016",
publisher = "Society for Industrial and Applied Mathematics Publications",
pages = "162--170",
editor = "Venkatasubramanian, {Sanjay Chawla} and Wagner Meira",
booktitle = "16th SIAM International Conference on Data Mining 2016, SDM 2016",

}

TY - GEN

T1 - A general framework to increase the robustness of model-based change point detection algorithms to outliers and noise

AU - Chen, Xi C.

AU - Yao, Yuanshun

AU - Shi, Sichao

AU - Chatterjee, Snigdhansu

AU - Kumar, Vipin

AU - Faghmous, James H.

PY - 2016/1/1

Y1 - 2016/1/1

N2 - The autonomous identification of time-steps where the behavior of a time-series significantly deviates from a predefined model, or time-series change point detection, is an active field of research with notable applications in finance, health, and advertising. One family of time-series change detection algorithms, referred to as "model-based methods", although useful for many applications, performs poor when the data are noisy and have outliers. We introduce a new framework that enables existing model-based methods to be more robust to these data challenges. We demonstrate the effectiveness of our approach on remote sensing and mobile health data. Our method introduces two new concepts: (i) a random sampling procedure allows us to overcome outliers, and (ii) a matrix-based representation of anomaly scores provides a flexible and intuitive way to identify multiple types of changes and test their significance. We show that our method performs better than several baseline methods, including application-specific algorithms, and provide all data and open-source code.

AB - The autonomous identification of time-steps where the behavior of a time-series significantly deviates from a predefined model, or time-series change point detection, is an active field of research with notable applications in finance, health, and advertising. One family of time-series change detection algorithms, referred to as "model-based methods", although useful for many applications, performs poor when the data are noisy and have outliers. We introduce a new framework that enables existing model-based methods to be more robust to these data challenges. We demonstrate the effectiveness of our approach on remote sensing and mobile health data. Our method introduces two new concepts: (i) a random sampling procedure allows us to overcome outliers, and (ii) a matrix-based representation of anomaly scores provides a flexible and intuitive way to identify multiple types of changes and test their significance. We show that our method performs better than several baseline methods, including application-specific algorithms, and provide all data and open-source code.

UR - http://www.scopus.com/inward/record.url?scp=84991694169&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84991694169&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84991694169

T3 - 16th SIAM International Conference on Data Mining 2016, SDM 2016

SP - 162

EP - 170

BT - 16th SIAM International Conference on Data Mining 2016, SDM 2016

A2 - Venkatasubramanian, Sanjay Chawla

A2 - Meira, Wagner

PB - Society for Industrial and Applied Mathematics Publications

ER -