Continuous decaying of telco big data with data postdiction

Constantinos Costa, Andreas Konstantinidis, Andreas Charalampous, Demetrios Zeinalipour-Yazti, Mohamed F Mokbel

Research output: Contribution to journalArticle

Abstract

In this paper, we present two novel decaying operators for Telco Big Data (TBD), coined TBD-DP and CTBD-DP that are founded on the notion of Data Postdiction. Unlike data prediction, which aims to make a statement about the future value of some tuple, our formulated data postdiction term, aims to make a statement about the past value of some tuple, which does not exist anymore as it had to be deleted to free up disk space. TBD-DP relies on existing Machine Learning (ML) algorithms to abstract TBD into compact models that can be stored and queried when necessary. Our proposed TBD-DP operator has the following two conceptual phases: (i) in an offline phase, it utilizes a LSTM-based hierarchical ML algorithm to learn a tree of models (coined TBD-DP tree) over time and space; (ii) in an online phase, it uses the TBD-DP tree to recover data within a certain accuracy. Additionally, we provide three decaying focus methods that can be plugged into the operators we propose, namely: (i) FIFO-amnesia, which is based on the time that the tuple was created; (ii) SPATIAL-amnesia, which is based on the cellular tower’s location related with the tuple; and (iii) UNIFORM-amnesia, which picks randomly the tuples to be decayed. Similarly, CTBD-DP enables the decaying of streaming data utilizing the TBD-DP tree to extend and update the stored models. In our experimental setup, we measure the efficiency of the proposed operator using a ∼10GB anonymized real telco network trace. Our experimental results in Tensorflow over HDFS are extremely encouraging as they show that TBD-DP saves an order of magnitude storage space while maintaining a high accuracy on the recovered data. Our experiments also show that CTBD-DP improves the accuracy over streaming data.

Original languageEnglish (US)
JournalGeoInformatica
DOIs
StatePublished - Jan 1 2019

Fingerprint

Learning algorithms
Learning systems
Big data
Towers
learning
Values
Experiments
efficiency
experiment
prediction
time
machine learning

Keywords

  • Data decaying
  • Data reduction
  • Machine learning
  • Spatio-temporal analytics
  • Telco big data

Cite this

Continuous decaying of telco big data with data postdiction. / Costa, Constantinos; Konstantinidis, Andreas; Charalampous, Andreas; Zeinalipour-Yazti, Demetrios; Mokbel, Mohamed F.

In: GeoInformatica, 01.01.2019.

Research output: Contribution to journalArticle

Costa, Constantinos ; Konstantinidis, Andreas ; Charalampous, Andreas ; Zeinalipour-Yazti, Demetrios ; Mokbel, Mohamed F. / Continuous decaying of telco big data with data postdiction. In: GeoInformatica. 2019.
@article{a49c99fe02734e2d8bf7887cc14b9ddc,
title = "Continuous decaying of telco big data with data postdiction",
abstract = "In this paper, we present two novel decaying operators for Telco Big Data (TBD), coined TBD-DP and CTBD-DP that are founded on the notion of Data Postdiction. Unlike data prediction, which aims to make a statement about the future value of some tuple, our formulated data postdiction term, aims to make a statement about the past value of some tuple, which does not exist anymore as it had to be deleted to free up disk space. TBD-DP relies on existing Machine Learning (ML) algorithms to abstract TBD into compact models that can be stored and queried when necessary. Our proposed TBD-DP operator has the following two conceptual phases: (i) in an offline phase, it utilizes a LSTM-based hierarchical ML algorithm to learn a tree of models (coined TBD-DP tree) over time and space; (ii) in an online phase, it uses the TBD-DP tree to recover data within a certain accuracy. Additionally, we provide three decaying focus methods that can be plugged into the operators we propose, namely: (i) FIFO-amnesia, which is based on the time that the tuple was created; (ii) SPATIAL-amnesia, which is based on the cellular tower’s location related with the tuple; and (iii) UNIFORM-amnesia, which picks randomly the tuples to be decayed. Similarly, CTBD-DP enables the decaying of streaming data utilizing the TBD-DP tree to extend and update the stored models. In our experimental setup, we measure the efficiency of the proposed operator using a ∼10GB anonymized real telco network trace. Our experimental results in Tensorflow over HDFS are extremely encouraging as they show that TBD-DP saves an order of magnitude storage space while maintaining a high accuracy on the recovered data. Our experiments also show that CTBD-DP improves the accuracy over streaming data.",
keywords = "Data decaying, Data reduction, Machine learning, Spatio-temporal analytics, Telco big data",
author = "Constantinos Costa and Andreas Konstantinidis and Andreas Charalampous and Demetrios Zeinalipour-Yazti and Mokbel, {Mohamed F}",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/s10707-019-00364-z",
language = "English (US)",
journal = "GeoInformatica",
issn = "1384-6175",
publisher = "Kluwer Academic Publishers",

}

TY - JOUR

T1 - Continuous decaying of telco big data with data postdiction

AU - Costa, Constantinos

AU - Konstantinidis, Andreas

AU - Charalampous, Andreas

AU - Zeinalipour-Yazti, Demetrios

AU - Mokbel, Mohamed F

PY - 2019/1/1

Y1 - 2019/1/1

N2 - In this paper, we present two novel decaying operators for Telco Big Data (TBD), coined TBD-DP and CTBD-DP that are founded on the notion of Data Postdiction. Unlike data prediction, which aims to make a statement about the future value of some tuple, our formulated data postdiction term, aims to make a statement about the past value of some tuple, which does not exist anymore as it had to be deleted to free up disk space. TBD-DP relies on existing Machine Learning (ML) algorithms to abstract TBD into compact models that can be stored and queried when necessary. Our proposed TBD-DP operator has the following two conceptual phases: (i) in an offline phase, it utilizes a LSTM-based hierarchical ML algorithm to learn a tree of models (coined TBD-DP tree) over time and space; (ii) in an online phase, it uses the TBD-DP tree to recover data within a certain accuracy. Additionally, we provide three decaying focus methods that can be plugged into the operators we propose, namely: (i) FIFO-amnesia, which is based on the time that the tuple was created; (ii) SPATIAL-amnesia, which is based on the cellular tower’s location related with the tuple; and (iii) UNIFORM-amnesia, which picks randomly the tuples to be decayed. Similarly, CTBD-DP enables the decaying of streaming data utilizing the TBD-DP tree to extend and update the stored models. In our experimental setup, we measure the efficiency of the proposed operator using a ∼10GB anonymized real telco network trace. Our experimental results in Tensorflow over HDFS are extremely encouraging as they show that TBD-DP saves an order of magnitude storage space while maintaining a high accuracy on the recovered data. Our experiments also show that CTBD-DP improves the accuracy over streaming data.

AB - In this paper, we present two novel decaying operators for Telco Big Data (TBD), coined TBD-DP and CTBD-DP that are founded on the notion of Data Postdiction. Unlike data prediction, which aims to make a statement about the future value of some tuple, our formulated data postdiction term, aims to make a statement about the past value of some tuple, which does not exist anymore as it had to be deleted to free up disk space. TBD-DP relies on existing Machine Learning (ML) algorithms to abstract TBD into compact models that can be stored and queried when necessary. Our proposed TBD-DP operator has the following two conceptual phases: (i) in an offline phase, it utilizes a LSTM-based hierarchical ML algorithm to learn a tree of models (coined TBD-DP tree) over time and space; (ii) in an online phase, it uses the TBD-DP tree to recover data within a certain accuracy. Additionally, we provide three decaying focus methods that can be plugged into the operators we propose, namely: (i) FIFO-amnesia, which is based on the time that the tuple was created; (ii) SPATIAL-amnesia, which is based on the cellular tower’s location related with the tuple; and (iii) UNIFORM-amnesia, which picks randomly the tuples to be decayed. Similarly, CTBD-DP enables the decaying of streaming data utilizing the TBD-DP tree to extend and update the stored models. In our experimental setup, we measure the efficiency of the proposed operator using a ∼10GB anonymized real telco network trace. Our experimental results in Tensorflow over HDFS are extremely encouraging as they show that TBD-DP saves an order of magnitude storage space while maintaining a high accuracy on the recovered data. Our experiments also show that CTBD-DP improves the accuracy over streaming data.

KW - Data decaying

KW - Data reduction

KW - Machine learning

KW - Spatio-temporal analytics

KW - Telco big data

UR - http://www.scopus.com/inward/record.url?scp=85068127112&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068127112&partnerID=8YFLogxK

U2 - 10.1007/s10707-019-00364-z

DO - 10.1007/s10707-019-00364-z

M3 - Article

AN - SCOPUS:85068127112

JO - GeoInformatica

JF - GeoInformatica

SN - 1384-6175

ER -