On main-memory flushing in microblogs data management systems

Amr Magdy, Rami Alghamdi, Mohamed F. Mokbel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations

Abstract

Searching microblogs, e.g., tweets and comments, is practically supported through main-memory indexing for scalable data digestion and efficient query evaluation. With continuity and excessive numbers of microblogs, it is infeasible to keep data in main-memory for long periods. Thus, once allocated memory budget is filled, a portion of data is flushed from memory to disk to continuously accommodate newly incoming data. Existing techniques come with either low memory hit ratio due to flushing items regardless of their relevance to incoming queries or significant overhead of tracking individual data items, which limit scalability of microblogs systems in either cases. In this paper, we propose kFlushing policy that exploits popularity of top-k queries in microblogs to smartly select a subset of microblogs to flush. kFlushing is mainly designed to increase memory hit ratio. To this end, it identifies and flushes in-memory data that does not contribute to incoming queries. The freed memory space is utilized to accumulate more useful data that is used to answer more queries from memory contents. When all memory is utilized for useful data, kFlushing flushes data that is less likely to degrade memory hit ratio. In addition, kFlushing comes with a little overhead that keeps high system scalability in terms of high digestion rates of incoming fast data. Extensive experimental evaluation shows the effectiveness and scalability of kFlushing to improve main-memory hit by 26-330% while coping up with fast microblog streams of up to 100K microblog/second.

Original languageEnglish (US)
Title of host publication2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages445-456
Number of pages12
ISBN (Electronic)9781509020195
DOIs
StatePublished - Jun 22 2016
Event32nd IEEE International Conference on Data Engineering, ICDE 2016 - Helsinki, Finland
Duration: May 16 2016May 20 2016

Publication series

Name2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016

Other

Other32nd IEEE International Conference on Data Engineering, ICDE 2016
CountryFinland
CityHelsinki
Period5/16/165/20/16

Fingerprint Dive into the research topics of 'On main-memory flushing in microblogs data management systems'. Together they form a unique fingerprint.

  • Cite this

    Magdy, A., Alghamdi, R., & Mokbel, M. F. (2016). On main-memory flushing in microblogs data management systems. In 2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016 (pp. 445-456). [7498261] (2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDE.2016.7498261