Streaming data reduction using low-memory factored representations

David Littau, Daniel L Boley

Research output: Contribution to journalArticlepeer-review

6 Scopus citations

Abstract

Many special purpose algorithms exist for extracting information from streaming data. Constraints are imposed on the total memory and on the average processing time per data item. These constraints are usually satisfied by deciding in advance the kind of information one wishes to extract, and then extracting only the data relevant for that goal. Here, we propose a general data representation that can be computed using modest memory requirements with limited processing power per data item, and yet permits the application of an arbitrary data mining algorithm chosen and/or adjusted after the data collection process has begun. The new representation allows for the at-once analysis of a significantly larger number of data items than would be possible using the original representation of the data. The method depends on a rapid computation of a factored form of the original data set. The method is illustrated with two real datasets, one with dense and one with sparse attribute values.

Original languageEnglish (US)
Pages (from-to)2016-2041
Number of pages26
JournalInformation Sciences
Volume176
Issue number14
DOIs
StatePublished - Jul 22 2006

Keywords

  • Clustering
  • Data reduction
  • Matrix approximation
  • PDDP
  • Streaming data

Fingerprint Dive into the research topics of 'Streaming data reduction using low-memory factored representations'. Together they form a unique fingerprint.

Cite this