Imputation of missing data in industrial databases

Research output: Contribution to journalArticlepeer-review

238 Scopus citations

Abstract

A limiting factor for the application of IDA methods in many domains is the incompleteness of data repositories. Many records have fields that are not filled in, especially, when data entry is manual. In addition, a significant fraction of the entries can be erroneous and there may be no alternative but to discard these records. But every cell in a database is not an independent datum. Statistical relationships will constrain and, often determine, missing values. Data imputation, the filling in of missing values for partially missing data, can thus be an invaluable first step in many IDA projects. New imputation methods that can handle the large-scale problems and large-scale sparsity of industrial databases are needed. To illustrate the incomplete database problem, we analyze one database with instrumentation maintenance and test records for an industrial process. Despite regulatory requirements for process data collection, this database is less than 50% complete. Next, we discuss possible solutions to the missing data problem. Several approaches to imputation are noted and classified into two categories: data-driven and model-based. We then describe two machine-learning-based approaches that we have worked with. These build upon well-known algorithms: AutoClass and C4.5. Several experiments are designed, all using the maintenance database as a common test-bed but with various data splits and algorithmic variations. Results are generally positive with up to 80% accuracies of imputation. We conclude the paper by outlining some considerations in selecting imputation methods, and by discussing applications of data imputation for intelligent data analysis.

Original languageEnglish (US)
Pages (from-to)259-275
Number of pages17
JournalApplied Intelligence
Volume11
Issue number3
DOIs
StatePublished - Nov 1999

Bibliographical note

Funding Information:
This work was funded by Honeywell Technology Center. Thanks to Brian Parsonnet of Honeywell HiSpec Solutions for insights on the data used in this research.

Fingerprint

Dive into the research topics of 'Imputation of missing data in industrial databases'. Together they form a unique fingerprint.

Cite this