Challenges of Large-Scale Data Processing in the 1990s: The IPUMS Experience

Diana L Magnuson, Steven Ruggles

Research output: Contribution to journalArticlepeer-review


When it was launched in 1991, the Integrated Public Use Microdata Series (IPUMS) project faced a challenging environment and limited resources. Few datasets were interoperable and much data collected at great public expense was inaccessible to most researchers. Documentation of datasets was nonstandardized, incomplete, and inadequate for automated processing. With insufficient attention to preservation, valuable scientific data were disappearing (see Bogue et al., 1976). IPUMS was established to address these critical issues. At the outset, IPUMS faced daunting barriers of inadequate data processing, storage, and network capacity. This anecdote describes the improvised computational infrastructure developed in the decade from 1989 to 1999 to process, manage, and disseminate the world's largest population datasets. We use a combination of archival sources, interviews, and our own memories to trace the development of the IPUMS computing environment during a period of explosive technical innovation. The development of IPUMS is part of a larger story of the development of social science infrastructure in the late 20th century and its contribution to democratizing data access.

Original languageEnglish (US)
Pages (from-to)71-83
Number of pages13
JournalIEEE Annals of the History of Computing
Issue number4
StatePublished - Oct 1 2022

Bibliographical note

Publisher Copyright:
© 2022 IEEE.


Dive into the research topics of 'Challenges of Large-Scale Data Processing in the 1990s: The IPUMS Experience'. Together they form a unique fingerprint.

Cite this