Risk intelligence: Profitting from uncertainty in data processing system

Si Zheng, Yunhuai Liu, Shanshan Li, Tian He, Xiangke Liao

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Fault-tolerance is essential in extreme-scale data processing systems. Pro-active fault-tolerance scheme (such as the speculative execution in MapReduce framework), can dramatically improve the response time of job executions when the failure becomes norm rather than an exception. Efficient pro-active fault-tolerance schemes require precise knowledge on the task executions, which has been an open challenges for decades. To well address the issue, in this paper we design and implement RiskI, a profile-based prediction algorithm in conjunction with a risk-aware task assignment algorithm to accelerate task executions, taking the uncertainty nature of tasks into account. Our design demonstrates that the nature uncertain not only brings great challenges but also new opportunities. With a careful design, we can benefit from such uncertainties. We implement the idea in Hadoop 0.21.0 systems and the experimental results show that compared with the traditional LATE algorithm, the response time can be improved by 46% with the same system throughput.

Original languageEnglish (US)
Title of host publicationProceedings
Subtitle of host publicationInternational Conference on Parallel Processing - The 42nd Annual Conference, ICPP 2013
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages10
ISBN (Print)9780769551173
StatePublished - 2013
Event42nd Annual International Conference on Parallel Processing, ICPP 2013 - Lyon, France
Duration: Oct 1 2013Oct 4 2013

Publication series

NameProceedings of the International Conference on Parallel Processing
ISSN (Print)0190-3918


Other42nd Annual International Conference on Parallel Processing, ICPP 2013


  • Data processing systems
  • Fault-tolerance
  • MapReduce
  • Prediction
  • Risk-management
  • Task assignment


Dive into the research topics of 'Risk intelligence: Profitting from uncertainty in data processing system'. Together they form a unique fingerprint.

Cite this