Automatically Identifying Security Checks for Detecting Kernel Semantic Bugs

Kangjie Lu, Aditya Pakki, Qiushi Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

OS kernels enforce a large number of security checks to validate system states. We observe that security checks are in fact very informative in inferring critical semantics in OS kernels. Specifically, security checks can reveal (1) whether an operation or a variable is critical but can be erroneous, (2) what particular errors may occur, and (3) constraints that should be enforced for the uses of a variable or a function. Such information is particularly valuable for detecting kernel semantic bugs because the detection typically requires understanding critical semantics. However, identifying security checks is challenging due to not only the lack of clear criteria but also the diversity of security checks. In this paper, we first systematically study security checks and propose a mostly-automated approach to identify security checks in OS kernels. Based on the information offered by the identified security checks, we then develop multiple analyzers that detect three classes of common yet critical semantic bugs in OS kernels, including NULL-pointer dereferencing, missing error handling, and double fetching. We implemented both the identification and the analyzers as LLVM passes and evaluated them using the Linux kernel and the FreeBSD kernel. Evaluation results show that our security-check identification has very low false-negative and false-positive rates. We also have found 164 new semantic bugs in both kernels, 88 of which have been fixed with our patches. The evaluation results confirm that our system can accurately identify security checks, which helps effectively identify numerous critical semantic bugs in complex OS kernels.

Original languageEnglish (US)
Title of host publicationComputer Security – ESORICS 2019 - 24th European Symposium on Research in Computer Security, Proceedings
EditorsKazue Sako, Steve Schneider, Peter Y.A. Ryan
PublisherSpringer
Pages3-25
Number of pages23
ISBN (Print)9783030299613
DOIs
StatePublished - Jan 1 2019
Event24th European Symposium on Research in Computer Security, ESORICS 2019 - Luxembourg, Luxembourg
Duration: Sep 23 2019Sep 27 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11736 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference24th European Symposium on Research in Computer Security, ESORICS 2019
CountryLuxembourg
CityLuxembourg
Period9/23/199/27/19

Keywords

  • Error handling
  • Missing check
  • OS kernel
  • Security check
  • Semantic bug

Fingerprint Dive into the research topics of 'Automatically Identifying Security Checks for Detecting Kernel Semantic Bugs'. Together they form a unique fingerprint.

  • Cite this

    Lu, K., Pakki, A., & Wu, Q. (2019). Automatically Identifying Security Checks for Detecting Kernel Semantic Bugs. In K. Sako, S. Schneider, & P. Y. A. Ryan (Eds.), Computer Security – ESORICS 2019 - 24th European Symposium on Research in Computer Security, Proceedings (pp. 3-25). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11736 LNCS). Springer. https://doi.org/10.1007/978-3-030-29962-0_1