Robust foreground detection in video using pixel layers

Kedar Patwardhan, Guillermo Sapiro, Vassilios Morellas

Research output: Contribution to journalArticlepeer-review

82 Scopus citations


A framework for robust foreground detection that works under difficult conditions such as dynamic background and moderately moving camera is presented in this paper. The proposed method includes two main components: coarse scene representation as the union of pixel layers, and foreground detection in video by propagating these layers using a maximum-likelihood assignment. We first cluster into "layers" those pixels that share similar statistics. The entire scene is then modeled as the union of such non-parametric layer-models. An in-coming pixel is detected as foreground if it does not adhere to these adaptive models of the background. A principled way of computing thresholds is used to achieve robust detection performance with a pre-specified number of false alarms. Correlation between pixels in the spatial vicinity is exploited to deal with camera motion without precise registration or optical flow. The proposed technique adapts to changes in the scene, and allows to automatically convert persistent foreground objects to background and re-convert them to foreground when they become {interesting}. This simple framework addresses the important problem of robust foreground and unusual region detection, at about 10 frames per second on a standard laptop computer. The presentation of the proposed approach is complemented by results on challenging real data and comparisons with other standard techniques.

Original languageEnglish (US)
Pages (from-to)746-751
Number of pages6
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number4
StatePublished - Apr 2008

Bibliographical note

Funding Information:
This work is partially supported by the US Office of Naval Research, the US National Science Foundation, NGA, ARO, and the US Defense Advanced Research Projects Agency. The authors would like to acknowledge the help of Honeywell Labs and Professor Larry Davis for providing comparative results shown in Fig. 4. The authors are also grateful to Professor Mubarak Shah and Dr. Yaser Sheikh for providing them with the original video and segmented ground-truth shown in Fig. 6. The AE Dr. Harpreet Sawhney and the reviewers selected by him have helped significantly in improving the presentation of key ideas and contributions of this work.


  • Background subtraction
  • Foreground detection
  • Layer tracking
  • Scene analysis
  • Surveillance
  • Video analysis


Dive into the research topics of 'Robust foreground detection in video using pixel layers'. Together they form a unique fingerprint.

Cite this