Ellipse R-CNN: Learning to Infer Elliptical Object from Clustering and Occlusion

Wenbo Dong, Pravakar Roy, Cheng Peng, Volkan Isler

Research output: Contribution to journalArticlepeer-review

36 Scopus citations


Images of heavily occluded objects in cluttered scenes, such as fruit clusters in trees, are hard to segment. To further retrieve the 3D size and 6D pose of each individual object in such cases, bounding boxes are not reliable from multiple views since only a little portion of the object's geometry is captured. We introduce the first CNN-based ellipse detector, called Ellipse R-CNN, to represent and infer occluded objects as ellipses. We first propose a robust and compact ellipse regression based on the Mask R-CNN architecture for elliptical object detection. Our method can infer the parameters of multiple elliptical objects even they are occluded by other neighboring objects. For better occlusion handling, we exploit refined feature regions for the regression stage, and integrate the U-Net structure for learning different occlusion patterns to compute the final detection score. The correctness of ellipse regression is validated through experiments performed on synthetic data of clustered ellipses. We further quantitatively and qualitatively demonstrate that our approach outperforms the state-of-the-art model (i.e., Mask R-CNN followed by ellipse fitting) and its three variants on both synthetic and real datasets of occluded and clustered elliptical objects.

Original languageEnglish (US)
Article number9329165
Pages (from-to)2193-2206
Number of pages14
JournalIEEE Transactions on Image Processing
StatePublished - 2021

Bibliographical note

Funding Information:
Manuscript received January 31, 2020; revised August 22, 2020 and January 3, 2021; accepted January 7, 2021. Date of publication January 20, 2021; date of current version January 27, 2021. This work was supported in part by USDA-NIFA under Grant MIN-98-G02, in part by the MnDrive Initiative, and in part by NSF under Grant 1722310 and Grant 1849107. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Jiaying Liu. (Corresponding author: Wenbo Dong.) Wenbo Dong and Pravakar Roy were with the Department of Computer Science and Engineering, University of Minnesota, Twin Cities, Minneapolis, MN 55455 USA (e-mail: dongx358@umn.edu; royxx268@umn.edu).

Publisher Copyright:
© 1992-2012 IEEE.


  • 3D object localization
  • Ellipse regression
  • convolutional neural networks
  • object detection
  • occlusion handling


Dive into the research topics of 'Ellipse R-CNN: Learning to Infer Elliptical Object from Clustering and Occlusion'. Together they form a unique fingerprint.

Cite this