Rapid photorealistic blendshape modeling from RGB-D sensors

Dan Casas, Andrew Feng, Oleg Alexander, Graham Fyffe, Paul Debevec, Ryosuke Ichikari, Hao Li, Kyle Olszewski, Evan Suma, Ari Shapiro

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations


Creating and animating realistic 3D human faces is an important element of virtual reality, video games, and other areas that involve interactive 3D graphics. In this paper, we propose a system to generate photorealistic 3D blendshape-based face models automatically using only a single consumer RGB-D sensor. The capture and processing requires no artistic expertise to operate, takes 15 seconds to capture and generate a single facial expression, and approximately 1 minute of processing time per expression to transform it into a blendshape model. Our main contributions include a complete end-To-end pipeline for capturing and generating photorealistic blendshape models automatically and a registration method that solves dense correspondences between two face scans by utilizing facial landmarks detection and optical flows. We demonstrate the effectiveness of the proposed method by capturing different human subjects with a variety of sensors and puppeteering their 3D faces with real-Time facial performance retargeting. The rapid nature of our method allows for just-in-Time construction of a digital face. To that end, we also integrated our pipeline with a virtual reality facial performance capture system that allows dynamic embodiment of the generated faces despite partial occlusion of the user's real face by the head-mounted display.

Original languageEnglish (US)
Title of host publicationProceedings of the 29th International Conference on Computer Animation and Social Agents, CASA 2016
PublisherAssociation for Computing Machinery
Number of pages9
ISBN (Electronic)9781595930361
StatePublished - May 23 2016
Externally publishedYes
Event29th International Conference on Computer Animation and Social Agents, CASA 2016 - Geneva, Switzerland
Duration: May 23 2016May 25 2016

Publication series

NameACM International Conference Proceeding Series


Other29th International Conference on Computer Animation and Social Agents, CASA 2016


  • Animation
  • Blendshapes
  • Face modeling
  • RGB-D

Fingerprint Dive into the research topics of 'Rapid photorealistic blendshape modeling from RGB-D sensors'. Together they form a unique fingerprint.

Cite this