Adapting user interfaces for gestural interaction with the flexible action and articulated skeleton toolkit

Evan A. Suma, David M. Krum, Belinda Lange, Sebastian Koenig, Albert Rizzo, Mark Bolas

Research output: Contribution to journalArticlepeer-review

54 Scopus citations


We present the Flexible Action and Articulated Skeleton Toolkit (FAAST), a middleware software framework for integrating full-body interaction with virtual environments, video games, and other user interfaces. This toolkit provides a complete end-to-end solution that includes a graphical user interface for custom gesture creation, sensor configuration, skeletal tracking, action recognition, and a variety of output mechanisms to control third party applications, allowing virtually any PC application to be repurposed for gestural control even if it does not explicit support input from motion sensors. To facilitate intuitive and transparent gesture design, we define a syntax for representing human gestures using rule sets that correspond to the basic spatial and temporal components of an action. These individual rules form primitives that, although conceptually simple on their own, can be combined both simultaneously and in sequence to form sophisticated gestural interactions. In addition to presenting the system architecture and our approach for representing and designing gestural interactions, we also describe two case studies that evaluated the use of FAAST for controlling first-person video games and improving the accessibility of computing interfaces for individuals with motor impairments. Thus, this work represents an important step toward making gestural interaction more accessible for practitioners, researchers, and hobbyists alike.

Original languageEnglish (US)
Pages (from-to)193-201
Number of pages9
JournalComputers and Graphics (Pergamon)
Issue number3
StatePublished - 2013
Externally publishedYes


  • Gesture
  • Middleware
  • Natural interaction
  • User interfaces
  • Video games


Dive into the research topics of 'Adapting user interfaces for gestural interaction with the flexible action and articulated skeleton toolkit'. Together they form a unique fingerprint.

Cite this