Understanding human motion and gestures for underwater human–robot collaboration

Md Jahidul Islam, Marc Ho, Junaed Sattar

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

In this paper, we present a number of robust methodologies for an underwater robot to visually detect, follow, and interact with a diver for collaborative task execution. We design and develop two autonomous diver-following algorithms, the first of which utilizes both spatial- and frequency-domain features pertaining to human swimming patterns to visually track a diver. The second algorithm uses a convolutional neural network-based model for robust tracking-by-detection. In addition, we propose a hand gesture-based human–robot communication framework that is syntactically simpler and computationally more efficient than the existing grammar-based frameworks. In the proposed interaction framework, deep visual detectors are used to provide accurate hand gesture recognition; subsequently, a finite-state machine performs robust and efficient gesture-to-instruction mapping. The distinguishing feature of this framework is that it can be easily adopted by divers for communicating with underwater robots without using artificial markers or requiring memorization of complex language rules. Furthermore, we validate the performance and effectiveness of the proposed methodologies through a number of field experiments in closed- and open-water environments. Finally, we perform a user interaction study to demonstrate the usability benefits of our proposed interaction framework compared to the existing methods.

Original languageEnglish (US)
JournalJournal of Field Robotics
DOIs
StateAccepted/In press - Jan 1 2018

Fingerprint

Robots
Gesture recognition
Finite automata
End effectors
Detectors
Neural networks
Communication
Water
Experiments
Swimming

Cite this

Understanding human motion and gestures for underwater human–robot collaboration. / Islam, Md Jahidul; Ho, Marc; Sattar, Junaed.

In: Journal of Field Robotics, 01.01.2018.

Research output: Contribution to journalArticle

@article{60968cb0e7a04efd8a930bb3b448b434,
title = "Understanding human motion and gestures for underwater human–robot collaboration",
abstract = "In this paper, we present a number of robust methodologies for an underwater robot to visually detect, follow, and interact with a diver for collaborative task execution. We design and develop two autonomous diver-following algorithms, the first of which utilizes both spatial- and frequency-domain features pertaining to human swimming patterns to visually track a diver. The second algorithm uses a convolutional neural network-based model for robust tracking-by-detection. In addition, we propose a hand gesture-based human–robot communication framework that is syntactically simpler and computationally more efficient than the existing grammar-based frameworks. In the proposed interaction framework, deep visual detectors are used to provide accurate hand gesture recognition; subsequently, a finite-state machine performs robust and efficient gesture-to-instruction mapping. The distinguishing feature of this framework is that it can be easily adopted by divers for communicating with underwater robots without using artificial markers or requiring memorization of complex language rules. Furthermore, we validate the performance and effectiveness of the proposed methodologies through a number of field experiments in closed- and open-water environments. Finally, we perform a user interaction study to demonstrate the usability benefits of our proposed interaction framework compared to the existing methods.",
author = "Islam, {Md Jahidul} and Marc Ho and Junaed Sattar",
year = "2018",
month = "1",
day = "1",
doi = "10.1002/rob.21837",
language = "English (US)",
journal = "Journal of Field Robotics",
issn = "1556-4959",
publisher = "John Wiley and Sons Inc.",

}

TY - JOUR

T1 - Understanding human motion and gestures for underwater human–robot collaboration

AU - Islam, Md Jahidul

AU - Ho, Marc

AU - Sattar, Junaed

PY - 2018/1/1

Y1 - 2018/1/1

N2 - In this paper, we present a number of robust methodologies for an underwater robot to visually detect, follow, and interact with a diver for collaborative task execution. We design and develop two autonomous diver-following algorithms, the first of which utilizes both spatial- and frequency-domain features pertaining to human swimming patterns to visually track a diver. The second algorithm uses a convolutional neural network-based model for robust tracking-by-detection. In addition, we propose a hand gesture-based human–robot communication framework that is syntactically simpler and computationally more efficient than the existing grammar-based frameworks. In the proposed interaction framework, deep visual detectors are used to provide accurate hand gesture recognition; subsequently, a finite-state machine performs robust and efficient gesture-to-instruction mapping. The distinguishing feature of this framework is that it can be easily adopted by divers for communicating with underwater robots without using artificial markers or requiring memorization of complex language rules. Furthermore, we validate the performance and effectiveness of the proposed methodologies through a number of field experiments in closed- and open-water environments. Finally, we perform a user interaction study to demonstrate the usability benefits of our proposed interaction framework compared to the existing methods.

AB - In this paper, we present a number of robust methodologies for an underwater robot to visually detect, follow, and interact with a diver for collaborative task execution. We design and develop two autonomous diver-following algorithms, the first of which utilizes both spatial- and frequency-domain features pertaining to human swimming patterns to visually track a diver. The second algorithm uses a convolutional neural network-based model for robust tracking-by-detection. In addition, we propose a hand gesture-based human–robot communication framework that is syntactically simpler and computationally more efficient than the existing grammar-based frameworks. In the proposed interaction framework, deep visual detectors are used to provide accurate hand gesture recognition; subsequently, a finite-state machine performs robust and efficient gesture-to-instruction mapping. The distinguishing feature of this framework is that it can be easily adopted by divers for communicating with underwater robots without using artificial markers or requiring memorization of complex language rules. Furthermore, we validate the performance and effectiveness of the proposed methodologies through a number of field experiments in closed- and open-water environments. Finally, we perform a user interaction study to demonstrate the usability benefits of our proposed interaction framework compared to the existing methods.

UR - http://www.scopus.com/inward/record.url?scp=85056660864&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056660864&partnerID=8YFLogxK

U2 - 10.1002/rob.21837

DO - 10.1002/rob.21837

M3 - Article

JO - Journal of Field Robotics

JF - Journal of Field Robotics

SN - 1556-4959

ER -