Audio-visual spontaneous emotion recognition

Zhihong Zeng; Yuxiao Hu; Glenn I. Roisman; Zhen Wen; Yun Fu; Thomas S. Huang

doi:10.1007/978-3-540-72348-6_4

Audio-visual spontaneous emotion recognition

Zhihong Zeng, Yuxiao Hu, Glenn I. Roisman, Zhen Wen, Yun Fu, Thomas S. Huang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

36 Scopus citations

Abstract

Automatic multimodal recognition of spontaneous emotional expressions is a largely unexplored and challenging problem. In this paper, we explore audio-visual emotion recognition in a realistic human conversation setting-the Adult Attachment Interview (AAI). Based on the assumption that facial expression and vocal expression are at the same coarse affective states, positive and negative emotion sequences are labeled according to Facial Action Coding System. Facial texture in visual channel and prosody in audio channel are integrated in the framework of Adaboost multi-stream hidden Markov model (AdaMHMM) in which the Adaboost learning scheme is used to build component HMM fusion. Our approach is evaluated in AAI spontaneous emotion recognition experiments.

Original language	English (US)
Title of host publication	Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers
Pages	72-90
Number of pages	19
DOIs	https://doi.org/10.1007/978-3-540-72348-6_4
State	Published - 2007
Externally published	Yes
Event	20th International Joint Conference on Artificial Intelligence, IJCAI 2007 - Workshop on Artifical Intelligence for Human Computing - Hyderabad, India Duration: Jan 6 2007 → Jan 6 2007

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	4451 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	20th International Joint Conference on Artificial Intelligence, IJCAI 2007 - Workshop on Artifical Intelligence for Human Computing
Country/Territory	India
City	Hyderabad
Period	1/6/07 → 1/6/07

Keywords

Affect recognition
Affective computing
Emotion recognition
Multimodal human-computer interaction

Publisher link

10.1007/978-3-540-72348-6_4

Find It

Find It at U of M Libraries

Cite this

Zeng, Z., Hu, Y., Roisman, G. I., Wen, Z., Fu, Y., & Huang, T. S. (2007). Audio-visual spontaneous emotion recognition. In Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers (pp. 72-90). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 4451 LNAI). https://doi.org/10.1007/978-3-540-72348-6_4

Audio-visual spontaneous emotion recognition. / Zeng, Zhihong; Hu, Yuxiao; Roisman, Glenn I. et al.

Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers. 2007. p. 72-90 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 4451 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Zeng, Z, Hu, Y, Roisman, GI, Wen, Z, Fu, Y & Huang, TS 2007, Audio-visual spontaneous emotion recognition. in Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 4451 LNAI, pp. 72-90, 20th International Joint Conference on Artificial Intelligence, IJCAI 2007 - Workshop on Artifical Intelligence for Human Computing, Hyderabad, India, 1/6/07. https://doi.org/10.1007/978-3-540-72348-6_4

Zeng Z, Hu Y, Roisman GI, Wen Z, Fu Y, Huang TS. Audio-visual spontaneous emotion recognition. In Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers. 2007. p. 72-90. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-540-72348-6_4

Zeng, Zhihong ; Hu, Yuxiao ; Roisman, Glenn I. et al. / Audio-visual spontaneous emotion recognition. Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers. 2007. pp. 72-90 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{41c32abc229d4e4abec41ce1724f38a5,

title = "Audio-visual spontaneous emotion recognition",

abstract = "Automatic multimodal recognition of spontaneous emotional expressions is a largely unexplored and challenging problem. In this paper, we explore audio-visual emotion recognition in a realistic human conversation setting-the Adult Attachment Interview (AAI). Based on the assumption that facial expression and vocal expression are at the same coarse affective states, positive and negative emotion sequences are labeled according to Facial Action Coding System. Facial texture in visual channel and prosody in audio channel are integrated in the framework of Adaboost multi-stream hidden Markov model (AdaMHMM) in which the Adaboost learning scheme is used to build component HMM fusion. Our approach is evaluated in AAI spontaneous emotion recognition experiments.",

keywords = "Affect recognition, Affective computing, Emotion recognition, Multimodal human-computer interaction",

author = "Zhihong Zeng and Yuxiao Hu and Roisman, {Glenn I.} and Zhen Wen and Yun Fu and Huang, {Thomas S.}",

note = "Copyright: Copyright 2008 Elsevier B.V., All rights reserved.; 20th International Joint Conference on Artificial Intelligence, IJCAI 2007 - Workshop on Artifical Intelligence for Human Computing ; Conference date: 06-01-2007 Through 06-01-2007",

year = "2007",

doi = "10.1007/978-3-540-72348-6_4",

language = "English (US)",

isbn = "3540723463",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "72--90",

booktitle = "Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers",

}

TY - GEN

T1 - Audio-visual spontaneous emotion recognition

AU - Zeng, Zhihong

AU - Hu, Yuxiao

AU - Roisman, Glenn I.

AU - Wen, Zhen

AU - Fu, Yun

AU - Huang, Thomas S.

PY - 2007

Y1 - 2007

N2 - Automatic multimodal recognition of spontaneous emotional expressions is a largely unexplored and challenging problem. In this paper, we explore audio-visual emotion recognition in a realistic human conversation setting-the Adult Attachment Interview (AAI). Based on the assumption that facial expression and vocal expression are at the same coarse affective states, positive and negative emotion sequences are labeled according to Facial Action Coding System. Facial texture in visual channel and prosody in audio channel are integrated in the framework of Adaboost multi-stream hidden Markov model (AdaMHMM) in which the Adaboost learning scheme is used to build component HMM fusion. Our approach is evaluated in AAI spontaneous emotion recognition experiments.

AB - Automatic multimodal recognition of spontaneous emotional expressions is a largely unexplored and challenging problem. In this paper, we explore audio-visual emotion recognition in a realistic human conversation setting-the Adult Attachment Interview (AAI). Based on the assumption that facial expression and vocal expression are at the same coarse affective states, positive and negative emotion sequences are labeled according to Facial Action Coding System. Facial texture in visual channel and prosody in audio channel are integrated in the framework of Adaboost multi-stream hidden Markov model (AdaMHMM) in which the Adaboost learning scheme is used to build component HMM fusion. Our approach is evaluated in AAI spontaneous emotion recognition experiments.

KW - Affect recognition

KW - Affective computing

KW - Emotion recognition

KW - Multimodal human-computer interaction

UR - http://www.scopus.com/inward/record.url?scp=49949108559&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=49949108559&partnerID=8YFLogxK

U2 - 10.1007/978-3-540-72348-6_4

DO - 10.1007/978-3-540-72348-6_4

M3 - Conference contribution

AN - SCOPUS:49949108559

SN - 3540723463

SN - 9783540723462

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 72

EP - 90

BT - Artifical Intelligence for Human Computing, ICMI 2006 and IJCAI 2007 International Workshops, Banff, Canada, November 3, 2006 and Hyderabad, India, January 6, 2007, Revised Seleced and Invited Papers

T2 - 20th International Joint Conference on Artificial Intelligence, IJCAI 2007 - Workshop on Artifical Intelligence for Human Computing

Y2 - 6 January 2007 through 6 January 2007

ER -

Audio-visual spontaneous emotion recognition

Abstract

Publication series

Other

Keywords

Publisher link

Find It

Other files and links

Fingerprint

Cite this