The role of visual attention in sentiment prediction

Shaojing Fan, Ming Jiang, Zhiqi Shen, Bryan L. Koenig, Mohan S. Kankanhalli, Qi Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations

Abstract

Automated assessment of visual sentiment has many applications, such as monitoring social media and facilitating online advertising. In current research on automated visual sentiment assessment, images are mainly input and processed as a whole. However, human attention is biased, and a focal region with high acuity can disproportionately influence visual sentiment. To investigate how attention influences visual sentiment, we conducted experiments that reveal critical insights into human perception. We discover that negative sentiments are elicited by the focal region without a notable influence of contextual information, whereas positive sentiments are influenced by both focal and contextual information. Building on these insights, we create new deep convolutional neural networks for sentiment prediction that have additional channels devoted to encoding focal information. On two benchmark datasets, the proposed models demonstrate superior performance compared with the state-of-the-art methods. Extensive visualizations and statistical analyses indicate that the focal channels are more effective on images with focal objects, especially for images that also elicit negative sentiments.

Original languageEnglish (US)
Title of host publicationMM 2017 - Proceedings of the 2017 ACM Multimedia Conference
PublisherAssociation for Computing Machinery, Inc
Pages217-225
Number of pages9
ISBN (Electronic)9781450349062
DOIs
StatePublished - Oct 23 2017
Event25th ACM International Conference on Multimedia, MM 2017 - Mountain View, United States
Duration: Oct 23 2017Oct 27 2017

Publication series

NameMM 2017 - Proceedings of the 2017 ACM Multimedia Conference

Other

Other25th ACM International Conference on Multimedia, MM 2017
CountryUnited States
CityMountain View
Period10/23/1710/27/17

Keywords

  • Neural network
  • Social multimedia
  • Visual sentiment

Fingerprint Dive into the research topics of 'The role of visual attention in sentiment prediction'. Together they form a unique fingerprint.

  • Cite this

    Fan, S., Jiang, M., Shen, Z., Koenig, B. L., Kankanhalli, M. S., & Zhao, Q. (2017). The role of visual attention in sentiment prediction. In MM 2017 - Proceedings of the 2017 ACM Multimedia Conference (pp. 217-225). (MM 2017 - Proceedings of the 2017 ACM Multimedia Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/3123266.3123445