A Comparative Study on Textual Saliency of Styles from Eye Tracking, Annotations, and Language Models

Karin de Langis, Dongyeop Kang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

There is growing interest in incorporating eye-tracking data and other implicit measures of human language processing into natural language processing (NLP) pipelines. The data from human language processing contain unique insight into human linguistic understanding that could be exploited by language models. However, many unanswered questions remain about the nature of this data and how it can best be utilized in downstream NLP tasks. In this paper, we present eyeStyliency, an eye-tracking dataset for human processing of stylistic text (e.g., politeness). We develop a variety of methods to derive style saliency scores over text using the collected eye dataset. We further investigate how this saliency data compares to both human annotation methods and model-based interpretability metrics. We find that while eye-tracking data is unique, it also intersects with both human annotations and model-based importance scores, providing a possible bridge between human- and machine-based perspectives. We propose utilizing this type of data to evaluate the cognitive plausibility of models that interpret style. Our eye-tracking data and processing code are publicly available.1

Original languageEnglish (US)
Title of host publicationCoNLL 2023 - 27th Conference on Computational Natural Language Learning, Proceedings
EditorsJing Jiang, David Reitter, Shumin Deng
PublisherAssociation for Computational Linguistics (ACL)
Pages108-121
Number of pages14
ISBN (Electronic)9798891760394
StatePublished - 2023
Event27th Conference on Computational Natural Language Learning, CoNLL 2023 - Singapore, Singapore
Duration: Dec 6 2023Dec 7 2023

Publication series

NameCoNLL 2023 - 27th Conference on Computational Natural Language Learning, Proceedings

Conference

Conference27th Conference on Computational Natural Language Learning, CoNLL 2023
Country/TerritorySingapore
CitySingapore
Period12/6/2312/7/23

Bibliographical note

Publisher Copyright:
© 2023 CoNLL 2023 - 27th Conference on Computational Natural Language Learning, Proceedings. All rights reserved.

Cite this