Think-aloud verbal protocols and natural language processing to capture coherence building across narrative media

This study tests two theoretically motived hypotheses—the Lexical Precision hypothesis and the Privileged Access hypothesis—regarding media affordances for coherence building. In this exploratory study, we examine whether and how we can effectively assess these hypothesized media differences through the computational analysis of cohesion. Specifically, we leverage natural language processing (NLP) techniques to analyze archival verbal protocol data from a study in which participants (n = 51) thought aloud while reading text and nonlinguistic graphic versions of narratives. Participants’ verbal protocols were analyzed for lexical and semantic cohesion, serving as indicators of the coherence of the textbase and situation model aspects of their mental models, respectively. NLP was used to analyze the cohesion of these protocols. Lexical cohesion was assessed by measuring content word overlap in participants’ verbal protocols, and semantic cohesion was assessed by measuring semantic overlap. Our analyses accounted for individual differences in familiarity with text and visual media as well as correlations between the cohesion of their verbal protocols with textbase and situation model recall scores. We observed higher lexical cohesion in verbal protocols for text narratives and higher semantic cohesion for graphic narratives. These exploratory findings suggest that text narratives may provide lexical precision, which may in turn afford readers’ coherence building of the textbase, whereas graphic narratives may provide privileged access to knowledge activation, which then may afford readers’ coherence building of the situation model. This study highlights the suitability of NLP techniques for analyzing verbal protocols as a means to assess coherence-building processes.