Keywords

1 Overview

How do we know what people perceive in a diagram? A diagram can be an excellent medium for communicating complex facts and relationships. Users may be able to learn a lot just from a quick glance at a well-designed diagram. Unfortunately, what users take from a diagram may not always be the same as what its designers intended to communicate. This can have enormous consequences, ranging from misinterpretation of research outputs to false representation in the media, to the point of misguided policy decisions coming from miscommunication of central research insights.

In this tutorial, we will look at the use of verbal protocols as a tool in the diagram design process. The way people talk about a diagram can reveal how they understand it, what they misinterpret, and what kinds of design features could be amended to enhance clarity, ensuring successful communication. This is supported by the methodology of Cognitive Discourse Analysis (CODA; Tenbrink 2015), which uses an in-depth linguistic approach to protocol analysis. Besides the (often quite revealing) content of what people say, the features of their language (how they say it) point to underlying conceptualisations and aspects that the speakers themselves may not be aware of: their focus of attention, aspects that are taken for granted or perceived as new, levels of granularity or detail, conceptual perspectives and switches between them, inferences and (possibly premature) conclusions, and so on.

The tutorial will start by briefly looking at the kinds of problems that frequently arise in diagram interpretation, such as cognitive biases, misinterpretation of accidental features (e.g. color, shape, or size of a symbol), and effects of lack of expertise. Then we will turn to the practical aspects of verbal data collection procedures, techniques for data preparation towards systematic analysis, features of content analysis, linguistic feature annotation (specific to CODA), reliability of annotation procedures, and identification of patterns in the results.

2 Background

Like other visualisation tools, diagrams can be excellent for facilitating the understanding of complex phenomena, in some areas more so than extensive descriptions in words (Larkin and Simon 1987). Since such tools represent relevant aspects of a situation in a schematic way, they are designed to support the perceiver in drawing strategical conclusions and making decisions based on useful heuristics, without necessarily considering every single aspect of the represented facts (Todd and Gigerenzer 2000).

When looking at a diagram, humans do not perceive all features and elements equally or objectively (as a computer might do). Instead, the perceiver’s attention is drawn towards visually salient elements (Fine and Minnery 2009) just as well as towards elements that are pertinent to a current task (Henderson et al. 2009). As a consequence, some aspects may remain entirely outside the perceiver’s consciousness.

Understanding these principles is vital for designing the visualisation of information in diagrams in a cognitively supportive way (Fabrikant and Goldsberry 2005). The mere inclusion of relevant information is not sufficient if it is not cognitively accessible in the way needed by the perceiver. Apart from failing to identify vital information, perceivers may also misinterpret the representation – for instance, they might confuse accidental features of a diagram as representing actual states or relationships in the real world. In this respect, conventions and expertise play a major role. Regular users of a particular type of diagram are less likely to be misguided than first-time observers. Likewise, the ability to extract relevant information from expert domain visualisations strongly depends on experience (Jee et al. 2009). Thus, perceivers of displayed information are biased by their background as well as by their perception of relevance in a given context. This will affect the inferences and decisions made on the basis of visualised information, leading to either desired or undesired outcomes.

Unfortunately, phenomena of this kind are not directly accessible to observation, since they concern structures and processes in the mind. Instead, access to internal processes is only possible indirectly, through analysis of external representations and measures. Methods to investigate the principles of understanding visualisations and diagrams along with their effectiveness for users encompass eye movement (Fabrikant and Goldsberry 2005) and sketch map (Jee et al. 2009) analysis, field usability studies (Sarjakoski and Nivala 2005), descriptions and arrow use (Heiser and Tversky 2006), etc.

One readily available external representation of cognition is language. Humans are used to speaking about their thoughts, and can normally express their understanding of a diagram in words. Nevertheless, interviewing people concerning their thoughts, as such, may not be sufficient; associated problems include issues around reliability, validity, and systematicity. Often, information gathered this way does not make its way into scientific reports, even if it serves to inform researchers on an informal basis.

Remedying these issues to some extent, Ericsson and Simon (1993) developed a systematic method to address higher-level cognitive processes by eliciting and analysing verbal protocols produced along with cognitively complex tasks, such as problem solving or decision-making. Think-aloud protocols and retrospective reports provide procedural information that systematically complements other data, such as decision outcomes and behavioural performance results. However, as the analysis of verbal reports in this paradigm typically remains on the content level, in many cases the insights gained in this way still remain illustrative and anecdotal, rather than being treated as substantial evidence. Also, since Ericsson and Simon (1993) primarily aimed to identify the sequence of cognitive steps taken to solve a problem or reach a decision, their approach may not be directly applicable to addressing the interpretation of diagrams. In this domain, the cognitively complex challenges do not necessarily emerge in distinct cognitive steps, even if there is a problem to solve and decisions to be made.

Cognitive Discourse Analysis (CODA; Tenbrink 2015) extends Ericsson and Simon’s (1993) approach in several respects. It provides an operationalised way of capturing verbalised content using linguistic insights, particularly from two areas that have been extensively studied and tested in English and some other Indo-European languages: Systemic Functional Linguistics (SFL; Halliday and Matthiessen 2014), and Cognitive Linguistics (CL; Talmy 2000, 2007; Evans and Green 2006). In particular, lexicogrammatical structures in language are systematically related to cognitive structures and processes. This structural fact carries over to principles of language in use: the way we think is related to the way we talk. This is true both generally in terms of what we can do with language, and specifically with respect to what we actually do – for example when verbalising thought related to visualisations such as expert domain diagrams.

When asked to verbalise their thoughts, speakers draw in systematic ways from their linguistic repertory to express currently relevant aspects. Their choices in relation to a cognitively demanding situation reveal crucial elements of their underlying conceptualisations and thought patterns. For instance, seemingly synonymous expressions such as over and above carry different implications and underlying concepts (Talmy 2007). While above clearly refers to the vertical dimension in There is a poster above the hole in the wall, over is actually polysemous. In There is a poster over the hole in the wall it seems reasonable to infer a functional sense of covering – a fundamentally different concept than verticality, and therefore a significant choice in the verbalisation.

In a verbal protocol describing the interpretation of a diagram, subtle differences such as that between over and above can become critical. It might matter to a high extent whether a particular feature of the diagram is perceived as geometrically vertical relative to another, or rather as functional in some sense, such as covering it.

A systematic analysis of such linguistic details provides a useful pathway to access cognition, drawing on knowledge about relevant features of language supported by grammatical theory, cognitive linguistic semantics, and other linguistic findings. Although linguistic expertise thus provides useful background, the general approach, to start with, is simple enough to be adopted by non-linguistic experts, with the most important feature being operationalisation and systematisation of language analysis. The methodological steps of CODA (Tenbrink 2015) are straightforwardly accessible to researchers across disciplines; they will be discussed in detail in this tutorial.

Besides building on established insights about the significance of particular linguistic choices, validating evidence for the relationship between patterns of language use and the associated cognitive processes can be gained by triangulation, i.e., the combination of linguistic analysis with other types of evidence such as memory or behavioural performance data, reaction times, eye movements, decision outcomes, or any other relevant data that can be collected in relation to diagrams. The outcome of a CODA-based analysis combined with such data is then a validated account of systematic features of diagram interpretation that may feed directly into design improvement.