Keywords

Introduction

Cogito, ergo sum. According to the standard philosophical interpretation of this well-known statement, Descartes expressed that we know that we exist because we are aware of our thoughts. In other words, our existence depends on our ability to be aware of ourselves as agents in the world. Alternatively, though, one could argue that sum, ergo cogito, for it is the very fact that we exist – that our brains reside in a physical body– that enables us to be conscious. That is, the experience of our thoughts depends on our physical manifestation and interaction with the world. In that way, cognition and perception are intertwined with action, and together, our minds and bodies interact in order to navigate the world around us.

The way we act upon the world around us is both constrained and driven by the affordances of our environment, which we learn through experience and knowledge acquisition throughout life. For instance, we know that a glass can hold liquid, that we can drink from it by picking it up and bringing it to the mouth, and that it shatters when it falls. This means that we can also make predictions: if a glass drops from one’s hands, there will be a noise followed by sharp shards lying on the floor. Hearing the noise and potentially freezing (because the new situation might be dangerous) are typical ways in which the body interacts with the environment: perception (a noise) leads to a prediction (danger), which leads to an action (freezing). This, in turn, leads to a new situation in which one checks the environment for sharp shards, followed again by predictions and actions. This process is known as the perception-action cycle (Fuster, 2002, 2004). The fact that we constantly make predictions about the environment that guide our actions means that we need to be aware of the affordances of the environment. This means that there is a tight coupling between our behavior and the environment such that strong predictions make movements almost automatic, while small deviations inevitably lead to accidents.

These deviations become more likely when our attention needs to be divided between the world around us and the world within us. During a substantial part of our daily lives, our thoughts are not focused outward toward external events and stimuli, but rather inward, processing internal states that are decoupled from the reality around us at that particular moment. For centuries, philosophers, writers, and scientists have tried to understand the purpose and dynamics of such thoughts, yet only more recently, we have begun to examine experimentally how and why humans entertain cognitions with little relation to external events, and how this process manifests in observable behavior. More interest in internally directed cognition has been ignited by the discovery of the default mode network by Raichle et al. (2001), who found that during resting episodes recorded using functional magnetic resonance imaging (fMRI), the brain is, in fact, far from idle. A stream of thoughts flows through our minds related to exteroceptive signals (e.g., lights, sounds, smells), interoceptive signals (e.g., hunger, tiredness), as well as internally generated, stimulus-independent thoughts akin to mental simulations and related to our memories, goals, and plans for the future. This process is often referred to as mind wandering, and can be characterized by a decoupling of attention and information processing from the external environment in favor of internally generated thoughts and feelings (Smallwood, 2013; Smallwood & Andrews-Hanna, 2013; Smallwood & Schooler, 2006). Numerous studies have shown that mind wandering sometimes arises spontaneously and without conscious awareness, while at other times, it appears to be a deliberative act where attention is consciously directed to a particular train of thought (Seli et al., 2016). A typical example of mind wandering without awareness, or “zoning out,” occurs during reading when we may find ourselves reaching the end of a page, but having no idea what we just read or where our thoughts went in the meantime (Schooler et al., 2011).

The main argument of this chapter is that mind wandering can influence bodily behavior by causing a partial breakdown of the perception-action cycle. Perception of external stimuli is attenuated during mind wandering, and, therefore, predictions and actions become more inaccurate or less efficient. In addition, mind wandering affects the body in other ways: one’s posture might change, and the change in mental state might be reflected in facial features. Interestingly, there are also specific non-instrumental behaviors, such as fidgeting, that are associated with mind wandering, suggesting that mind wandering not only changes how the body interacts with the environment, but also that mind wandering is (at least to some extent) embodied.

On the Costs and Benefits of Mind Wandering

Mind wandering seems to be an essential human characteristic which enables us to remember the past, to plan for the future (Baird et al., 2011; Mooneyham & Schooler, 2013; Smallwood & Schooler, 2015), and to be creative (Baird et al., 2012). It also provides us with freedom from immediacy (Smallwood & Andrews-Hanna, 2013) and makes it possible for us to travel through time as we daydream (Baird et al., 2011). In fact, mind wandering seems to allow us to integrate our past and present selves with our future and imaginative experiences, serving to consolidate our memories (Wamsley, 2018) and to create and maintain a coherent sense of self (Mooneyham & Schooler, 2013; Ottaviani et al., 2013; Smallwood & Andrews-Hanna, 2013; Tulving, 1987). Some researchers assume that mind wandering can be linked to the default state of the human brain (Mills et al., 2018). In this default state, thoughts ceaselessly move from one topic to the next, with heightened variability over time. This flow and variability might serve to improve episodic memory efficiency (Faber, 2020). This seems to be supported by studies both in laboratory settings (Wamsley & Summer, 2020) and in daily life (Smith et al., 2018).

Despite its various benefits, mind wandering has been found to be detrimental in a wide variety of contexts, including both nondemanding and challenging tasks. It has been associated with decreased text comprehension (Krawietz et al., 2012; Smallwood et al., 2008b) as well as increased number of errors in memory (Riby et al., 2008), including working memory (Banks & Boals, 2017; Mrazek et al., 2012) and vigilance tasks (McVay & Kane, 2012; Stawarczyk et al., 2011). In daily life, mind wandering has been related to lower performance in general aptitude tests (Mrazek et al., 2012), and learning and performance in academic contexts (Wammes et al., 2018). It has even been used to explain differences in the Socioeconomic Status Academic Achievement Gap (Gearin et al., 2018).

The ebb and flow of mind-wandering thoughts is dependent on a variety of factors. As such, the extent to which mind wandering is detrimental (or conversely, beneficial) varies largely according to the context in which it takes place. Across a variety of experience sampling studies, studentsFootnote 1 report mind wandering during around 30% of their daily lives (Kane et al., 2007; McVay et al., 2009). Unsworth and Mcmillan (2012) found that three quarters of these mind-wandering reports take place in the classroom. Mind wandering is therefore more likely to take place during classroom-related activities than in everyday life. However, mind-wandering rates vary according to the type of activity being performed in the classroom. For example, Schoen (1970) notes that students report being focused approximately 67% of the time during lectures, 75% of the time during discussions, and 83% of the time when problem-solving. Not surprisingly, less mind wandering takes place in more interactive and engaging activities in the classroom. More recently, Wammes et al. (2016) found mind wandering in the classroom to be related to both short-term (quizzes) and long-term (exams) performance decrements.Footnote 2

As educational activities require considerably more sustained attentional focus than everyday activities, it is not surprising that more mind wandering takes place in classroom settings. In line with this, the negative consequences of mind wandering in tasks requiring sustained attention – such as attending a lecture, reading an article, or studying for an exam – are also greater than in largely automatized day-to-day tasks, such as having breakfast, checking e-mail, or scrolling through social media feed (Szpunar et al., 2013). Relatedly, mind-wandering frequency generally decreases with task difficulty. However, once a task becomes too difficult (Smallwood, 2013; Smallwood & Andrews-Hanna, 2013), mind-wandering rates increase again. Whenever a task is easy, there are sufficient attentional resources both for task performance and for mind wandering. Once a task becomes exceedingly difficult, because of either lack of knowledge or resource depletion, attention is decoupled from the task at hand and mind wandering ensues (Randall et al., 2014; Smallwood, 2013; Smallwood & Andrews-Hanna, 2013).

Given the prevalence and the detrimental effects of mind wandering in classroom settings, it would be helpful if educators could be provided with tools to detect when these episodes take place. Such information might also be particularly useful for intelligent tutoring systems. In what follows, we discuss the relationship between cognition and bodily behaviors in the context of performance and learning. We then address the value of integrating bodily behaviors into cognitive architectures in order to further our understanding of mind wandering in the context of perception and action.

Mind Wandering and Bodily Behaviors

During most of our waking moments, we are engaged in some sort of movement, often well practiced and automatized, such as reaching for objects, walking, and speaking. We do not need to actively think about these well-practiced actions – we simply perform them as we engage in goal-oriented behavior, with little to no demands on our attention (or conscious input). There are, however, bodily movements that do not serve a clear purpose in the outside world – such as fidgeting, tapping one’s fingers or feet, rubbing the chin, or twirling the hair while paying attention to an unrelated external stimulus. Arguably, these types of behaviors are indicative of mind wandering and thus represent physical expressions of our mental state, i.e., its embodiment. In the following sections, we first address the manifestation of mind wandering in disrupted executions of bodily behaviors associated with goal-oriented, attentive actions and then discuss the links between mind wandering and non-instrumental bodily behaviors.

Sensory-Motor Decoupling

There is a bidirectional relationship between attention and correlated body movements such that regions in the brain associated with motor-planning influence attention (Armstrong & Moore, 2007; Knudsen, 2007; Moore et al., 2003), and in turn attention influences sensorimotor brain areas (Rosenkranz & Rothwell, 2004) as well as sensorimotor integration (Velasques et al., 2013). When the mind wanders, there is an attenuation of processing in neural systems that are often engaged with the external sensory-motor environment in order to guide behavior (Kam & Handy, 2013; Smallwood & Andrews-Hanna, 2013). Various studies indicate that there is a decrease in alertness and sensory processing during mind wandering. In support of this claim, experimental studies consistently report higher variability in reaction times (e.g., McVay & Kane, 2009; van Vugt & Broers, 2016) and reduced accuracy in a variety of tasks (e.g., Smallwood & Schooler, 2015) during mind-wandering states. Although it is not yet clear at what point behavior starts to waver, previous work has shown that in a metronome task, behavioral variability is significantly higher across the five trials prior to a mind-wandering report than before an on-task response (Seli et al., 2013). Indeed, there is evidence to suggest that the response time variability in the four to eight trials preceding a positive mind-wandering probe is a robust predictor of mind wandering (e.g., Bastian & Sackur, 2013).

Although the increase in behavioral variability has been firmly established in the literature, there are conflicting findings with regard to whether responses speed up or slow down during mind wandering across a variety of tasks. While some studies have shown that faster responses are associated with mind wandering (e.g., SART; McVay & Kane, 2012; McVay et al., 2009, collapsed across four trials preceding a report), others have demonstrated that response times linearly decrease during mind-wandering episodes (Bastian & Sackur, 2013; Smallwood et al., 2008a). When further investigating the time course of responses prior to a mind-wandering report, there is evidence to suggest that response times are in fact faster in the five to two trials before a mind-wandering report, followed by a sharp decrease in the trial just before the report (Henríquez et al., 2016). Despite methodological differences across the cited studies, these findings appear to suggest that the variability associated with mind wandering is not simply a result of linearly slowing down, but potentially speeding up and then slowing down. Further research that scrutinizes the time course of on- and off-task behavior across larger time scales might shed light on these time-dependent relationships. Taken together, and irrespective of the direction of the relationship, these findings point towards the idea that bodily behavior (in this case, response time) deviates from on-task behavior during mind wandering, suggesting that there is a degree of decoupling between the perception of the external environment and the bodily action.

Hand Movements

Behavioral measures such as reaction times provide valuable insight with regard to the relation between mind wandering and task performance. However, these measures are unable to capture the fine-grained dynamics of movement leading up to the crucial moment during which performance is measured (usually a click or a button press). As embodied cognizers, we adaptively monitor and minutely adjust our movements in response to external demands on a moment-to-moment basis in our daily lives. Therefore, it is likely that the effects of mind wandering on behavioral control involve more than just speed and accuracy of responses (Kam et al., 2012). Dynamic measures across time obtained from process tracing methods are a promising method that allows for collecting more detailed information about this process.

Mind Wandering During a Forced-Choice Reaching Task

In the study reported by Dias da Silva and Postma (2020), we tracked participants’ hand (motor) movements and measured mind wandering under an engaging and cognitively demanding task.Footnote 3 During this task, participants were instructed to memorize a series of letters while at the same time performing mathematical operations for approximately 20 minutes. After each set of letter recall and math operations, participants could have selected one out of three probe responses: (1) I was focused on the task, (2) I was concerned about my performance on the task, or (3) I was thinking about something unrelated to the task, where the third alternative indicated mind wandering (operationalized as task-unrelated thought). We extracted various mouse tracking measures from x- and y-coordinates recorded across time. Using these measures as features in several machine learning models, we were able to predict mind wandering above chance level. We found that computer mouse movements become more complex (operationalized by more direction changes along the x- and y-axes), less direct, and slower during mind wandering than during moments of focused attention. Upon closer observation of the speed of the movements, we found that not only were movements slower in general, but also the first phase of the reaching movement toward a response was slower. More specifically, this means that individuals took longer to commit to a response whenever they were mind wandering.

Mind Wandering During a Visuomotor Tracking Task

In a second study reported by Dias da Silva and Postma (2021), we investigated the relationship between fine motor movements during a monotonous tracking task, lasting approximately 1 hour. Participants were instructed to trace the path of a moving ball on a screen while intermittently reporting whether or not they were focused on the task. Whenever they were mind wandering, participants indicated to what extent their attention was decoupled from the environment, to what extent they imagined being somewhere else, and to what extent the content of their thoughts varied. We found that whenever participants were mind wandering, their hand movements deviated more from the path of the ball and were less variable. Moreover, the deeper the reported episodes of mind wandering, the more erratic and less variable their hand movements.

In line with previous work (Kam et al., 2012), we found that fine motor movements change in relation to one’s attentional state. During both a reaching task and during a tracking task, the action-perception loop appears to be disrupted by the mind-wandering process resulting in less efficient hand movements.

Eye Movements

Changes in eye movements have also been extensively investigated in relation to mind wandering in a variety of tasks, ranging from reading (Bixler & D’Mello, 2016) and online lectures (Khorrami et al., 2014) to interactions with automatic tutoring systems (Hutt et al., 2016), and have been found to be good predictors of mind wandering and attention. Taken together, deviations in gaze patterns from on-task, instrumental behavior in various tasks suggest a decoupling between gaze and the external environment. Studies related to eye tracking are reported extensively in Dias da Silva et al. (2022), this book.

Vocal Movements

The production of speech constitutes a highly automated type of movement involving precise actions of the muscles in the vocal apparatus. Among these, the opening and closing of vocal folds results in minor changes in pitch, the perceptual correlate of fundamental frequency. In terms of the auditory characteristics of speech, pitch is an important indicator of the identity, emotions, and attitudes of a speaker (Postma-Nilsenová et al., 2013). Moreover, the ability to correctly perceive pitch in another’s speech and to adapt one’s pitch accordingly is indicative of rapport, cooperation, and social proximity (Dias da Silva et al., 2018; Giles, 2008; Pardo, 2006; Postma-Nilsenová et al., 2013; Postma-Nilsenová & Postma, 2013). In fact, during vocal interactions, speakers unknowingly accommodate to one another’s pitch patterns. In a study with a virtual agent (Dias da Silva et al., 2018), we observed that participants who were induced into a repetitive, self-focused style of thinking, characteristic of ruminative mind wandering, exhibited a reduction in pitch accommodation. As such, we thus provide initial evidence for the manifestation of mind wandering in less adjusted vocal movements.

Non-instrumental Movement

As discussed earlier, many of our goal-related actions are expressed through bodily movements. For instance, the eyes might move across the screen to sample visual information, the head might move to get a better viewing angle, one might lean in to take a closer look, or one might operate a computer mouse or touch screen using their hand to navigate the screen (Witchel et al., 2014). However, not all movements are associated with goal-oriented actions. For example, fidgeting is a common, but non-instrumental, behavior we exhibit. Other examples include changes in posture that are not instrumental to the task, such as leaning back as a sign of disengagement; hand movements that are non-instrumental, such as touching the face, rubbing the eyes, or scratching; and facial expressions (Witchel et al., 2014). Relatedly, recent evidence also suggests that what was previously thought to be “nonessential” behavior plays an invaluable role in shaping the neural activity in expert mice performing tasks (Mathis, 2019). There are, however, mixed findings with regard to the relationship between mind wandering and non-instrumental movement. Sometimes non-instrumental movement (e.g., tapping fingers along with the rhythm of a song) is associated with attention or engagement toward a task, while others are associated with disengagement or mind wandering (e.g., restless foot or leg movement). Witchel et al. (2014) reconcile such discrepancies by suggesting that the attentional state can be distinguished by whether or not movements are entrained – that is, whether movements are timed to the rhythm of an external stimulus. We propose here that mind wandering may be reflected in such types of non-instrumental movements, which are not entrained to stimuli, suggesting that non-instrumental movement could be seen as an “embodied” manifestation of mind wandering.

In support of this notion, studies have shown fidgeting to significantly increase during unintentional mind wandering (Carriere et al., 2013; Seli et al., 2014). In a first questionnaire study, Carriere et al. (2013) found that participants who report mind wandering more (both deliberately and spontaneously) also report more fidgeting. In a study assessing fidgeting behavior (as coded by external observers) while students watched an online lecture, Farley et al. (2013) found both macro fidgeting behavior (operationalized as a complete spatial displacement of a body part relative to a starting position, such as moving the arm to a completely new location) and mind wandering to be related to one another and to increase with time on task. Moreover, Seli et al. (2014) found particularly deep levels of mind wandering to be associated with fidgeting (operationalized as the total amount of movement detected by a Wii Balance Board) during a Metronome Response Task. Finally, Witchel et al. (2019) found that while reading an interesting novel, students fidgeted less than when reading a boring novel. Similarly, doodling or humming a tune (a vocal movement) during performance of a monotonous task could also be an indicator of mind wandering (Farley et al., 2013; Smallwood & O’Connor, 2011). These findings may suggest that non-instrumental movements, reflective of mind wandering, may be a way to cope with boredom during a task (Elpidorou, 2018).

Facial Features

Facial expressions are another non-instrumental behavior commonly found to accompany attentional states during task performance. Various studies have found that facial features can be used to detect engagement and attentional focus (or a lack thereof) during computerized tasks (Monkaresi et al., 2017; Whitehill et al., 2014). In a recent study by Benedek et al. (2018), participants were asked to determine the locus of attention of people in various videos. The videos showed the faces of people either who were focusing their attention externally on a task or who were focused internally while performing the task in their mind’s eye. People in the video were asked to perform the following four tasks: solve an anagram in the computer screen (demanding external condition), solve an anagram in their mind’s eye (demanding internal condition), count the number of journeys made by the tractor in a video on the screen (easy external condition), and imagine themselves on a beach and exploring this environment (easy internal condition). Participants who evaluated the videos found that the eye region was the most important determining factor for their judgments. They found a different pattern (e.g., directed eye movements in external attention vs. empty gaze during internal attention) and speed of eye movements to be discriminative of internal and external focused attention. Overall, participants were able to determine people’s locus of attention at above chance levels from the videos, but had difficulty distinguishing between internal and external attention during more demanding tasks. This is not surprising, considering the fact that solving an anagram in the mind’s eye is equally difficult, if not more difficult, than on a screen. Working memory and executive resources are engaged in both of these tasks, resulting in more tenseFootnote 4 facial expressions for both conditions. Moreover, mental imagery under this demanding task likely resulted in similar eye movements to those from actual perception of stimuli on the screen (Johnson & Whisman, 2013). Taken together, these findings indicate that there are overt indicators of attention which enable us to detect others’ attentional states from facial expressions. Other studies have instead used machine learning techniques in combination with self-reported measures of mind wandering, demonstrating initial evidence for automatic mind-wandering detectors outperforming human observers in determining other’s attentional states (Bosch, 2016). For example, both Stewart et al. (2017) and Bosch (2016) used facial and upper body features extracted from video recordings both in the lab and in the classroom settings to detect student’s self-reported attentional states above chance levels. Stewart et al. (2017) found that lip tightening and jaw dropping facial action units seemed to be able to generalize across task contexts (reading a scientific text and watching a narrative film). Bosch (2016) found that texture features, which indicate changes in facial expressions, were the strongest predictors of mind wandering during reading and interacting with an intelligent tutoring system.

Non-instrumental Behavior as an Exploratory State

An interesting line of research indicates that non-instrumental behavior in the form of doodling while performing a boring task actually enhances performance on the primary task (Andrade, 2010). Such “nonessential” behavior (fidgeting, doodling, humming) potentially enhances arousal to levels associated with optimal task performance (Farley et al., 2013; Risko et al., 2013). A similar account of mind wandering has been proposed in the context of Attentional Blink studies, where inducing participants to mind wander actually improved participant performance. If non-instrumental behavior could be indicative of mind wandering, it seems counterintuitive then that performance would be enhanced, especially considering the substantial amount of literature that demonstrates that mind wandering is actually detrimental to performance (Smallwood & Schooler, 2015). In the context of the Attentional Blink, it has been proposed that mind wandering actually helps distribute attention more broadly in the environment (Forster & Lavie, 2014), reflecting an adaptive cognitive style intended to maximize the efficient processing of events (MacLean et al., 2012). Relatedly, it could be that non-instrumental behavior associated with mind wandering is indicative of exploratory off-task states described by Mittner et al. (2016). During such off-task states, attention is dispersed as we broadly scan both our external and internal environments in order to determine if on-task goal directed thinking or mind wandering should be the next state to be exploited. Consequently, we are able to maintain reasonable levels of performance while at the same time mind wandering. Farley et al. (2013) propose that non-instrumental behavior could potentially reflect an attempt to combat waning attention. It could be that fidgeting, doodling, or humming, for example, stabilizes arousal (to optimal levels) in order to facilitate performance on a primary task (Andrade, 2010; Farley et al., 2013; Risko et al., 2013). Alternatively, such behaviors could reflect the transition into a state of inattention and in turn, mind wandering. In both of these explanations, non-instrumental behavior is linked to the presence of inattention, either in order to redirect attention to the task at hand or as the marker of internally directed attention (Farley et al., 2013).

Computationally Modeling Mind-Wandering and Related Body Movements

Clearly, there is a rich body of literature relating mind wandering to bodily behaviors. It seems that mind wandering, when defined as a decoupling of attention from the external environment, is associated with an attenuation in bodily behaviors that are instrumental. Additionally, it could be that more exploratory forms of mind wandering manifest as non-instrumental behaviors which are not entrained to external stimuli, such as fidgeting, doodling, or humming.

Formalizing the dynamics of thought in computational models allows us to directly test hypotheses and theories concerning how mind wandering takes place and manifests in bodily behaviors. Computational models enable a moment-to-moment simulation of the ebb and flow of our thoughts and can help us to understand how a variety of task and cognitive factors affect behavior. They serve as theories to explain how different psychological phenomena work, accounting for complete tasks, starting with perception through to response execution (Borst & Anderson, 2015). The better a simulation fits the actual data, the better the cognitive model. To date, most of the quantitative computational cognitive models of mind wandering have been based on data collected in the SART (with a few exceptions). The Adaptive Control of Thought-Rational (ACT-R) is the most widely used cognitive architecture that computationally models processes from perception to action for a wide range of cognitive tasks (Anderson, 2007). Several computational models have been implemented in the ACT-R to describe fluctuations in states of focused attention and mind wandering during SART (Hiatt & Trafton, 2015; Van Vugt et al., 2015; van Vugt & van der Velde, 2018). Recent studies have successfully combined cognitive modeling with neurophysiological data. For example, Klaproth et al. (2020) used EEG data to inform and constrain their cognitive architecture. In addition, Borst and Anderson (2015) used ACT-R for modeling complex fMRI data. Similarly, integrating data collected from our bodily behaviors with cognitive architectures could serve to provide a more faithful representation of how perception is coupled to (or decoupled from) action.

An alternative to the ACT-R approach is provided by the sequential sampling models (McVay & Kane, 2012; Mittner et al., 2014) which are based on the assumption that sensory information is gradually being accumulated before it reaches a threshold and a decision can be made with respect to the course of action (Forstmann et al., 2016). The sequential sampling models also offer the possibility to account for actions being performed during episodes of mind wandering accompanied by perceptual decoupling, by acting on an autopilot.

In general, the computational models of mind wandering focus on simulated behavior in terms of the trade-off between accuracy and speed. As we have seen throughout the course of this chapter, our behaviors while performing a task can be more complex than reaction times and accuracy alone in that mind wandering is associated not only with a change in magnitude or the variability of any one type of bodily behavior, but rather with a systematic covariation of bodily behaviors (D’Mello et al., 2012).

Conclusion

In the course of this chapter, we have highlighted the impact of mind wandering on the tight link between perception and action. Our overview of existing findings shows that mind wandering may manifest as an attenuation in sensory-motor responses to the environment (e.g., more variable response times, more complex – or more idle hand movements – and reduction in vocal adjustment to context). Moreover, it may be “embodied” through non-instrumental behavior, such as fidgeting and facial expressions, which could be reflective on an exploratory off-task state, serving to determine the next attentional state (“on-task” or “mind wander”). Finally, we discussed the importance of integrating bodily behaviors into computational models of mind wandering in order to better understand both the processes and the consequences of mind wandering in different settings.