1 Introduction

In his 1947 “Film and the New Psychology,” a piece drawing on Gestalt theory, Maurice Merleau-Ponty discussed the Kuleshov effect—a cinematic phenomenon wherein audiences perceive different emotions on faces when identical shots of performers are contextualized differently. Exploring this, I conducted formal and informal experiments, but including performers typically regarded as incapable of altering facial expressions, namely, Darth Vader and non-primates. Varying expressions appeared even here, highlighting the importance of situational context.

Although the importance of situations, gestalts or contextual wholes should be obvious, studies using snapshots of exaggerated expressions cut off from bodies and environments adopt methodologies assuming otherwise. Combined with the insistence of ranking researchers such as Noël Carroll (1993) and Paul Ekman (e.g., Ekman and Friesen 2003) that emotional expression is only somewhat underdetermined by faces, this distracts from the importance of situations. More specifically, it obscures the fact that perceived emotional qualities are not exclusively signs of what people are feeling. Without denying this is sometimes so, I argue that what is critical in expression perception is registering situations in order to deal with them.

Emotions connect to action (Frijda 1986, 2010), whether withdrawing, pausing, advancing or engaging in more specific behaviors such as seeking shelter when threatened or sad. Something comparable holds for expressions in that friendly looks, like extended hands, are openings for engagement, whereas intransigent demeanors are closures. As such, expressions may helpfully be understood in terms of what Gibson (1979) called “affordances.” I locate expression as a subcategory of environmental affordances, and accordingly argue that while perceiving emotion is sometimes about reading minds, it is more directly about reading situations in order to deal with them. This is not entirely new and hinted at in Gestalt theory and Gibson’s work. However, some interesting consequences follow.

First, the position suggests taking contextual wholes and not parts as starting points—a thesis Gibson pointed to, along with Gestalt, phenomenological and pragmatic schools influencing him. If one accordingly recognizes situations as primary, then Kuleshov-like scenarios need not be illusions, here defined uncontroversially as wrongly perceived events. They may instead be gestalt-products that increase sensitivity to what is occurring, thereby facilitating action. Second, Kuleshov-like scenarios can be regarded as instances of more general cases in which emotions infuse contextual wholes and emerge as reflections of interrelationships. Thus what might otherwise appear as a peaceful landscape may become ominous in company with stormy clouds. Angry cloudscapes may likewise become serene when combined with different vistas. Interrelations include the organism, so that perception of any of this may vary with individual situations, with a farmer seeing approaching rain with hope and a lawyer with dreariness—this, however, because they live in worlds having overlapping but still different objective needs. All of this squares roughly with Gibson and some of his predecessors. It is to assert that environments, like faces, have expressive properties—or what Gestalt theorists following Werner (1927) called “physiognomic qualities”—that manifest according to interrelated wholes.

The situational account I advance is more radical than some, albeit in line with Koffka (1935), in that it holds we can sense emotions in environments without feeling them, as when perceiving gloomy landscapes while cheerful. Relatedly, situations—and not just intersubjectivity—are central to the experience of emotional expression, demonstrated again with gloomy landscapes and in many other cases. However, I also go further than Koffka and others in Gestalt traditions by arguing not merely that perception is emotionally infused, but that expressive qualities are affordances and accordingly ontologically in situations as much as other action-guiding attributes. I defend and elaborate these points, while highlighting their relation to affordances, by locating them in aesthetic, perceptual and cognitive research, both philosophical and experimental. Taken together, my arguments are that perception is expressively charged, that expressive qualities are consequences of ecological relations and that they are accordingly in the world and not just in our heads.

2 Faces and masks in context

In “The Art of Cinema,” the Russian filmmaker and theorist Kuleshov (1929) recollected a film in which he saw a portrait of Lenin smiling. Only in this then familiar portrait Lenin was in fact not smiling. “But,” added Kuleshov, “that spot in the film was so funny,” and “so uproariously received by the public, that I, myself, scrutinizing the portrait several times saw… Lenin smiling” (p. 54). Kuleshov went on to say that “[t]he montage was so edited that we involuntarily imbued a serious face with a changed expression characteristic of that playful moment,” and concluded montage can alter the appearance of expression (p. 55).

Kuleshov, in company with the filmmaker and theorist Vsevolod Pudovkin, conducted casual tests confirming this lesson, with the latter later detailing what is now remembered as the “Kuleshov effect.” Pudovkin (1926, p. 168) recounted taking close-ups of the actor Ivan Mosjoukin with an impassive face, and cutting it to shots with a woman in a coffin, bowl of soup and a girl with a toy bear. Though the camera recorded Mosjoukin looking straight ahead, audiences reported him directing his gaze towards the woman, soup and child. They further said that he wore a sorrowful expression, pensive stare and warm smile for the woman, soup and girl respectively, highlighting the Gestalt principle that the whole is different than the sum of individual parts, as Merleau-Ponty (1947) later noted. While this account may be apocryphal (Prince and Hensley 1992; Carroll 1993), with some struggling to replicate it in controlled experiments (Prince and Hensley 1992), recent studies suggest the effect is genuine (Mobbs et al. 2006; Barratt et al. 2016; Calbi et al. 2017). More to the point, the experiment as originally described was simple, and with richer juxtapositions, most experience something comparable to the Kuleshov effect. At least, this is what my own formal and informal experiments indicate.

Some experiments—in this case conducted informally with discussion groups—revolved around non-primates. Such animals of course occasionally display features resembling human expressions, and do so even in isolation of context, as with the sad droop of Basset Hounds (Robinson 2011), or the dour look of the “Grumpy Cat” meme. Such cases aside, it is typically not through resemblance, and much less because cats and dogs contort their faces expressively, that we register emotion. More often it is because of how the animal is situated. Thus when cats look up at mealtime while next to food bowls, many see hopeful eyes, even if questioning whether the emotion is really there. Furthermore, through a kind of auditory Kuleshov effect, we may hear imploring meows that in other contexts manifest as murmurs of affection, just as the same yelps of a dog are in one instance registered as concern and in another as joy at the return of a master.

The original Star Wars films offer comparable examples, this time formally investigated. As with cats, Darth Vader obviously does not expressively contort his mask. Nonetheless, roughly 93% [χ2(2, N = 117) = 188.66, p < .0001] reported experiencing negative emotion on his mask and eyes when, after being defeated by his son, his son begs for help as the Emperor torments him with lightning-like energy; and Vader, long hesitating, finally hurls the Emperor down a shaft.Footnote 1 In the same test, 5% reported neutrality and 2% identified positive emotion.

A more compelling instance is a re-edited sequence—sometimes titled “Darth Vader being a smartass”—that has made the rounds on the internet.Footnote 2 The original sequence features Vader in an office-sized black pod with a white interior. The capsule forms a private space from which he commands, and Vader can open and close it, almost like a clamshell. As an officer arrives, Vader opens the pod, and a short conversation ensues during which the latter voices dissatisfaction. Given a choice between seven expressions—confused, disgusted, fearful, happy, neutral, sad or sarcastic—60% reported neutrality and 27% disgusted [χ2(6, N = 119) = 243.06, p < .0001], with negligible numbers choosing other categories. Given the same choices with the officer, 63% reported fear and 27% neutrality [χ2(6, N = 120) = 271.53, p < .0001], with other categories again negligible. In the re-edited version, Vader, as the titled suggests, behaves like a smartass. He opens the pod, only to close it whenever the officer speaks. At one point he rapidly opens and closes it. Near the end, the officer stares with confusion according to 74% of respondents and disgust for 14% [χ2(6, N = 120) = 365.33, p < .0001], with few selecting other options. These expressions show up as a Kuleshov-like effect since few saw confusion (6%) and disgust (3%) in the original version. Far more interesting, however, is this: a sarcastic expression appears on Vader’s mask for 73% of respondents, compared to about 3% in the original clip, with neutrality dropping to 13% and other categories becoming negligible [χ2(6, N = 120) = 341.77, p < .0001].

In less controlled settings focused on qualitative experiences, I have shown these clips on three continents to people from Europe, North America, Asia and Africa, and consensus is not only stable; it appears stronger, which makes sense since joking happiness, sarcasm, snide disgust and other possible categories overlap, meaning the forced-choice options used in formal experiments likely dilute outcomes. The critical point, in any case, is that what people experience overwhelmingly does not depend on what occurs on isolated faces. This is because Vader wears a mask and varying cuts of the same footage are used for the officer, with perceived expressions nevertheless changing, even when participants know it is the same footage. This suggests perception of emotional expression depends on overall situations. In Gestalt language, one might say it varies with the whole, which precedes and determines how the parts—in this case, expressions on individual faces—show up.

While unusual, the Vader cases typify what occurs in everyday life, a point illustrated by a pair of sequences prepared for a 1964 interview between Alfred Hitchcock and Fletcher Markle.Footnote 3 Here Hitchcock took a shot of himself looking at as yet unidentified subject matter and another of himself smiling. In one instance, he inserted a mother with a baby in between him looking and smiling; and in a second, a young woman in a bikini. “What is he as a character?” asked Hitchcock, referring to the first sequence. “He’s sympathetic.” But “now we’ll put in a piece of film of a girl in a bikini. [The man] looks… He smiles. What is he now? A dirty old man. He’s no longer the benign old gentleman who loves babies.” Tests conducted with the footage but without Hitchcock’s narration and in which participants had five options—dirty, disgusted, friendly, neutral or sad—bear out these conclusions. About 67% identified the smile as friendly in the first instance and 13% as dirty, with other categories below 10% [χ2(4, N = 120) = 164.92, p < .0001]. In the second case, about 94% [χ2(4, N = 120) = 413.17, p < .0001] saw it as dirty, with other options negligible.

Summarizing views Hitchcock related in an interview with the influential filmmaker and critic François Truffaut, psychologist Wallbott (1988) writes “that in a film it is not the actor’s task to produce effects and tension.” Filmmakers, rather, create “contextual information, using editing techniques, which guides a viewer to infer certain attitudes or emotions not immediately evident from the actor’s expressive behaviour itself” (p. 357). Although this accounts for certain cinematic cases, it over-intellectualizes what is happening here, for people report Hitchcock’s smile looks different and not that they are inferring different emotions—and I am convinced those taking time to screen the clip will find the same. Most have comparable experiences when viewing the re-edited clip of Vader. Consequently emotional expression is immediately evident in Hitchcock’s behavior, and for many in Vader’s too, meaning context changes what we see.

Ideas from Gestalt psychology and pragmatist philosophy provide a framework for understanding Kuleshov phenomena. One idea has to do with physiognomic perception, introduced by Werner (1927) and developed extensively by Koffka (1935, esp. Chs. 8, 9, 14), with both attending to fine art, as well as everyday life. By “physiognomic,” Werner did not suggest we see things as face-like because of their geometrical properties. In his own words:

One cannot totally define a face by objective characteristics such as a crooked nose, a facial angle of x degrees, blue eyes, white skin, and so forth. Those are physical or geometric properties. But the face also has physiognomic properties. It is either grumpy or cheerful, energetic or tired, lascivious or pure (Werner 1927, p. 149).

Put another way, seeing face-like qualities in Werner’s sense “does not mean faces (or facial features) in terms of their configuration… or the ability to imagine or see faces in rock or cloud formations,” along lines of resemblance theories (Rosar 1994, p. 156; also see Stern 2010, esp. Ch. 3). This points to a subtle aspect of the experimentally explored Kuleshov scenarios. It does because contexts change how faces look, yet without obvious alterations in angles of facial features—and I again urge readers to screen examples. It is much the same when similar clouds show up as angry or peaceful depending on landscapes accompanying them.

Werner’s theory was a holistic account that took contexts and perceivers as a totality. This position is also central in Gestalt psychology, Gibson’s work, phenomenology and a great deal of pragmatism. Dewey (1938) summarized the outlook nicely when he wrote: “In actual experience, there is never any… isolated singular object or event; an object or event is always a special part, phase, or aspect, of an environing experienced world—a situation” (p. 67). Using language that could be mistaken for that of Gestalt psychologists to articulate what he had held since at least the 1890s, Dewey added that “we never experience nor form judgments about objects or events in isolation, but only in connection with a contextual whole. The latter is what is called a situation” (p. 66). The Kuleshov effect and perception of emotional expression reiterate this.

3 An illusion?

Since the 1960s, a number of prominent psychologists have argued there are basic emotions, each having universal corresponding facial expressions (e.g., Tomkins and McCarter 1964; Izard 1971; Ekman 1972, 1992; Buck 1994; Smith et al. 2005; Jack et al. 2016). Almost using British empiricist language, defenders suggest expressions are built up from discrete Action Units (see Ekman and Friesen 1978) and that basic expressions combine into complex or “compound” ones (e.g., Du et al. 2014). The outlook has mechanistic connotations insofar as it asserts that basic expressions occur involuntarily, so that differences between genuine and feigned expressions can be detected. The most prominent advocate, Paul Ekman, has marketed the view to the security industry. The position, moreover, has been propagated in numerous police dramas, including Lie to Me, a series grounded in Ekman’s work and having a character based on him. Despite the standpoint’s popularity, some critics charge that research methods employed—which funnel responses into limited numbers of options—bias results towards identifying small numbers of basic emotions and expressions (e.g., Russell 1993, 1994, 2006). Recent reviews suggest further that Ekman’s deception detection techniques fair little better than random guessing and prove ineffective in security screening (see Weinberger 2010).

Gestalt psychology—in company with pragmatic and phenomenological work that also influenced Gibson (Reed 1988; Heft 2001; Chemero and Käufer 2016)—offers one explanation as to why: that contextual wholes precede and indeed alter the appearance of isolated parts. The Kuleshov effect highlights this in regard to expressions, and other research likewise draws attention to the importance of context (e.g., Gendron et al. 2013; Calbi et al. 2017). By contrast, Ekman’s work overwhelmingly focuses on isolated faces, occasionally in combination with body postures, but in any case cut off from environments that modulate what we perceive. It is, to be sure, true that Hitchcock and Vader are not completely in charge of what they express, as Ekman maintains, yet not for the reasons he posits since their facial muscles do not involuntarily move. Although this sometimes happens in everyday life (see Damasio 1994, pp. 139–142), it does not occur here since the same shot is used in the case of Hitchcock, and Vader wears a mask, with expressions varying according to context.

While one might want to regard these Kuleshov-like effects as illusions, that is, mistaken perception of phenomena not present, this is unproductive and arguably inaccurate for a number of reasons. First, it may be—and this will be expanded later—that perceiving expression is foremost about reading situations in order to deal with them, as opposed to reading minds per se. Gibson (1979, esp. p. 142) remarked that a cliff-edge affords falling and looks dangerous and in fact is so. Without getting around to imputing intentions, Hitchcock’s smile in the second instance is comparable: it looks predatory and in everyday life might be so, accordingly provoking vigilance or retreat. The clip with Vader behaving as a smartass is comparable. The situation would be exasperating and elicit a variety of courses, whether attacking, fleeing or staying passive in the face of a threatening being. This would remain so even if it turned out Vader was a mindless robot with no mental intentions to read. Understood thus, expressions showing up in Kuleshov-like scenarios perceptually mark what is happening in situations, in addition to which they may sometimes indicate what is occurring in minds.

Second, the word “illusion” suggests we misperceive situations. Yet if encountering Hitchcock in the world, we likely would think people who register different expressions would—with notable exceptions to be discussed—perceive the situations more fully than those who do not. Experiments by Carroll and Russell (1996) help elaborate the point. Specifically, their findings suggest if you see a man wide-eyed in the wee hours, you will likely conclude he is undergoing emotional arousal, yet without grasping what kind. But depending on whether you learn he just received an offensive letter or an obituary notice, you may decide he is angry or grief-stricken. This raises a question: When context moderates expression, can we claim confidence in our perception? Suppose we perceive the lascivious leer of an old man gazing at a bikini-clad teenager. Later, however, we learn he is her grandfather, thereby coming to perceive his smile more charitably. Might we conclude we misperceived the expression? Certainly, but this reaffirms that situations, rather than deluding us, help us register differences otherwise missed. Reinforcing this, Hillel Aviezer and colleague’s (Aviezer 2012a, b; Aviezer et al. 2008) found participants unable to distinguish isolated faces expressing intensely positive experiences from negative ones, for instance, sporting victories versus defeats or orgasms versus nipple piercings. However, context from body language was such that defeated faces were read as elated when placed on victorious bodies, and vice versa. The Hitchcock and especially Vader examples reinforce and extend this conclusion to include environments that ground expressions.

Third, the word “illusion” customarily suggests perception of something not physically registered by sense organs and hence not actually there; yet perceiving what is not received by the senses can in fact mean perceiving qualities that are there. Merleau-Ponty (1945, Pt. 2.3, 1947) illustrated this using the example of color constancy, also cited extensively by Gestalt psychologists. Color constancy refers to the fact, for example, that when yellow candlelight envelops a white tablecloth, we perceive it as white cloth bathed in yellow rather than yellow cloth, even though it chiefly reflects this color. Only when encountering an isolated patch—as when gazing through a hole in a screen—would we perceive it otherwise. Notice, then, that perceiving the white linen means perceiving what is not present in isolated patches of cloth physically received by the senses. Affairs are comparable when we perceive differences between Hitchcock’s smiles. We see what is absent from Hitchcock’s face in isolation, and perceive differences the senses do not physically receive. However, leaving aside the notion that color is a secondary quality, most want to say the linen is white in spite of the fact yellow light chiefly hits the eye. Many would likewise conclude that in actual life situations it is legitimate to perceive the smiles differently. Here emotional expressions—like color and its constancy—appear as effects of an interrelated whole. Such consequences, it is true, only show up in the presence of observers, who are in the interrelationship. However, this is also so with color, as the constancy example shows.

Relational accounts of properties in fact appear remarkably robust. This is so insofar as color and even length and mass vary with velocity relative to observation. Particles likewise show up only through one hitting another. Further, they can appear as waves if one sets up the detecting apparatus from that perspective, or as particles if using equipment designed to register them as such. Thus while it has long been argued that “no sentiment represents what is really in the object,” only marking “a certain conformity or relation betwixt the object and the organs or faculties of the mind” (Hume 1757, p. 208); and while philosophers have reached like conclusions about color, smell, taste and other so-called secondary properties, the foregoing suggests such also holds even with alleged primary properties since they likewise vary with relations and perspectives (Crippen 2010). This idea runs deep in pragmatism, starting with C. S. Peirce’s (1878) maxim about meaning, which holds that our sense of what a thing is depends on possible effects arising out of contact with other entities. Hence a hard object is one not easily scratched by other things. The idea is carried in Dewey’s work too, which stresses the relativity and quantum effects just described. Dewey adds that this is not a peculiarity of extreme physics, but instead characterizes everyday life where we thump and rattle things to better perceive what they are, and if we cannot do this, we alter conditions under which they are observed, as when bending light with lenses (see Dewey 1908, p. 45; Dewey 1920, Ch. 5; 1929, Chs. 4, 8). The take home point is that relations between things and also between us and them often determine their appearances and that this is not sufficient grounds for rejecting them as realities in the world.

For Merleau-Ponty (1947), the Kuleshov effect and comparable worldly scenarios are additional cases of relational qualities, these paralleling perceptual grouping and like phenomena explored by Gestalt psychologists. Also fitting this pattern are experimental findings on size and color constancy that Merleau-Ponty (1945, 1947) and Gestalt theorists (e.g., Koffka 1935) often cited in counter to empiricist theories that hold the experiential field is built out of sensory atoms, like pixels on a TV screen. Melodies, which remain intact with transpositions to new keys that alter individual notes, were another illustration that Merleau-Ponty (1945, 1947) deployed, again following his Gestalt contemporaries (e.g., Koffka 1935, Ch. 10; Wertheimer 1938). Kuleshov-like scenarios fit this template, which is to say, they are products of the interrelated whole and vary according to it. Much else—as just argued—is similar, and an alternative to rejecting such as illusory is to conclude relational contexts are primary. They are where things and events with specifiable qualities initially and actually show up. On these grounds, one can build a case that contextually modulated expressive qualities, like the range of others discussed, are actually in arrangements that make up our worlds.

4 Physiognomy in spaces and faces

In his last and most recognized book, Gibson (1979) detailed how Gestalt and closely associated theorists influenced his thought. He credited Kurt Lewin’s notion of Aufforderungscharakter, a term variously translated as invitation or demand character and appropriated by Gestalt thinkers. Morphologically and conceptually similar to Gibson’s concept of “affordances,” it also resonates with pragmatic thinking, which Gibson likewise credited. Quoting from Koffka (1935), who extensively appropriated the concept, Gibson explained demand characters in the following way:

“Each thing says what it is… a fruit says ‘Eat me’; water says ‘Drink me’; thunder says ‘Fear me’; and woman says ‘Love me.’” These values are vivid and essential features of the experience itself. The postbox “invites: the mailing of a letter, the handle “wants to be grasped,” and things “tell us what to do with them.” Hence, they have what Koffka called “demand character” (p. 138).

A second idea discussed in these same portions is the concept of physiognomic qualities. Customarily such characterizes faces, but Werner and his Gestalt contemporaries also applied it to environmental objects. My view expands on theirs, adding that we read human faces like environmental spaces, namely, as affordances grounded in relations.

That environments and things in them have physiognomic or expressive qualities such that we can read them like faces was central to Gestalt work and in the background of Gibson’s thought as well, which is why he spent time condensing the view. As Gibson (1979) wrote:

The gestalt psychologists recognized that the meaning or the value of a thing seems to be perceived just as immediately as its color. The value is clear on the face of it, as we say, and thus it has a physiognomic quality in the way that the emotions of a man appear on his face (p. 138).

Although physiognomic and demand characters overlap and can be one, they differ insofar as the latter link to temporary need, with the former independent of it, as when clouds or faces look angry regardless of vested interest. Both are consonant with Gibson’s notion of affordances, but with physiognomic qualities more so. This is because Gibson insisted that while we might notice openings and closures according to needs, they remain independent of vested interest. They do because they are objective consequences of what bodily structures allow or disallow in given settings (see Gibson 1979, Ch. 8). Demand and physiognomic characters, however, equally point to the valuative nature of human–environment contact, and this is key for Gibson.

Elaborating on his own views, Gibson (1979) urged that the perceptual field of affordances is not a value free world on which meaning is superimposed. It is, as he put it, “a process of perceiving … value-rich ecological object[s]. Any substance, any surface, any layout has some affordance for benefit or injury to someone” (p. 140). Thus while suggesting differences between affordances and demand characters, Gibson acknowledged that biological needs—and one presumes accompanying interests and desires—heighten valuative awareness, as pragmatists such as James (e.g., 1879) also emphasized (also see Hufendiek 2017). That said, the externalist bent of the position remains since pragmatists also stressed, in company with Merleau-Ponty (1945, 1947), that needs and desires only have meaning in worldly contexts. James, to expand a little, argued that our waking attitude is always one of wanting to do, get or avoid something and therefore one of concerned interest. Thus, to employ one of James’s illustrations, a mechanic values oil as a lubricant, a carpenter as a wood darkener. In other words, what the oil is for us varies with our interest or aims. At the same time, oil continues to be a lubricant and darkener, irrespective of our desires, just as an affordance remains, even if we are uninterested in exploiting it (Gibson 1979, Ch. 8). It is worth adding that these properties, like others discussed, are relational, so that oil shows up as a lubricant because of the way it allows metal parts or greasy fingers to more easily slide over one another.

Valuative notions of environmental perception recur in Rachel and Stephen Kaplan’s experimental research on aesthetics, with some of it oriented by Gibson and James’s ideas (Kaplan 1987; Kaplan and Kaplan 1989). The Kaplans specifically propose aesthetic experience reflects rapid and pre-reflective appraisals of actions afforded by settings, comparable to assessments made about people at only a glance. One repeatedly confirmed finding is that people like settings with trails disappearing around bends. Such scenes have mystery, that is, allure arising from things partially hidden or unclear, with the promise of something new soliciting deeper penetration. This illustration in fact exemplifies physiognomic characters and by extension Kuleshov scenarios. To begin with, Werner (1927) said that such “qualities are not feelings but, rather, are perceptions” (p. 149), and the word “mystery,” as Rosar (1994) writes, “is customarily reserved for other people and objects, not oneself” (p. 155). Thus while mysteriousness might emotionally attract us, the quality is outwardly observed, whether in a setting or an enigmatic smile. A second parallel is the relational underpinnings of such qualities, their grounding in gestalts or contextual wholes, this connecting back to Kuleshov scenarios. Thus it is rarely a smile alone that makes for mysteriousness, but instead a total setting that may include a tuxedo clad Bond-like character, cigarette in hand, nonchalantly hovering next to a polished wood bar, with jazz pattering in the background. Likewise, winding trails alone do not heighten mystery, nor do forests by themselves. Rather, the quality emerges from the combination of such elements that introduce both navigability and something out of view. Other environmental physiognomic qualities are similar, so that a seascape may be tranquil or grim depending on the cloud backdrop. Weeping willows and suchlike can be exceptions, but it is plausible that surrounding context alters even the physiognomic characters of these.

All of this fits with the thesis that while perceiving emotional expression can be about registering outward signs of inward states, it is more squarely about perceiving situations. This is such that we have an immediate sense of obstacles and openings, and how we might negotiate them. This is obviously so in environmental settings such as those described by the Kaplans, but also for facial expressions. Like affordances, moreover, facial and environmental physiognomies remain independently of whether we are pulled to them, pushed away or neutral. Thus with an old man leering at a bikini clad teenager, the dirty smile remains whether or not we are its object. This is just as hope continues shining in the eyes of a cat by a bowl, even if we are not the one to feed it, and that we likewise see the mysteriousness of a setting regardless of our intention to explore it. Were we to witness a being, even a mindless robot, who slammed doors at every initiation of conversation, we would see an angry-making situation, and would grasp that were we in it, we would wish to escape. The sarcasm and befuddlement most see on Vader and the officer’s face are markers of this.

Expressions on faces, then, compare to emotional qualities in environments, but not necessarily because the latter derives from and approximates the former, as resemblance or contour theories posit (Robinson 2011; Trivedi 2011; also see Davies 2006). To be sure, this sometimes occurs, as when weeping willows look sad because they resemble drooped, demoralized bodies. However, we often read faces like environmental affordances, so that hostile expressions may close further engagement, while friendly ones may open it, all this modulated by context, as the Kuleshov effect shows. Mysterious smiles and settings likewise can be understood in terms of affordances because both suggest something worthwhile is hidden, thereby inviting approach, exploration and deeper penetration.

5 Emotion-laden worlds

The overall picture advanced so far supports the view that situations are epistemologically and ontologically primary. That is, they are where things with observable and hence knowable properties first show up and exist. This makes emotional qualities perceived in situations not so different from properties registered through traditional perceptual modalities. In this last section, I aim to cement this outlook. The account offered not only challenges the idea that emotional perception is wholly subjective. It also counters the notion that it necessarily arises through sensing what others are feeling and that we must feel emotional qualities that we perceive. The account offered has the additional advantage of highlighting possible complementarity between internalist and externalist theories of emotion, rather than simply repudiating the former.

Jesse Prinz and Charles Nussbaum may be taken as respective representatives of internalism and externalism. In his older work, Prinz (2004) argues that emotion perception involves sensing internal bodily changes, which may in turn be reactions to the world. He asserts further that emotions such as rage are unlike color perception because we project the latter into environments, whereas we experience the former within us (p. 61). Nussbaum (2007) responds by stating: “I think this description of the phenomenology of emotional experience is just mistaken. Rage is not experienced as a state within us, but is projected onto the offending object, which is perceived as hateful” (p. 193). Leaving aside whether it makes sense to speak of visual or emotional consciousness as projected, Werner’s (1927) work on physiognomic qualities goes some way to resolving opposition. It does so by highlighting that language obscures distinctions between internally felt emotions and externally perceived physiognomic expressions. That said, emotions are sometimes simultaneously felt and perceived, and it is clear that experiencing rage in an offending object does not preclude internal feelings and vice versa. Feelings are unequivocally there, but the visceral stirrings and impulses we undergo only make sense and properly become rage when located in environing circumstances, without which there is little to distinguish what the emotion is (see Merleau-Ponty 1947; Barrett 2014).

This is so in everyday life where perception, emotion, action, cognition and world tend to fuse. Thus in the context of a precipice, a river might appear treacherous and we might experience the trepidation that accompanies danger. Notice that this simultaneously means seeing the river in terms of actions we might take and their effects on us, and hence in terms of use-value and consequently interests in James’s language, or affordances in Gibson’s, or again concernful attitudes in phenomenological parlance (see Ratcliffe 2002). Individual motivations likewise alter perception. If thirsty, drinkability of water becomes salient; if dirty, it may stand out as a solvent, just as a rock may be a paperweight, hammer, missile or pendulum base in different circumstances (Koffka 1935, Ch. 9; Gibson 1979, Ch. 8). This is why Lewin observed that “a person’s world undergoes a fundamental change when his fundamental aims are changed” (quoted in Koffka 1935, p. 345). Experiments confirm this last point, with studies showing cigarettes look longer to deprived smokers, glasses of water taller to the thirsty, and tools such as shovels larger if experiencing positive emotions towards related tasks (Brendl et al. 2003; Veltkamp et al. 2008). This suggests desires and motivations enhance openings, with emotional tones of the world modulating accordingly (see Ratcliffe and Broome, forthcoming).

Dewey (1934) approached this position when insisting that we do not “project emotions into … objects experienced.” “Nature is kind and hateful, bland and morose, irritating and comforting, long before she is mathematically qualified or even congeries of ‘secondary’ qualities” (p. 16). As Dewey elaborated:

Even such words as long and short, solid and hollow, still carry to all, but those who are intellectually specialized, a moral and emotional connotation. The dictionary will inform any one who consults it that the early use of words like sweet and bitter was not to denote qualities of sense as such but to discriminate things as favorable and hostile (p. 16; also see Köhler 1947, Ch. 10).

Approaching what is favorable and avoiding what is hostile is of course not unique to human life, but extends even to brainless creatures such as the unicellular Physarum polycephalum. By withdrawing from slime trails externally marking where they have been in ways reminiscent of the extended mind thesis (see Clark and Chalmers 1998; Clark 2008), these organisms successfully traverse mazes set up by experimenters; they also select from nutrient options and avoid harmful substances (e.g., Dussutour et al. 2010; Reid et al. 2012). In such ways, they negotiate what may be regarded as primordial gustatory and chemical affordances.

Koffka (1935) in fact regarded physiognomic perception as primordial in the sense that it prevails more in early development and pre-scientific attitudes, noting that the gruesome awe of corpses are lost to medical students who dissect them by the score (p. 359). His analysis further suggests such properties are phenomenologically of the world. Thus we intelligibly say that “we … see a gloomy landscape,” and do this “even when … perfectly cheerful” (p. 326). A book may look “proud, a young birch shy, and has not Wordsworth immortalized the glee of the daffodils” (p. 326; also see Werner 1948, p. 69; Frijda, p. 188). For reasons inherent in Gestalt psychology, Koffka should have elaborated on how physiognomic attributes of these entities can vary depending on situations. Kuleshov scenarios and the earlier example of mysterious settings are affirmations, and suggest that situations modulate physiognomic perception. In line with this, we commonly describe situations as depressing, intolerable, threatening or triumphant (Dewey 1934, p. 67). Likewise, we talk more of being in moods than them being in us (Dreyfus, 1991, p. 171), and sometimes this means being in public emotional atmospheres (see Böhme 1993). This may be so even when nobody is in the scene, as when joyfulness or coziness manifest through juxtaposition of mountains and sunshine or settings with a fir, garlands and wood fire. This suggests emotional attributes of settings are more than aggregates of what we presume individuals are feeling, in addition to highlighting their relational character.

A number of more recent commentators capture some, but not all, of this. Schmitz et al. (2011) elaborate on the public character of certain emotions in their introduction to Hermann Schmitz’s “Emotions Outside the Box.” They note shame, grief and anger “seem to possess a room-filling authority that regularly affects or ‘grips’ even total strangers” (p. 246) such that “situation and feeling are inextricable” (Slaby 2014, p. 36). Yet notice once more that—similarly to perceiving affordances—we can perceive emotional qualities in situations without becoming them, as when sensing merriment while sad or gloominess when cheerful. Something comparable occurs with art. Thus while Prinz (2004, p. 235) expresses awe at how good composers send listeners into an abyss with combinations of notes harmonizing into minor chords, more remarkable is that we can hear exultation in Beethoven’s Ode to Joy theme when blue, and sense anguish in Pergolesi’s Stabat Mater when happy. At the same time, these and many other artistic compositions can offer openings for experiencing emotions they express (see Krueger 2010). But regardless of whether we feel them, we continue to perceive these emotional timbres in the compositions, this paralleling what has been said about physiognomic qualities and affordances. The same holds when perceiving expressive qualities such as grief on faces that may open us to sadness, but that we can also register with or without feeling the emotion perceived.

The last claim does not need defending, but it points to one introduced at the outset that does, namely, that expressive qualities of environments are no more subjective than affordances. Dewey’s (1934) ideas about perception, particularly of art, help elaborate this and specifically why it is misguided to dismiss expressive characters as necessarily subjective or flat out illusions. He urged that subject and object are built up in one and the same continuing operation (p. 177). This anticipates Gibson’s (1979, p. 129) claim that affordances are equally about the environment and behavior, physical and psychological. As Dewey expressed it in a mix of rationalist and empiricist terminology, perception is an “act of the going out … in order to receive” (p. 53). Thus when fingers go out and caress a desktop, smoothness is realized as a perceptual effect. His approach to art proceeded similarly: It is not a photograph or painting “as a picture that causes certain aesthetic effects ‘in us.’ The picture is itself a total effect brought about by the interaction of external and organic causes” (p. 250). This means the picture only exists in relation to entities having capacities to see it as such; and “its ‘beauty,’ which… in being an intrinsic part of the total effect, belongs to the picture just as much as … the rest of its properties” (p. 251). Recognizing the expressiveness of aesthetic characteristics, an art critic might describe an Ansell Adams photograph as having a gloomy mountainscape, shy birches, turbulent clouds and so on; and for reasons just mentioned might conclude these emotional attributes, as parts of the total effect, are no less an objective outcome than the picture itself. If encountering the expressive vista in person, we might say the same still.

Gibson (1975) summed this up a half-century later by asserting that aesthetic experience is not perception plus emotion, feeling, significance and beauty; “it is just perception, not plus anything” (p. 321). For Gibson, moreover, perception is not a subjective state. Thus it is no mere subjective difference that separates a cat’s experience of digging into lacquered would with claws and a human’s experience of caressing it with fingertips. Nor is it a mental difference that makes water walkable for some insects, but not for us. These are differences in action, realizing different objective effects in the world and hence in the perception of organisms insofar as they have it (Crippen 20162017, esp. pp. 126–127). The point is empirically supported by affordance studies showing that longer arms make things appear closer (Proffitt and Linkenauger 2013), the perceptual difference indicative of varying capacities, habits and hence potential actions. For like reasons, the claim is reinforced by experiments finding that people perceive grades as steeper or farther away when fatigued, burdened with backpacks, suffering low blood sugar or in poor health (Bhalla and Proffitt 1999; Proffitt 2006; Schnall et al. 2010; Zadra et al. 2010). Insofar as cats and individual humans are endowed with different possibilities of action in a given environment, we might further say they inhabit objectively different worlds—“worlds” here defined as intersections of behavior and environment.

Emotional timbres likewise can reflect differences in worlds. Deflation may infuse perceived steepness or distance. So too may elation and diminishment of perceived obstacles if one is a fit and enthusiastic hiker. Standard interpretations would take this as evidence of the non-veridical nature of perception. However, these spatial-emotional modulations correspond to the difficulty or ease of navigating (see Dewey 1934, p. 209), that is, from our capacity to do certain things. In line with this, depressed, lethargic moods turn out to correlate with aesthetic preferences for enclosed and hence protected spaces (Mealey and Theis 1995; also see Krueger and Colombetti 2018). Hills likewise appear steeper when sad (Riener et al. 2011), which typically means less energetic. Conversely, energetic moods engender predilections for open and therefore explorable spaces (Mealey and Theis 1995). Here moods and emotions delineate worlds according to objective possibilities of action, just as arm length or the presence or absence of claws does. Aesthetic and emotional perception reflects this, altering according to the ease or difficulty with which we can act, our vulnerability at a given moment—in short, according to differences in our worlds.

So while emotions help us to navigate the social world, there are worlds beyond with which they also help. Our dealings with nearly everything involve likings, dislikings, preferences, mysterious pulls and more, whether reflected on faces or in mountain vistas. How exactly this plays out varies with situations. Hence when suffering bereavement, we enter a different emotional space than before the death, just as farmers inhabit different emotional worlds than lawyers. Heat and rain have emotional import and hence values or meanings for farmers that is lacking for most lawyers, whereas family becomes particularly meaningful in the world of one bereft of a beloved grandparent. The emotional worlds of the farmer, lawyer and grieving person accordingly vary in no small part because they come with different pushes and pulls. Yet their emotional worlds also overlap out of being pushed and pulled similarly, experiencing common threats and hopes and consequently needing to do similar things.

6 Conclusion

I began this article by considering the Kuleshov effect and the primacy of situations in expression perception. Next I argued Kuleshov-like scenarios reflect relationships in the world, as do many other properties, and that such is not grounds for deeming any of this illusory. I then compared the physiognomies of faces and environments, arguing both fall within the range of Gibson’s affordances. I finished by framing environmental emotional characters in terms of affordances, or in Dewey’s terminology, as qualities of interactions in which both extra-organic things and organisms partake. I will review what has been affirmed.

One point is that perception, action, cognition and indeed the world are emotionally laden. While most accept some version of this, commentators typically discuss objects of disgust, rage and so forth, as opposed to expressive, physiognomic qualities. Some, moreover, emphasize that perceiving emotional qualities in environments is a byproduct of social perception, as when weeping willows evoke demoralized postures. Few recognize the reverse—that, to repeat an example, mysterious smiles may be like mysterious environments, not to mention mysterious artworks, in that all are total effects promising something worthwhile while not revealing too much, therewith pulling us into them. This does not mean faces do not convey a lot of information; they do, but typically in the context of situations; and moreover, as the Kuleshov effect demonstrates, along with studies showing people cannot differentiate orgasmic ecstasy from agony when encountering isolated faces, situations are primary. That reading emotion is about reading situations and not just minds fits also with an evolutionary picture in which advanced cognition and knowledge of other minds developed recently, and where it is more important to see a situation with rampaging lions or people as fearsome and hence dangerous than impute mental intention to them. The same goes for a man glaring at bikini-clad teenagers, whom we see as predatory.

The second-generation Gestalt theorist Arnheim (1971) emphasized much of this. Summarizing his teacher Wolfgang Köhler’s views (e.g., 1946, Ch. 10), he noted that his predecessor held “that people normally respond to external behavior in itself, rather than thinking of it explicitly as a mere reflection of mental attitudes” (p. 451). Arnheim further stressed the primacy of physiognomic qualities in environments, noting that the “senses are not self-contained recording devices” and that the “organism is primarily interested in the forces that are active around it—their place, strength, direction” (p. 430). He went on to say: “Hostility and friendliness are attributes of forces. And the perceived impact of forces makes for what we call expression” (p. 430). Of faces, Arnheim added that we are more sensitive to alertness, tenseness and other expressions than to such geometric properties as triangular shape. As shown, however, context changes the expressions we see. Moreover, since faces and other expressive objects are not encountered over and against situations, but in them, they also alter what we perceive. A fan smiling and cheering at a hard earned goal makes for a very different situation than one standing in shocked silence at the same climactic moment. This is just as similar clouds have different physiognomic qualities when set against different landscapes, and likewise with trails winding through trees versus open space.

These observations stand as yet additional repudiations of the view that our world is built up of basic bits—an outlook developed by British empiricists and maintained in various ways by behaviorists, Ekman and likeminded researchers. Gibson’s work challenges this thesis, as do Gestalt, pragmatic and phenomenological interpretations. When it comes to expressiveness of faces and postures, this indicates that situational context enhances perception. The same holds elsewhere, as in color constancy, or when perceiving mystery as a total integration of trees, trails and other things. Taken together, this suggests that expressive and indeed other qualities show up as consequences of ecological relations. It suggests further that perception is emotionally infused and that it is misguided to reject expressive qualities in environments or Kuleshov-like scenarios as projections from our heads.