Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Human Computer Interaction (HCI) and Interactive Media

Media, at it’s core, being a manifestation and simultaneously a constituent of culture, is often looked at solely from the perspective of being a cultural artefact. Understanding interactive media from an HCI perspective, where context of use is simultaneously as important as culture could be useful. In HCI the interpretation of the function of a control object does have cultural dependency, while simultaneously, often having a physical dependency. Although interactive media, and especially ‘e-media’ has relied mostly on cultural interpretation, there is a growing need to understand physical representation of meaning within an interactive framework. Hopefully the ideas in this chapter can help structure one perspective on how this may happen.

A paper at a conference in Australia entitled, Making sense of Electronic Communication: Media Richness, Media Genre, and Marshall McLuhan [13] presents an interesting comparison of two of the more common media theories in the Information Systems (IS) literature, namely media genre and media richness, with a third media theory developed by Marshall McLuhan that is less common. Connecting this work to the field of HCI in relation to affordance and mapping, is a useful exercise, as it allows us to apply communication theory in a field where people tend to focus on the ‘tangible’ technologies and physiological effects; however, at the core of the interaction between people and technology is communication, whether it be physical or conceptual. Holding a well designed hammer ‘tells’ you that it is comfortable and designed to be held the way it sits in your hand. Seeing a bright red button ‘communicates’ more significant consequence than the other, not so colourful, buttons.

Media is a very broad term, referring to any and all modes of communication, ‘electric media’ has been described as having a structural vacuum [10], as the formation of the message within the carrying technologies encodes no structure in and of itself. Individuals seek meaning of the message from clues within how the media has appeared to have been structured. As we move toward media as being interactive and embedded with cues and objects for manipulation, we move away from an understanding of media as being something for consumption or interpretation, toward media that is more deeply engaged with as part of a dialogue of interaction. Individuals then, have to interpret their engagement methods in relation to what they perceive in the message. Within HCI, the concepts of context and culture dually dictate the ways in which the individual makes sense of the affordance of an artefact, and rely on familiarity and enculturement respectively. These terms could apply to interactive media and are discussed later in the chapter.

2 McLuhan

Marshall McLuhan passed away well before the proliferation of computer technology; however, he was present at the dawn of our electronic awakening and has made some comments about television and other communication technologies that could be highly related to our current technological challenges, although McLuhan was of the ‘electric’ age rather than the electronic one. McLuhan notes that our thinking about technology must change in order to understand ‘electric’ media, and one might equally extend this to electronic media, based on the content of what he said. Following on from a discussion of weapons as an extension of our body in order to illustrate our changing ways of thinking McLuhan says,

Our highly literate societies are at a loss as they encounter the new structures of opinion and feeling that result from instant and global communication. They are still in the grip of “points of view” and of habits of dealing with things one at a time. Such habits are quite crippling in any electric structure of information movement, yet they could be controlled if we recognized whence they had been acquired. But literate society thinks of its artificial visual bias as a thing natural and innate [9: 342].

And McLuhan didn’t have the benefit of Twitter or Facebook on which to form these ideas, merely television and radio. On the one hand McLuhan questions the visual bias that society has developed as opposed to acoustic. McLuhan posits that western phonetic language sits in a left brain dominance of linear thinking rather than spatial cognition, whereas the suggestion is that a different hemispherical bias may be prominent in new media; however, he also suggests that electric communication is without structure, in that;

… there is a component in the automation story that is as basic as tactility to the TV image. It is the fact that, in any automatic machine, or galaxy of machines and functions, the generation and transmission of power is quite separate from the work operation that uses the power. The same is true in all servo-mechanist structures that involve feedback. … Electricity has brought a strange elasticity in this matter, much as light itself illuminates a total field and does not dictate what shall be done. The same light can make possible a multiplicity of tasks, just as with electric power. Light is a nonspecialist kind of energy or power that is identical with information and knowledge. Such is also the relation of electricity to automation, since both energy and information can be applied in a great variety of ways [9: 350].

In an emergent media based interactive environment, such confining or pre arrangement that has been prominent in computer interaction, through traditional scripted software, is often not appropriate. Media genre do not exist in the media channel, but in the manifestation of culturally structured media instantiations, and this may be problematic in ad hoc interactive media methods.

Some of the ideas that came out of McLuhan’s rethinking of media theory was a chart associated with different media, essentially similar to Fig. 1.

Fig. 1
figure 1

McLuhan’s grid, from a description in Marchand [8]

Full size image

This grid was experimental and incorporated many of McLuhan’s ideas including, apparently, Frances Bacon’s vestigia communis, [8:150], meaning that our human senses have a vestige of commonality, sculptors gain tactility from vision, musicians can see colour in music, and dancers can create ‘poetry’ in motion. Unfortunately there is not a comprehensive description of the full relationships between the quadrants as McLuhan apparently never included this in a publication; however, what does exist appears to be tantalisingly significant.

Sensory closure relates to the separation of sensory impression with sensory effect, whilst structural impact was not wholly explained it seems to relate to social consequence, social objectification, or structuration, being essentially an interpretation within a culture. The terms and the chart suggest that subjective completion occurs at a learned perceptive level rather than at a contemplative level, where structural impact is the link to action, intentions, and consequence. It is here, with these two concepts that we can imagine a connection with the strong cultural and contextual elements of affordance. If we follow McLuhan’s thinking we can imagine that sensory closure, subjective completion, and subsequently structural impact relate to the process arising from the physical stimuli through to the socially constructed implications of interactive media or the affordances of tangible artefacts.

To McLuhan the most resounding part of this chart was the distinction between high definition and low definition media [8:159]. The connection to media richness theory is strong; however, where media richness theory associates a level of ability to reduce uncertainty and carry social ‘language’, McLuhans level of ‘definition’ is associated with the quality of the evoked image and the amount of processing required by the eye, ear, mouth, body, and importantly the brain, in order to understand the intent. From this perspective, plain text which is considered to be a ‘lean’ media, or low in social presence, would also probably be considered to be ‘low definition’ by McLuhan and therefore require a larger amount of processing or transformation to ‘find’ the social information—rather than not containing the social information in the first instance. So McLuhan would suggest that there is a higher emotional engagement in a low definition/lean media as there is a greater higher function cognitive involvement in the interpretation. On the other hand there is also likely to be higher equivocality as there is a greater subject dependent interpretation. Therefore, a traditional typed novel can be emotionally engaging whilst a high definition movie may not be, a richer media does not equate to greater engagement.

3 Context

When it comes to non-interactive media, context—or context of use—is mostly subjugated by the cultural influences of the media. That is, when the media is populated by a genre of education or information exchange, or is purely for entertainment purposes, then the context of use is much less important in the analysis of the media than the culturally determined message it delivers. If the media was used for instructional purposes such as a ‘how to’ then we may see an incipient context of use influence. As media becomes instrumental in the interaction with a system, then context of use becomes apparent, and related to the activities of the users of the system.

Here, context will be described in relation to affordances. Context, or context of use describes the manifest physical work or activity that the interaction is involved in. Context, then, is apparent at the physical and the physiological level of the interaction. However, it is more often the case that context is prominent in the interaction of physical rather than media artefacts, although in theory both contribute to the constitution of the affordances, and if we interpret context and culture with McLuhans grid, as discussed below, they contribute to our understanding in sequence.

4 Culture

Culture is synonymous with communication, and most obvious in the concept of media and media genre. Culture could be thought of as being the knowledge a person needs to attain in order to operate successfully in a specific social group, whether that be an; ethnic, professional, occupational, organisational, or special interest group for example. Culture is not purely explicit language, it is also the combination of symbols and behaviours associated with that social group that have meaning to the other members of the group, including the use of, and engagement in, technology. Therefore, culture is most obviously important in the understanding of interactive media.

5 Affordance

Affordance has quite a history; with Gibson [5] given some credit for the original concept, and Norman [11] the popularity in HCI. Gibson promoted the ecological theory of affordance as part of social theory, later referred to in HCI as real affordance. Here, there is the concept of affordance being inter-subjective and inherent in material artefacts or natural objects.

Affordance has been taken further by other authors in the HCI area, such as Gaver [4], Hartson [6], and Turner [12]. Hartson’s [6] definitions of ‘cognitive, physical, sensory, and functional affordances’ have been widely cited, and constitute some theory development from Norman, and some clarity between ‘real’ and ‘perceived’ affordances. However, the physical, ecological, and task focus of affordance theorising still dominate and it has been suggested it could be re-grounded using a ‘socio-cultural framework’ [7], and that Gibson’s theory of affordance still needs to be extended to include such phenomena as ‘communication, culture, and context’ [7:967].

Affordance being synonymous with context [12] is a common perspective, and familiarity with the context being necessary for the understanding of affordance cues. At one end of a continuum of affordance presented in this paper, being an extension of the work in Wellington [13], you have the context (Fig. 2), this is the physical reality that represents human activity and artefacts and that dominates the literature on affordance; however, culture is not context, it is not the physical ecological view that sits well with task analysis, or work flow, it is the virtual socially constructed set of social objectifications (Berger and Luckman [2] that allow us to navigate in a cultural environment. Indeed, culture at its core has meaning and is propagated through communication, and hence the human interaction with the system, comprising of communication artefacts, is significantly cultural. There are references to affordance as communication [6:319, 1:136], but seldom are there references to communication theory. Considering that artefacts in HCI often have text labels and symbolism it is a wonder that there is not a greater connection to general language and communication theory and semiotics in the HCI literature. Communication is synonymous with culture, and indeed we can find reference to culture in the affordances literature, although it’s presentation is mixed.

Fig. 2
figure 2

A unified theory of affordance and mapping

Full size image

If we put culture at the other end of the continuum, as shown in Fig. 2, then we have culture (language/semiotics etc.) at one end and context (physical reality) at the other. Culture and context provide two useful points between which a range of research and development strategies into affordance and mapping can be designed.

Affordance theory suggests that affordance cues come from contextually and cultural sources simultaneously. Hartson [6:322] says “To put Gibson’s ecological view in HCI terms, affordances have a relational ontology: their existence as an affordance is relative to the environment of users and usage. In HCI, the user’s environment is the work context plus the interaction design.” Indeed culture and context may not be practicably mutually exclusive, but conceptually able to be discriminated. More often than not, culture and context are bundled together or confused in the literature on affordances, and often elsewhere in HCI and Information Systems (IS) Literature.

As suggested above, context and culture are replete in the literature about affordance. There have been some attempts to formulate the relationship between them, such as a layered model (Turner and Turner 2002 cited in [12]), but generally they are considered as being separate influences, and some times part of a classification scheme of mutually exclusive characteristics, such as Fragoso et al. [3] representational, technical, and social affordances. More often culture is considered as a mediator to the way that ‘real’ affordance is perceived, and although in hindsight this has seemed to be a naïve perspective, when contemplating McLuhan’s grid there may be some substance to this relationship.

McLuhan’s grid has two stages that intermediate the consumption and interpretation of these cues; sensory closure, and structural impact. These in turn are effected by the relative level of definition of the media. McLuhan also uses the concept of subjective completion, that is not wholly explained but infers the transition between sensory closure and structural impact. As one matches up aspects of affordance with McLuhan’s grid it helps to understand the use of common interaction conventions.

A volume control for example either rotates clockwise to increase volume, or slides up. If you ask a roomful of people why, they indicate that as the volume increases it goes ‘up’, and that is the most sensible and self referential explanation. Higher is more. Similarly, concepts such as ‘forward in time’ or ‘turning on a tap’ are so strongly embedded in our culture that they appear to be contextual rather than cultural. When you realise that in China there are vertical references to past and future, before and after noon is above and below noon, or if you go to a Muslim country and find they use a left hand thread on soda bottles, you begin to realise the arbitrariness of many of our deeply embedded cultural norms. McLuhan’s grid can allow us to investigate the points at which affordance cues are processed in our navigation of the world, and the implications of changing them.

Sensory closure can be considered to be at the physical/physiological end of the process, where such phenomena as gestalt theory are relevant. Whereas structural impact implies greater cultural significance in interpretation. Physical contextual cues will be subject to some subjective completion, variance in levels of effort of interpretation in sensory closure are likely to impact on subconscious cognitive processes, most of which will not have much impact on emotional and attitudinal associations. Although there are some specific known exceptions to this, for example the difference in minor and major scales in music. In affordance theory, the ecological perspective of Gibson most directly relates to sensory closure, being objective and being of physical properties of things—objective and inherent.

Structural impact on the other hand is responsible for a great deal more emotional and attitudinal variance of interpretation. Learned and habituated response to sensory cues, and the cultural significances of them relate to Norman’s perceived affordance and to the body of work of social constructionism. Far from being totally subjective, structural impact relates to social objectifications. However, in the affordance literature these extremes are discussed as opposing theories, but in McLuhan’s grid they are stages within a process. In this theory of affordance and mapping below they are treated as being dually influential in the understanding of what a control does, but the continuum distinguishes the range of influence within the process.

The impact of culture and communication on affordance, especially in computing and in media is very important. Taking two different cultures for example, encapsulated in New Zealand English, and Mandarin Chinese can give us some perspective on McLuhans grid. Mandarin Chinese is spoken and written, ostensibly in quite different forms. There are two forms of intermediate versions of Mandarin that incorporate phonetic or tonal properties; Pin yin, and Zhuyin fuhao, both of which were invented for foreigners struggling with the complex relationship between written and spoken Chinese. The way that humans utilise these very different language systems has a very large effect on the way that the human brain works in interacting with technology. Translation between languages is completely imperfect and evidence of the significance of the effect of culture on interaction. Having struggled with an air-conditioning remote control in a Chinese hotel, and having a dictionary and a beginners level of Mandarin, I can most adamantly say that a direct translation of the characters, the symbols, and the functionality was not possible.

6 Mapping

Donald Norman [11] talked about mapping in his seminal book “The Design of Everyday Things” and the concept occurs often in text books on HCI. However, the references to mapping are often quite simple, and the authors will often say that there needs to be ‘good mapping’ or ‘natural mapping’. Wikipedia even uses Norman’s example of a stove top from the book. The problem then arises as to how do you define good mapping and what sorts of issues can there be with it, when the treatment in the literature is so simplistic. In comparison with other HCI phenomena there has been very little theory on mapping. Affordance has seen more attention in the literature, with; Gibson, Norman, Hartson, and many others all contributing to a better understanding of it. Prima facie, a theory of mapping cannot ignore the concepts of affordance, as it is this theory that helps us understand how someone might perceive that there is any mapping in a control at all. Similarly, one could argue that mapping is perceived in the interpretation of the control in relation to it’s function, which would naturally be defined as affordance. In this way we could very easily conceive of mapping as being a characteristic of affordance.

The model of mapping here is constructed using the terms already used to describe it in the literature. Norman refers to ‘arbitrary’ mapping, and ‘natural’ mapping, and often the term that crops up frequently is the ‘relationship’, as in the relationship between the real world and the control. Where Norman uses the term natural mapping he says that it relates to ‘physical analogies’ and ‘cultural standards’ [11], and in the model in this paper this concept is quite literary split in half by an important distinction related to these different categories and then elaborated in relation to communication and social constructionist concepts.

7 Communication Theory and Social Constructionism

Media genre could be considered context in communication, it has some cross-cultural propagation, but is still culturally dependent, and it transcends specific communication channels [13], much as context transcends cultural boundaries and technological platforms in the HCI field. The core of Social Constructionism is communication [2], and similarly many of the tools we use in computer interaction are socially constructed [12:791, 793] and gain legitimacy through communication within a culture {talking about them}, and furthermore, in that the meaning (purpose, utility, what it ‘offers’ [6:316]) of the tools is communicated through affordance, the tools themselves are communication artefacts. The labeling and referencing through language creates social objectifications, and so it is with HCI tangible and intangible interaction elements. As such, semiotics, denotation, connotation, ambiguity, equivocality, are concepts that are all applicable in the study of HCI, and material artefacts.

Even though affordance is often considered from a social constructionist perspective, there is not generally an incorporation of both ecology and culture in the design and communication of what affordances offer. Creating a continuum from culturally dependent communication cues of affordance to context dependent communication cues could provide a framework to understand affordance better and aid in design choices for interactive media.

8 Mapping as a Characteristic of Affordance

Using a descriminated continuum of affordance to explain different types of mapping is useful at this point, shown graphically in Fig. 2. As was stated in the introduction, the premise here is that mapping and affordance are linked very closely, if not being aspects of the same thing, here it is explained that affordance is the parent concept and mapping a characteristic of it. You can’t have mapping without affordance, but you can have affordance without mapping. Although the intention here is to discuss interactive media, it is much easier to develop an understanding of these concepts with physical artefacts, so this explanation revolves around controls in a car, being familiar to most readers.

8.1 Arbitrary Mapping

Working along the continuum from the left hand side, we could consider Berger and Luckmans [2:111] concepts of ‘incipient’, and ‘referential’ legitimacy as a move toward the conceptually derived affordance, but to start with, we have the artefacts that have yet to achieve any cultural significance and that are seemingly divorced from context. In this case we call could call the interaction as arbitrary. One might also infer that since no useful information is communicated, then no affordance or mapping exists according to some of the literature. Consider the example in the model of ‘pre-select radio channel button’. The car audio unit has a few numbered pre-select buttons, but looking at the buttons, there is nothing that tells you what they do, and the labels are just numbers and the go from left to right, so within the English speaking cultural context it may just look like someone was counting buttons, and they give you very little indication of the exact action that will occur. This category is linked with the basic language and symbolism in the culture. Although it could be argued that arrows on buttons, for example, fall into the next category, and the buttons of ‘rewind’, ‘play’, and ‘fast forward’ on many media devices go beyond arbitrariness. The key test of whether the mapping is arbitrary is whether the language or symbolism have any meaning external to the control itself in relation to the context of use, or whether it simply identifies the control as being unique. At the very end of the spectrum the control is completely disconnected and has no association with the context of use. Media that have no—apparent—interactive properties belong at the far left of this continuum.

8.2 Abstract Mapping

Using the ‘pre-select radio channel buttons’ again, consider a user that has a partner, and they programme the pre-select buttons so that at one end of the buttons they suit the users preferences, and at the other they suit his/her partners preferences. There is a level of abstract ordering being used to organise the meaning of the buttons, but this is now based on the specific culture evident in that particular vehicle.

Abstract mapping could also relate to culturally created categories or groupings, for example the distinction between what goes into the file menu as opposed to the format menu could be considered to be an abstract mapping. This category is then linked with ontologies, or what Berger and Luckman call, social objectifications, or Norman’s reference to ‘cultural standards’, that would all also extend into the next category ‘conceptual mapping’. The abstract mapping may very well relate to a feature of media interaction that needs to be learned in relation to a specific instantiation.

8.3 Conceptual Mapping

When the user can begin to interpret the function of a control based on physical analogies then you at least have conceptual mapping. Relating a control to ‘turning on a tap’ to increase the volume or rate of activity relates to conceptual mapping. We could also think of clicking on a mouse button to activate an object as conceptual mapping. I have included the ‘accelerator pedal’ in the conceptual category as there is not an immediate relationship between the movement of your foot and acceleration. Acceleration would be intermediated by a range of factors including the current rpm of the engine, and the demands on the vehicle (load, gradient). The experienced driver uses the accelerator pedal in sympathy with the state of the engine, and the concept of accelerating a car appropriately becomes in itself a well developed conceptually mapped phenomena. To a novice it may still be arbitrary or abstract, and of course that is just one of their problems.

In a media environment such as a video game, conceptual mapping might relate to connections between auditory output and volume control, or playback controls. The concepts of audio and video playback hardware is a strongly formed set of control concepts that the bulk of the audience can relate to. There are strong conceptual models in a variety of media environments, gaming for example offers many concepts that are reinforced constantly, such as the way a car racing game is controlled, or how you navigate in a ‘first person shooter’ game.

8.4 Relational Mapping

Some of the early conceptions of mapping in HCI were related to a predominantly ‘PC’ based environment, and so the conception of ‘physical analogies’ is now pushed a little as much computing is coming out of the box, as it were, and rather than being analogous to physical actions, are actual physical interactions with the actual controls. Being stimulated by driving controls, this theory has a natural physical component, and so the intention of the relational mapping category was that it related to actual physical movements. However, with mapping we cannot constrain the model entirely in the spatial realm. Relational mapping, or any of the other mapping categories for that matter, may relate to sound (pitch, tone), pressure, time, taste, etc. The best way to describe how a control could be considered to have relational mapping is to determine whether the mapping is implicit within the context of use. For example, a wheel control on a train is unlikely to map to steering. Scissors may be considered to have relational mapping, whilst a knife would quite obviously be ‘direct’. I see Norman’s category of ‘physical analogies’ being split between the conceptual and the relational mapping categories in my model. Opening or closing a window with a manual winder is just as obvious an interaction with a control in this model as using a rocker button to operate an electric window motor. Whereas, the winder converts a rotational manipulation into a vertical movement, the electronic button uses concepts of ‘up’ and ‘down’ to activate movement of one or the other, so where the manual system is relational, the electric one is conceptual. One might argue that the physical displacement of the manual winder, and the translation from rotary to linear, increases transformation complexity more than the tradeoff of introducing the concept of ‘up’ and ‘down’. However, if the rocker switch is placed out of alignment with the direction of the motion of the window, then this could increase it’s transformation complexity to make it a worse choice. We find relational mapping in the navigation of 3D virtual immersive environments, in some of these environments with something such as a ‘Treadport’ the user can ‘walk’ within the environment as there is a relationship between the walking motion and the virtual environment ‘scrolling’.

8.5 Direct Mapping

Direct mapping may also be considered as not having any mapping at all, although that would be at the very end of the spectrum where the control was entirely integral with the context of use. Hitting someone with a stick could be considered to be at the extreme end, and also using a knife to cut something as described in the previous example. Slightly off the end of the spectrum you have examples where you manipulate a control that directly operates something, such as a slide bolt, or perhaps bicycle handlebars. One could imagine that direct mapping could never be problematic, but the ingenuity of designers can sometimes be astounding, and excessive transformation complexity can ruin anyone’s day. There is also an opportunity to mimic direct manipulation through the technology. One can imagine that if it appears that the user is directly controlling something even though it is being intermediated by a complex system, then you would still have direct mapping. I had an opportunity to sit in the pilot seat of a relatively old (but still in service) military passenger aircraft whilst I was interviewing the pilots about controls. This aircraft used ‘old school’ cable technology to operate the control surfaces, and whilst there was most probably a significant collection of apparatus between the yoke and the wing, it felt as though there was none at all. It was as ‘smooth as butter’ as they say and felt as though you had immediate and direct control with no mediating ‘relationship’ between the operation of the yoke and the movement of the control surface. You would also feel the air flow and turbulence directly through the controls when flying. In the world of media and immersive technologies, using motion capture and physical motion a user can navigate through a virtual world through directly mapped motion.

8.6 Transformation Complexity

Consider a sliding bolt to secure a cupboard or a door, perhaps a toilet door in an office building. If the bolt has a knob directly fixed to it, and all you do is slide it across to lock the door, then that would be a simple mapping with no transformation in the action, and for all intents and purposes this is the same as an affordance. Whereas, if the latch was fitted with a rotating knob, engaged into the sliding bolt with a gear, then the rotation of the knob would then translate into the sliding motion; however, the knob needs to be rotated in the opposite direction if it is under the bolt as opposed to over it. This would suggest one transformation in the mapping, although you could say it is still a directly mapped action. You can also see in the model that ‘steering wheel’ is considered to have some level of transformation in the action, where the vehicle direction has a relationship with the degree of rotation of the wheel; however, the relationships transform through the speed of the vehicle, the steering ratio, and the level of steering ‘assist’ from the steering booster system. There are also occasions where you could assert that the transformation complexity was in flux. Two examples come to mind, one an excavator, the other, the indicators/turn signals on the steering wheel of Ferrari’s Italia 458. I have had the pleasure of driving one of these, but unfortunately it wasn’t the Ferrari. A common design for an excavator is to have a tracked base on top of which sits the rotating body complete with driver, excavator arm, etc. The driver has two levers available to move the tracks (although the first time you sit in one of these you’ll have no idea which of the many levers to use) and you push them away from yourself to operate the two sets of tracks to go forward. Push them together and you go in a relatively straight line, push the left, and you go to the right, etc. However, and this is a big ‘however’, if you happen to have rotated 180° on the base unit, the levers will have the reverse consequence, and you could drive over something you were particularly fond of, or your bosses car. On the other hand, the Ferrari Italia 458 has indicators/turn signals on the steering wheel, either side of the central boss. This is all very well when you are indicating an intended turn from a straight road, but if you are attempting to indicate a turn off a roundabout then you’ll find that the left indicator is no longer on the left of the steering wheel, and quite possibly pointing right or at some obscure angle. Where the Ferrari’s indicator problem is only momentary, and once the steering wheel is re-centred the transformation complexity disappears through the perception of the correct orientation of the steering wheel, the problem with the excavator is likely to continue, as the operator has limited affordance in the control itself, but has to work out in their head the transformations by seeing which way the base is pointing, and then determining which way to push or pull the controls. In fact, an excavator parked outside my office, as shown in Fig. 3, shows that this can be so problematic that some people (or manufacturers) have painted arrows on the track units to show the operator which way they go. There are some opportunities to screw up transformations with media, certainly navigating through media within which you can rotate and stretch could affect the relations, concepts, or abstractions of mapping.

Fig. 3
figure 3

Directional indicator

Full size image

9 Using McLuhan’s Grid to Interpret Interactive Media

If we take an example of interaction from interactive media, such as a menu navigation using both visual and auditory media objects, we can explore McLuhan’s theory of media interpretation, and the theories of affordance and mapping above, in relation to the interaction.

A colleague was working on a game design for both visually impaired and sighted users. Using a tablet and relying on swiping motions, the player can navigate spatially throughout the game. Part of this navigation is through the menu system itself. The menu system can be thought of as being visual and auditory media, you visually navigate from one side to the other of the menu ‘swatches’, or for the visually impaired, you navigate through a soundscape, where the sounds travel binaurally across space. A problem had arisen, where the swiping motion to travel aurally, was in conflict with the visual representation of the menu movement in relation to the swiping motion. It wasn’t possible to reverse the menu order for sighted individuals as this would then run in the opposite direction to the sounds, and it was considered important that sighted users had the same sound experience as the non-sighted, and also that the design needed to accommodate varying degrees of visual impairment, meaning that some individuals may only rely on the visual cues partially. The two media channels needed to match, and the navigation needed to synchronise. One option was to consider the swiping motion as moving liquid from one menu swatch to the next, therefore selecting it. It was considered too much work at that point to implement that alternative, and in hindsight, without the visual information, visually impaired users might get confused if the liquid analogy was propagated into the auditory channel. I suggested that they use a ‘target’ icon, the swiping of the finger moves the target to the next menu swatch and selects it, it then moves into focus at the centre of the screen. This was considered ‘not very zen’ by one of the other project participants.

This is a non-trivial problem, and the context and culture of the different users overlaps, but has significant differences within the technological platform being employed. The culture and the context that the player appropriates is in relation to the entire game, that which is experienced, but also the cues that are evident in explicit labels and in marketing and promotional material. The media may be navigated separately, or entwined. The cues that exist in each of the media need to be sympathetic to each other and also have the entirety of the meaning that is associated with either, substantively as individual cues as well. Someone that is going to play a single person puzzle game will have different interaction expectations than someone who is going to ‘play’ in a group creativity activity, but there is an expectation that they will be married by the menu system, and the interaction in different game genre will be sympathetic to the menu interaction method.

Visually the institutionalised design practice in negotiating a menu is to have the menu stationary and the user selects an element within it. Because the game is designed with the intent of being solely navigable/playable in an auditory way as well, the menu needs to reflect the movement of sounds, and macro gestures on the screen. If I swipe my finger on the screen from the left to the right then I am selecting a sound on the right, and apparently this is a cultural convention. Whereas the menu needs to visually move left.

If we use physical analogy in the affordance of the menu function then we would be reinforcing a stronger involvement in sensory closure, and lesser in structural impact. There may be less equivocality and therefore better ‘usability’, but potentially less emotional and attitudinal involvement, less fun. If you take McLuhans grid and the models above for affordance and mapping as useful and accurate, then there is not any real chance that the designers could create an interaction method across the two media that gave the same experience. Prima facie, as the media change, the experience must do as well. The mapping of the controls with their functional effect cannot be perfectly matched across the different methods of interaction. However, the interaction through a touch screen, using a combination of gesture inputs, has cultural properties that are weakly institutionalised, and therefore could be re-learned. Much as the ‘natural’ scrolling direction in a mouse wheel has recently changed in Mac OSX.

Ultimately, in designing a solution for the auditory and visual menu navigation it is important to develop an understanding of both the context of use—the activities and physical flow representations of the artifact, and of the cultures of the different users—in this case, sighted and visually impaired users. It is only with this knowledge that you can determine a design that provides appropriate affordances and mapping that provides the right level of cognitive involvement. Where in some instances you are trying to minimise equivocality, in others you are comfortable with lower definition media that has a higher level of interpretive involvement from the user.