Social Inference Through Technology

Awareness cues are computer-mediated, real-time indicators of people’s undertakings, whereabouts, and intentions. Already in the mid-1970 s, UNIX users could use commands such as “finger” and “talk” to find out who was online and to chat. The small icons in instant messaging (IM) applications that indicate coconversants’ presence in the discussion space are the successors of “finger” output. Similar indicators can be found in online communities, media-sharing services, Internet relay chat (IRC), and location-based messaging applications. But presence and availability indicators are only the tip of the iceberg. Technological progress has enabled richer, more accurate, and more intimate indicators. For example, there are mobile services that allow friends to query and follow each other’s locations. Remote monitoring systems developed for health care allow relatives and doctors to assess the wellbeing of homebound patients (see, e.g., Tang and Venables 2000). But users also utilize cues that have not been deliberately designed for this purpose. For example, online gamers pay attention to other characters’ behavior to infer what the other players are like “in real life.” There is a common denominator underlying these examples: shared activities rely on the technology’s representation of the remote person. The other human being is not physically present but present only through a narrow technological channel.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Awareness cues are computer-mediated, real-time indicators of people’s undertakings, whereabouts, and intentions. Already in the mid-1970s, UNIX users could use commands such as “finger” and “talk” to find out who was online and to chat. The small icons in instant messaging (IM) applications that indicate coconversants’ presence in the discussion space are the successors of “finger” output. Similar indicators can be found in online communities, media-sharing services, Internet relay chat (IRC), and location-based messaging applications. But presence and availability indicators are only the tip of the iceberg. Technological progress has enabled richer, more accurate, and more intimate indicators. For example, there are mobile services that allow friends to query and follow each other’s locations. Remote monitoring systems developed for health care allow relatives and doctors to assess the wellbeing of homebound patients (see, e.g., Tang and Venables 2000). But users also utilize cues that have not been deliberately designed for this purpose. For example, online gamers pay attention to other characters’ behavior to infer what the other players are like “in real life.” There is a common denominator underlying these examples: shared activities rely on the technology’s representation of the remote person. The other human being is not physically present but present only through a narrow technological channel.

There are, roughly speaking, three ways to conceptualize how awareness cues influence human action. First, one can argue that an awareness cue, such as “Antti is away from the keyboard,” is associated with one dominant interpretation—for example, an availability inference like “Antti is not available for chat.” In other words, the content of a cue (more or less) determines the interpretations it can reliably serve. Just as the buttons of a dialog box determine the possible uses, awareness cues are interface elements tied to certain functions and designers decide these functions. Thus, designers design the uses of the system and there is little flexibility on the users’ part. Second, according to one reading of situated action theory (Suchman 1987), human action is best understood in terms of how it makes use of contextual circumstances. Human action is poorly explainable in terms of fixed mental capacities or preprogrammed action plans. Awareness cues are a natural part of these circumstances, of the context. Cues therefore feature in human action as situational resources, one source among many. As “the situation” is the arbiter of how humans act, any given cue is in principle infinitely “flexible.” A priori, designers have little or no control over how the system will be used. Designers must study the contexts of use if they are to understand in which situations cues can be utilized as resources and in which not.

These two views represent two extremes of a sort of determinism: either the cue determines its use or the user’s situation does. Between these two extremes is a third view, which places the locus not to the cue or to the context but to the human mind. There must be some logic in and limitations on making sense of a cue, and this logic must be sensitive to the situation at hand. We know from our everyday experience, and from research, that we sometimes succeed in our interpretations and can go beyond the literal meaning of a given piece of information, and we also know that sometimes our interpretations go wrong. What explains these two different outcomes? Moreover, given the same description of events, people can identify action at different levels (Vallacher and Wegner 1985). For example, the action of throwing of a brick can be identified at the lower level of picking up the brick and throwing it or at the higher level of a robbery or a misdemeanor. Proper explanations of “awareness” require answers to these questions.

This chapter puts forward a particular approach to the problem: a “translation” of Social Cognition research¹—traditionally focused on face-to-face situations—to a setting where the cues are not “direct” or “natural” but essentially mediated by technology. Social Cognition has its roots with classic European thinkers like Wundt, Le Bon, and Durkheim and with later Gestalt psychologists such as Lewin and Koffka, who theorized about the relationship between the psychological and the social. The birth of Social Cognition as a scientific field coincided with the cognitive revolution that puts forward an antithesis to behaviorism’s nonacceptance of the inclusion of mental or cultural elements in scientific explanations of human behavior. The foundational question of Social Cognition is that of the (mental) processing and storage of social information. Under this umbrella, modern Social Cognition has studied attitudes, attributions, categories, prototypes, and other representations, as well as group dynamics, social identity, and many other themes, and many of the concepts have gained status within folk psychology. Generally speaking, Social Cognition subscribes to methodological individualism, which treats “the social” as factors in the analysis of an individual. Present-day Social Cognition is divided, roughly, into two schools: the American school, adhering to the experimental method and methodological individualism, and the continental European school, which is more philosophically oriented and opposes some of the presumptions of the American school. This chapter adheres to the American, “mainstream” version of Social Cognition as presented in textbooks like those of Fiske and Taylor (1991) and Moskowitz (2004).

According to this view, an awareness cue is devoid of meaning. An awareness cue is the end result of a long chain of material causation influenced by a remote person —“the inferree.” Although awareness cues can be essential ingredients in social inference, they by no means dictate its outcome. The social cognitive account of this problem covers not only the negative side of human cognition—the errors, illusions, and biases inherent in social inference—but also the positive side—those factors that are essential to our ability to make sense of the world and go beyond the literal meaning of cues. Social Cognition provides an explanation concerning how social inference both constitutes and is constituted by action and intentions.

The approach maintains that individual acts of interpretation are the basis of a user’s “being aware” of another human—that is, so-called awareness. The atom of awareness is an individual act of inference, which encompasses both the processing of cues and its outcome. Inferences partially rely on technologically produced cues, which, in comparison to face-to-face situations, are relatively incomplete and uncertain. Technologically produced cues can be augmented with additional “unnatural” elements that are not found in face-to-face situations. Bearing in mind the fundamental limitations of the human cognitive system, one must find it surprising that reasonable inferences can be drawn at all from such cues. According to the findings of Social Cognition, people do not thoroughly evaluate all available information, as is implied by normative theories of rationality, but apply shortcuts and simple rules to overcome these limitations. These rules draw heavily from prior knowledge about the other person and are sensitive to perceived frequencies of events. Pre-knowledge and inferential shortcuts together enable “jumping to conclusions” quickly, and they enable arriving at interpretations that go beyond the literal meaning of the cue. The downside is that biases and errors are bound to occur, and effort is needed to turn aside or override routinely produced interpretations.

The goal of this chapter is to outline a social cognitive view of awareness research and to review the underlying premises. In particular, we look at how the relationship of the cue, the cognition, and the context is constructed. Several key questions about the nature of awareness will be addressed. To provide illustrations of what this approach implies for design and engineering practices, examples from studies of mobile awareness will be analyzed.

This chapter by no means constitutes the first application of social psychological research to the study of awareness systems. One influential framework has been that of Gutwin and Greenberg (2002), who borrowed Neisser’s notion of schema and Endsley’s (1995) notion of situation awareness to explain workspace awareness. Another influential piece of work, although carried out in a different application domain, has been the analysis of social cues in e-mail by Sproull and Kiesler (1986). The work in this chapter was precipitated by the regrettable fact that, despite such endeavors, social cognitive approaches have generally failed to secure a position that would reflect the depth and breadth of work done in the parent field. Another problem has been that social psychological approaches have been confined to certain narrow areas of application and the more general questions of social inference through technology have not been addressed holistically and systematically. Unfortunately, the majority of research on awareness systems remains technology-driven and scientific progress is accomplished mainly by trial and error. This hampers the accumulation of empirical findings and construction of theories. Consequently, awareness, as a scientific concept, has come to mean almost everything and therefore nothing (Gross et al. 2005; see also Rittenbrush and McEwan, Chapter 1). That said, the goal of this chapter is first and foremost to start rebuilding the bridges between Social Cognition and awareness technologies by updating the general theoretical framework. It is generally a reasonable strategy to attempt to replace concepts stemming from common sense with more accurate scientific concepts. Therefore, this work involves revisiting a few of the basic notions in earlier literature.

2 Projections of the Social

This section analyzes the notion of an awareness cue. An awareness cue (hereafter referred to simply as a cue) is essentially any signal or symbol or mark in the user interface—typically textual, graphical, or auditory—the content of which is produced (or influenced), in real time, by the actions or properties of a remote person. Computationally, a cue is produced by gathering data from hardware or software sensors and applying computational transformations, then transmitting the outcome and presenting it in some human-understandable way. All awareness cues together in the user interface constitute what is called here its projection of a remote person.

It makes sense to define awareness cues in this way, somewhat narrowly, when they are to be analyzed via Social Cognition’s theories. The criterion that they are automatic makes sense, because, as soon as user-created messages and signals are included, the uses of awareness systems become dominated by social interactive phenomena like self-presentation, performances, discourse, and coordinations. It makes sense to narrow awareness cues to live cues as opposed to, for example, histories and trajectories of past events. The Social Cognition approach is by no means limited to real-time cues, but adding time as a factor would complicate the analysis. For the purposes of this work, we deliberately apply a narrow scope. Finally, if awareness cues ought to be analyzed through social cognitive theories, it makes sense to focus on cues related to other people, as opposed to location-centered, object-centered, or document-centered cues. Many well-known findings of Social Cognition are instances of more general principles of cognition, yet it makes sense to restrict the scope of the approach for the sake of simplicity. It cannot be presumed that inferences of nonliving entities are governed by the same tendencies. Despite this admittedly narrow definition, awareness cues defined like this are truly ubiquitous. Basically, all automatic and real-time sources of information that people appropriate to apprehend the undertakings of a remote person can be analyzed as awareness cues. If a piece of information that was not originally designed to be an awareness cue is in fact utilized as such, it must be analyzed accordingly. The domain of social inference through technology is therefore broader than the domain of awareness technologies.

In this section, we discuss two important topics: (1) the ontological status of a cue and (2) limits to what a cue can be.

2.1 Anatomy of a Cue

It would be a fallacy to postulate that cues are meaningful because “meaning” was designed or imbued “in” them. Let us consider an example at a very concrete level. Awareness cues essentially consist of changes in the transparencies of R-, G-, and B-type liquid crystals that we call “pixels” on a display. The immediate causes of changes are controlled by the computer according to some programmed logic. This logic links the changes produced to changes in sensor data registered by the program. Thus, an icon we see on the display is the product of a long chain of material transformations and causations, the effective source of which is a change in the material state of an object that we call a sensor. Now, to the extent that this state was influenced by a human being—for example, by his or her Bluetooth device introducing a detectable pattern in the proximate electromagnetic radiation field—the cue can be said to be materially caused by the behavior of that person.

Cues are essentially material objects that can be perceived and interpreted by a human. The meaning-giving process starts with the individual’s perception of the cue.

2.2 Limitations of All Intermediaries

In fact, everything we perceive of the outside world is mediated in one way or another. For example, my “seeing” of green leaves from the window is the outcome of a chain of causation that commenced from the scattering of 510-nm-wavelength photons from the surface of leaves. There is no substantiated reason to claim that the injection of a technological intermediary in the chain would have to transform the outcome to something “unnatural,” given that it does not distort the outcome in any way that is noticeable to the human observer. This argument has a radical implication: Technology can, in principle, mediate perception and action in a natural and perfect way.

As we know, perfect mediation is impossible, because of fundamental limitations of the mediating capacity of technological intermediaries:

• 1. The noise problem: Technological sensing of the world, as any sensing, is prone to errors and noise. This may mean something as simple as lag or something as devastating as one of the sensors being cut off totally.

• 2. The augmentation problem: Technological sensing is not only incomplete but can also at times fabricate, embellish, and confabulate details. This is not solely a matter of errors in sensor-based measurement. For example, the literal meaning of a location cue is that a particular mobile device, the one that senses it, is in a particular location. Nevertheless, the interpretation that awareness applications are an attempt to convey is that the person (the owner of the device) is in that location. Anybody who uses more than one phone or does not always carry his or her phone knows that this assumption is problematic.

• 3. The keyhole problem: The projection is inherently limited in scope and misses some aspects of the to-be-presented situation, which at some point may be important. In such a situation, the user must “move the keyhole”—proactively make the missing information perceivable, which may not always be possible.

A resulting empirical question is how users learn to cope with the implications of these limitations (for an interesting discussion of ways to exploit these problems for the benefit of the user, see Chalmers and Galani 2004).

2.3 Can the Social be Projected?

If we forget the quest for perfect cues, the cues must simplify and abstract the remote state of affairs in order to be of practical value. This, in essence, involves replacing a stream of data with simpler data or some human-recognizable symbol or label. This brings about a problem that is here called

• 4. the symbolization problem: Can computers adequately—that is, in a human-approvable manner—give labels to states of affairs outside its “skin”?

An abstraction inherently omits information. For example, district labels attached to GSM cell IDs are abstractions of electrical changes in the receiver device. A consequent problem for the user is that high-fidelity abstractions are needed in some situations, while in others these might be unnecessarily detailed. For example, Weilenmann (2003) has studied the location-telling practices of people, reaching the conclusion that it is impossible to find one generic method to operationalize location in awareness systems.

There may be an even more serious problem, one posed by computers’ limits in “understanding” human activities (Dreyfus 1992; Searle 1980). It is fundamentally difficult to program a computer to recognize social events—for instance, something as mundane as a person inviting another person or that a “game” is going on. There is no one pattern in language, turn-taking, gaze, posture, or other overt behavior that could be preprogrammed into an infallible “invitation recognition machine.” Even if a very large number of different styles of invitation were “hard-wired” into the computer, or learned by means of supervised learning from a data set, the machine would still perform only very locally, devoid of more fundamental understanding of those social practices where invitations are defined. The source of this problem is that understanding sociocultural meanings of human actions requires active participation in the culture within which those actions occur. For example, through social learning with their parents, infants learn perspective-taking, the ability to simulate the intentions and reactions of another human being, and this ability is central not only for social interaction but also for many complex cognitive feats (Tomasello et al., 2005). On account of being excluded from the interactive practices, the computer cannot achieve the level of an adept human member of that culture. Unable to learn concepts by grounding them to bodily and social interaction, the computer is doomed to function as the Chinese room of Searle (1980), translating meaningless symbols into other symbols, never truly understanding the activity.

The moral is that the limits of awareness systems may be in cues that represent aspiration to represent phenomena that are constituted in or by social interaction. Unfortunately, many of the automatic cues people would want to include in their awareness systems—like interruptibility, availability, activity, and place—are examples of such cues.

3 Inferring a Projection

Despite the aforementioned limitations, users can in certain circumstances arrive at high-level interpretations from lower level cues and vice versa. This section discusses how this is possible. It first presents a definition of social inference adopted from the literature. We then investigate some of the key phenomena, involving (1) the use of pre-knowledge, (2) the temporal order of processing, and (3) the use of inferential rules and heuristics. We finally turn to criticisms that can be raised against this kind of mentalist explanation. The purpose is not to summarize findings in the field of Social Cognition; rather, the aim is to offer a cursory overview of some of the key analytical concepts.

3.1 Elements of Social Inference

The locus of the meaning of a cue is the intrapsychic process of inference. Generally speaking, an inference has three components: (a) a set of premises; (b) a conclusion; and (c) rules, principles, templates, or procedures that connect the premises to the conclusion in a reasonable manner (Hastie 1983). Hence, the analysis of social inference involves both (1) the process and (2) its outcome. This definition is pragmatic as well; designers need to know not only how a certain awareness cue was used but also how the interface was processed for arrival at a given conclusion.

Social inference is a special case of inference wherein the conclusion concerns another person or a group of people. Social inference is so fundamental to all social interaction that the topic has gained much ground in today’s Social Cognition literature. Causality attributions, attitudes, schemas that influence the process, memory for information about other people, heuristics, and biases are among the key topics. In the discussion that follows, we provide one interpretation of these theories, a theoretical cross-section from pre-knowledge to heuristics.

3.2 The Cognitive Miser

Cognitive psychology has revealed fundamental limitations to inference. These limitations are so pervasive that Fiske and Taylor (1991) call people “cognitive misers.” Table 5.1 presents their view of what social inference entails and how each phase in the process is compromised by potential biases. Three cognitive processes take the foreground in interpreting technological projections of people (adopted and modified from Fiske and Taylor, 1991):

• 1. Schemas. Schemas are cognitive structures that represent knowledge about a concept or a type of stimulus, including its attributes and the relations among those attributes (Hastie 1983). They facilitate top-down, conceptually driven, processes. They are concerned with the general case, abstract generic knowledge that holds across many particular instances. They store knowledge at a molar level rather than including all of the individual experiences in their “raw form.” For example, an inferrer may know, on reading an alarm profile cue, that the inferree is in a classroom, because students are supposed to keep audio alarms silent during classes. Perhaps the most well-known subtype of schemas is the script, a schema that organizes temporal and causal dependencies in a social event such as going to a restaurant (Schank and Abelson 1977). Research has amassed findings about the contents, conditions, and effects of schema use. It has identified factors relating to goals; social identity; whether role or trait schemas are used first; how visual and physical cues trigger schemas; and the effects of mood, accessibility, power, and salience. Importantly, the way in which cues are presented can affect the activation of relevant schemas and vice versa. A cue can activate a schema, and an active schema can guide the selection and processing of cues.

• 2. Person memory. Person memory entails the encoding processes, representations, and access of memories involving other people. Person memories contain information about the appearance, behavior, and traits of another person. Several categories of explanatory models have arisen, ranging from associative network models to procedural memory, exemplar models, and parallel distributed processing, each organized either by person or by group. Again, how cues are presented can affect the encoding and reactivation of person memories, and vice versa: the active person memories can guide the selection and processing of cues. Users have been found to use both person-specific knowledge and general schematic knowledge when inferring mobile awareness cues (Oulasvirta et al. 2007).

• 3. Heuristics: People often use heuristics or shortcuts that reduce complex problem solving to more simple judgment operations. Four well-known heuristics are (1) the representativeness heuristic, (2) the availability heuristic, (3) the simulation heuristic, and (4) anchoring and adjustment (Tversky and Kahneman 1974). Without heuristics, the inference process would often be unmanageable. We will discuss how the way cues are presented in the interface may affect the use of heuristics.

Table 5.1 The standard model of social inference paraphrased from Fiske and Taylor (1991, p. 348): Inferential processes and potential sources of bias

Despite the fact that the description of social cognitive mechanisms is framed in terms of judgment errors, there are many virtues achieved by using heuristics:

• Familiarity: the ability to transform a problem into a more familiar form, in relation to which previous knowledge can be applied.

• Selectivity: the ability to sift the relevant from the irrelevant.

• Anticipation: the ability to utilize knowledge of event structures and frequencies for anticipating future events.

• Situational sensitivity: the ability to take into account situation-specific knowledge (e.g., schemas) and person-specific knowledge (e.g., person memories) in one’s interpretation. On the one hand, coherence of action over time can be reached only if a new situation does not arbitrarily change a well-known course of action. On the other hand, one must be sensitive to the unique features of the situation at hand. If preconceptions dictate the inference, the cues will have no informative value and they will be useless.

• Rapidity: the ability to process information rapidly enough to accommodate the moment-to-moment demands of social interactions.

We return to these virtues later in this section and in Section 5.5.

3.3 The Mediated “Social” as an Intrapsychic Process

Of the many questions these arguments have raised but left unanswered, let us answer one that is more philosophical.

On the surface, the social cognitive approach may look harmless. It may seem merely adding to preexisting conceptions of awareness the individual’s mind, seated somewhere “between” the user interface and the subsequent action. However, there is an ontological payload with deep implications.

Social Cognition, as a field, is generally regarded as holding an intrapsychic model of “the social.” That is, other humans are treated as factors and content of cognitive processing, but another human is never the sole or direct cause of behavior. However, the social cognitive approach is not an utterly mentalist view of the mind, because the psychological construction of reality is itself heavily affected by “the social” outside the mind, in turn, because the relevant representations and inferential skills have been learned from interaction with comembers of the culture (Augoustinos and Walker 1995).

Awareness, from this perspective, is the outcome of inferential processing, and this outcome is in the mental realm. Awareness is essentially a mental representation, or a belief state, that concerns somebody else’s current situation. Awareness exists, ontologically speaking, primarily as a mind’s construction rather than as a practice or an activity. This does not mean that the activities wherein inferences take place are not important, as they evidently are. However, it does imply that it would be a fallacy to hold action (or practice) and inference (or awareness) to be tantamount to each other. Neither are the two analytically inseparable (see Schmidt 2002).

On the surface, this characterization resembles the influential definition of awareness put forth by Gutwin et al. (1996). Following Endsley’s (1995) notion of situation awareness, they defined awareness as the collection of knowledge a person holds about another person. Similarly, Dourish and Bly (1992) characterized awareness as “knowing who is around, what activities are occurring, who is talking with whom” (p. 541, emphasis added). The model of awareness proposed by Gutwin and Greenberg (2002) follows Neisser’s (1976) perception—schema—action cycle. In that model, environment modifies knowledge schemas, which in turn direct exploration and affect the sampling of the environment.

The present framework and that of Gutwin and Greenberg (2002) disagree on the question of “where” the meaning of a cue lies. An engineer might claim that the meaning of a cue has been predetermined by the engineer or the designer. In their framework, Gutwin and Greenberg treat awareness cues as if there were a mapping from an “awareness widget” to a set of inferences. They categorize awareness cues into three classes—identity, location, and action. They further claim that users can accurately predict the other’s actions and intentions if the cue shows the other’s body and movement in real time. By contrast, the present framework posits that the range of potential inferences of a cue is much broader, although not unlimited or arbitrary. This is particularly salient in the context of mobile awareness, where the processing goals and settings are more diverse and unpredictable than in the groupware settings that Gutwin and Greenberg studied.

Consider Table 5.2. The table is based on interviews of three user groups who used a mobile awareness system (see Fig. 5.1) for a longer period of time. The table collects their self-reports on inferences of the location cue. These inferences were collected through the cue-based narrative interview method, as reported in Oulasvirta et al. (2007). The extent of inferences that can be based on a single cue, such as location, is quite an indisputable evidence against the claim that cues prescribe the user’s inference, although a cue certainly contributes to this process by constraining the space of meaningful and valid inferences. The mind’s inference process is a necessary condition for a cue to achieve personal meaning to the user. Inferences are not products of a deterministic process but conditioned by epistemic skills and the processing goals in the pursuit of which they are produced.

Table 5.2 Inferences of a location cue in three different user groups (Oulasvirta et al. 2007). Self-reports from posttrial interviews (the location cue was implemented as a district label for frequently visited GSM cells, district labels being fetched from a teleoperator’s database)

Fig. 5.1

(a) The standard ContextContacts integrated with the contact book of a mobile phone, (b) a detailed view for a selected contact, and (c) a version with the free-text cue. Callouts for cues: (1) district (automatic label for a GSM cell ID) or place (user-defined label), (2) duration of stay cue in hh:mm format, (3) phone manipulation (gray hand = no use for over 15 min, red hand = recent use), (4) alarm profile (audio/vibra and on/off), (5) number of people or friends in the vicinity (yellow/green person icons displayed according to the presence of unknown/recognized Bluetooth phones), and (6) free-text cue

4 The Psychology of Action

The question of how action shapes and is shaped by inference should be a central topic in social cognitive analysis of awareness. In 1992, Dourish and Bellotti defined awareness as something that “provides context” for one’s own “activity” (p. 107). Social inferences are embedded in the continuous construction and management of social relationships. We know the key inferences of a few social activities. For example, inferences central to collaborative tasks include the copresence, visibility, audibility, cotemporality, simultaneity, sequentiality, reviewability, and revisability of action (Fussell et al. 2005). Within psychology, the relevant area of inquiry is called the psychology of action (Gollwitzer and Bargh 1996).

Three elements jointly influence which kind of action can be based on the cues, as well as the will to continue with a chosen course of action (following Covington, 2005):

• 1. The kinds of goals that the user brings to the inference situation.

• 2. The motivating properties of these goals.

• 3. The relevant reward structures.

Again, the purpose of this chapter is not to lay out a theory but to illustrate opportunities that modern psychology can furnish for research. These opportunities go beyond those of Neisser’s 1976 model utilized by Gutwin and Greenberg (2002). Importantly, modern theories may explain why awareness cues are appropriated differently in different settings and by different users.

4.1 Goals and Motivations

One argument in modern psychology is that there are basic needs shared by all humans that are individually translated into motivations, which in turn are situationally translated into goals for action. In other words, the inference process is “sandwiched” between motivations and goals.

According to Deci and Ryan’s (2000) influential theory, competence, autonomy, and relatedness are the three basic psychological needs shared by all humans (see Oulasvirta and Blom 2008 for a more thorough treatment of the topic). Let us take one of these under closer scrutiny.

Relatedness is the need to establish close emotional bonds and attachments with other people. It reflects the desire to be emotionally connected to and interpersonally involved in warm relationships (Reeve 2001). Deci and Ryan (2000) describe it as the desire to feel connected with other people through loving and caring and also being loved and cared for. Awareness features that help to convey emotions, feelings, and intentions to other people may support the need for relatedness. They may provide possibilities for people to please others and gain their approval. For example, users of online multiplayer games have appropriated their aliases to show group (clan) affiliation to others.

The need for relatedness is ultimately translated into goals that regulate action in concrete situations where awareness cues are utilized. One conceptualization of these goals is that through them users can approach awareness cues in the pursuit of certain influence on other people or with a more exploratory mindset. It is known that active perceivers, immersed in the interactions that they seek to interpret, who can affect the objects of their perception, have different motivations and information-processing goals than do passive perceivers, who cannot affect the objects of their perceptions (Jones and Nisbett 1971). Active perceivers in general “concentrate primarily on the relation between their influencing behaviors and the responsive behaviors of their target and ignore other important sources of information relevant to social inference” (Gilbert et al. 1987, p. 861).

In Oulasvirta et al.’s (2007) study of a mobile awareness system, the distinction between active and passive perception is made clear. The cues (see Fig. 5.1) were used for tasks like coordinating decisions on mobility and calling—for example, whether the other person concerned is approaching the meeting point; if the other person is close; and whether he or she is interruptible, likely to answer the telephone and available, etc. For such task-driven interpretations, the timeliness, accuracy, and reliability of cues are of crucial importance.

However, many users also mentioned feeling of companionship as the main benefit of using the system, a use rather different from task-oriented ones. Users expressed feelings of presence, closeness, affection, communality, and connectedness as being mediated by mobile awareness cues. Oulasvirta et al. (2007) argued that this manifests a less task-oriented processing goal and involves more holistic reading of cues. The cues were used for staying in touch and for gaining reassurance about the well-being of others (e.g., to know whether others had arrived home safely)—in other words, benefits that can be plausibly inferred to be relevant in terms of the basic need for relatedness. Moreover, there were instances of the users tracking each other and looking at awareness cues for a long time. This resembles “awareness moments” reported in the use of IM (Nardi et al. 2000). Moreover, there were a few reports evidencing that extraneous effort was put into keeping the phone close to oneself just to maintain a connection to others. Repeatedly, looking at cues may associatively prime the related person memory and thus increase the probability that it “pops into mind” and is actually acted upon (Bargh and Ferguson 2000). This may constitute a feeling of “awareness” that continues even in the absence of the cues themselves. In exploratory processing, the aim is in constructing a more holistic representation of the other so that he or she can be felt as “present.”

4.2 Perception, Action, and Feedback

The arena of motivational dynamics is shaped and modulated by cognitive processes of two kinds: perceptions and expectations. The outcome achieved is evaluated and the process may restart at a later time with new expectations, motivations, and perceptions. The following mental events may take place when one is using the cues to make a decision to call a friend:

• 1. Expectations: for example, believing that a friend is interruptible on his or her lunch hour.

• 2. Inference: inferring from the cues that there is an opportunity to call the friend.

• 3. Action: placing a call but failing to reach the callee.

• 4. Evaluation of feedback: judging whether the chosen time was wrong.

• 5. Updating of expectations and restarting from step 1 at a later time.

One psychological event that bridges goal-setting and the user interface is forethought. Future-directed plans are rarely specified in full detail at the outset, because anticipation of situation details is difficult. Therefore, through the exercise of what Bandura (2001) calls forethought, people motivate themselves and guide their actions in anticipation of future events. Via representation of future states in the present, a foreseeable future is converted into current drivers and regulators of behavior.

An interesting hypothesis is that forethought plays a particularly vital role in the initial stages of learning an awareness system. The outcomes of action alternatives are envisioned and utilized as a basis for selecting one’s action. However, utility mispredictions can surface, stemming from biased retrospective evaluations and from misguided theories of the future. Because action based on incorrectly predicted utility can lead to mismatch of expectations and achievements, and thus discourage the user from using the system, it is essential that attention be paid to designing the cues so as to minimize the possibility of such mismatches. The other component in expectations is cost. Cost can be conceptualized in terms of the negative aspects of engaging in the task, such as fear of failure, or the lost opportunities resulting from making one choice rather than another. In practical circumstances, when one is deciding on whether to act upon an inference or not, cost could mean, for example, time, effort, or money. A possible implication of design is that, if forethought is to be supported, awareness cues should be descriptive (Antti is busy writing a paper) instead of prescriptive (Don’t call), because cues of the latter type provide poorer means for imagining alternative outcomes for actions.

The lesson is that inference of cues links to action through interplay of expectations that motivate action, actions that produce effects, and effects that are evaluated as feedback.

5 Examples From Mobile Awareness Systems

The body of research studying social inference through technology is tremendous in volume, at least when interpreted in the broadest sense of the notion. However, the efforts are distributed across several topics, covering a broadrange—like videoconferencing, telepresence, social behavior in online communities, awareness, and remote collaboration. This unfortunate “balkanization” is not merely symptomatic of applied Social Cognition; it characterizes almost all applied psychology that examines Human-Computer Interaction (Carroll 1997). Consequently, the foundations of inference through technology have not been thought out systematically.

The following examples are from a particular application domain, mobile awareness. They come from studies that have not concentrated solely on the evaluation of applications but explored more fundamental questions of social inference of awareness cues.

The awareness system studied is ContextContacts, one of many systems of its type (Bardram and Hansen 2003; Burak and Sharon 2003; Holmquist et al. 1999; Isaacs et al. 2002; Marmasse et al. 2004; Milewski and Smith 2000; Tang et al. 2001). The interface is presented in Fig. 5.1. The driving idea in the system’s design was to integrate awareness cues into the contact book of a mobile phone, thus transforming it into a group-oriented venue in which the presence and undertakings of other members can be easily viewed and acted upon.

Controlled laboratory studies (e.g., the work of Oulasvirta et al. 2005) have been conducted concomitant with A—B intervention trials in the field (Oulasvirta et al. 2007). These studies examined (1) pre-knowledge, (2) task orientation, and (3) selectivity. The results are, by and large, aligned with the predictions of Social Cognition research.

5.1 Effects of Pre-Knowledge

The earlier paper (Oulasvirta et al. 2005) reported a set of experiments that was conducted to find out how background knowledge about another person features in the inference process. Simulacra of real mobile awareness cues were used.

In the first experiment, 10 participants were presented with five different cues adopted from ContextContacts (district, district with duration of stay, current alarm profile, whether the phone has been in use in the past 2 minutes, and whether the person is in the company of six or more people), one at a time. The participants were asked to list as many situations as possible in which a given person could realistically be, given that information. There were two conditions in which such enumeration was carried out: (1) the person is unknown and (2) the person is known, a coworker we named and with whom everybody was at least familiar. The idea behind this manipulation was to assess the effect of background knowledge by comparing inferences for a known and unknown person. The participants were also asked to place an imaginary monetary bet to represent confidence in their guess. In studies of decision-making, this is commonly used as an intrasubject metric for perceived value.

The results (shown in Figs. 5.2 and 5.3) support the hypotheses of Social Cognition:

• 1. In general, pre-knowledge helps one to make more elaborate inferences.

• 2. However, pre-knowledge works only when the cue is familiar. In the set of five cues we studied, two cues, “duration of stay” and “number of people present,” did not see a benefit from pre-knowledge.

• 3. Inferrers think that they can make more accurate inferences when they can utilize their pre-knowledge about the person.

• 4. Inferrers also draw more inferences when they can utilize pre-knowledge.

Fig. 5.2

The effect of pre-knowledge on the certainty felt about an inference

Fig. 5.3

The effect of pre-knowledge on the number of inferences (activities mentioned)

The main thrust of these findings is that pre-knowledge is necessary for interpretive flexibility.

A related question is what kind of pre-knowledge is useful. This question can be studied only in a situation where the inferrers have developed inferential skills with the particular set of cues involved. In a field trial of ContextContacts (Oulasvirta et al. 2007), both person-specific memories and more general types of knowledge, schemas, were identified as being utilized in the interpretation of cues:

• 1. Person-specific knowledge, similar to person memories, of the other person’s current activity. One participant, for example, said that if the phone is not in silent mode but she knows that the person is at school, then she knows when to call because she knows when the breaks start and end. Known regularities of the movement patterns of a person also helped to augment, and at times overcome, the low granularity of the district cue in the system. For example, the information that two schoolmates always “hung out” together after school in a certain place was utilized. Patterns of alarm profile switching were also employed.

• 2. Schematic, general knowledge concerning typical activities for the time of day was utilized often. Social knowledge (e.g., that phones should be silent during classes) was also utilized to explain the alarm profiles observed. Moreover, semantic knowledge of an area was used to draw conclusions as to possible activities when another was seen in an unexpected or unusual location.

The key point from these findings is that the availability of specific and general pre-knowledge constrains the range of inferences possible for a cue. Novel cues most likely require more practice before becoming useful. However, if general and person-specific memories can be directly applied, this can be avoided, as Fig. 5.2 indicates.

5.2 Selective Processing of Cues and Transfer

In the second laboratory experiment reported in the 2005 paper by Oulasvirta et al., we studied whether the order in which cues are processed affects interpretation. If order affects outcomes, as the anchoring heuristic suggests (Tversky and Kahneman 1974), the user interface should guide users to process the best anchors first.

In the study, 10 participants were asked to make interpretations based on pairs of cues. An anchor was presented first—for example, the current district of another person—and an inference of the other person‘s current undertaking was committed to paper. Then, another cue was presented, and the participant was asked whether this additional information would change or augment the original inference. The results are presented in Table 5.3.

Table 5.3 Utility attributions with interference based on one cue (“Bet” Column) and two cues (“Utility” columns)

The results are symptomatic of anchoring: for some pairs. Dramatically different inferences were reported, and subjective utilities ascribed, when the order of presentation was reversed. Consider the case where “Turnover of BT devices in range” (e.g., “20 devices recently”) is the anchor for the phone manipulation cue: the perceived utility was 0.34. By comparison, when the order was reversed and the phone manipulation cue presented first, the perceived utility doubled to 0.74.

These findings are backed up by users’ self-reports in the field study of Oulasvirta et al. (2007). In that study, the users reported using heuristic-like cue-to-inference mappings when using the system. Consider the following excerpt:

I’ve often had situations when I check—for example, during a school day—whether a friend is available or not. In practice, I check whether the phone is silent or not. You cannot call a person with no [audio] alarm. [The respondent was an entrepreneur.]

However, inference based on a single cue was not always achievable. The following quote illustrates anchoring on the “hand cue:”

The hand was mostly white, but it did give more hope when it was red. At the point when you call, I do not often look at whether it’s red or not. The only thing is that when there’s no audio or tactile alarm, there’s no hope of reaching the other person if the hand is white. But if it is a red hand, you usually thought that he or she might notice your call. [The respondent was an entrepreneur.]

In addition to the hand cue, the district cue was a dominant anchor, while the Bluetooth-based cues were predominantly used as secondary cues:

When I haven’t been able to participate in some group work, I’ve been looking [at the yellow person] to see when they are leaving. Then the place thing [district] has been used. And then how many people there are, and my friend Julia is visible as a yellow guy. Nina is usually also there but there are no traces of her; she’s not visible as a yellow guy. [The respondent was a student.]

Without these rapid and effortless, yet effective, means for selecting cues, users would easily become overwhelmed with “information overload.” Selection appeared to be natural to the users; only one expressed difficulties, complaining about “symbols starting to flash in your eyes.”

We also learned that this kind of selection can be very rapid. We measured what we dubbed the “pre-called delay”: the time for which users keep a contact highlighted in the contact list before pressing the “place call” button. This is indicative of their reading of the cues. Most probably this time is spent in assessing whether the person is going to be available. When we compared cue-augmented contacts to unaugmented contacts, a substantial difference was observed; when there are no cues for a contact, 60% of calls are placed within less than 1 second after moving the selector on top of the contact. When the cues are present, 60% of the time the users pause for 1—3 seconds before placing the call. This is tentative evidence for the usefulness of integrating mobile awareness cues with communication functionality.

One explanation for the apparent fluency in selection is that interpretations were facilitated by the transfer of interpretation skills. The users had already used time of day as an implicit cue long before this trial, for example, in their daily decisions of when to call a person. Transfer is a good candidate to explain the fluent use of the hand cue as well, because of its resemblance to availability cues in IM. Similarly, locations are related at the beginning of phone calls, particularly in mobile coordination (Laurier 2001), which may have provided a source of transfer for the district cue. From this perspective, the case of Bluetooth cues is interesting, because, whereas one group found almost no uses for it, another developed an inferential skill for its use. They reported multiple interpretations of those cues, but only toward the end of the trial. Learning of interpretation skills may help to reconcile the dilemma involving the known limitations of the cognitive machinery versus the skilled worker’s ability to use tacit cues innovatively (Heath and Luff 1992; Schmidt 2002).

6 Conclusions

Computer-mediated indicators related to remote people feature so commonly in contemporary information and communication technologies, not only in awareness systems, that the question of how they are interpreted and acted on lies at the heart of one of the most significant developments in the history of personal computing: the transformation in common thought of the personal computer from a tool for an individual into a tool for social interaction. The stakes are high—if it is found that no fundamental limitation to awareness cues exists, they can ultimately serve as artificial proxies for humans, perfectly replicating the companionship and presence of other human beings. On the other hand, if theoretical or practical limitations do exist, it is important to chart what they are.

The quest for an answer is fueled by the ongoing “ubiquitous computing” revolution, which will soon warrant more imaginative and pervasive forms of awareness. However, attempts to find a conceptual basis for developing these systems have been disappointing. According to Schmidt (2002) and Gross et al. (2005), the term “awareness” has not been used consistently among researchers and its very definition is problematic (see also the historical review offered by Rittenbrush and McEwan, Chapter 1). Flavors of awareness, such as “general awareness,” “collaboration awareness,” “peripheral awareness,” “background awareness,” “passive awareness,” reciprocal awareness,” “mutual awareness,” and “workspace awareness,” are seen frequently in the literature. From the Social Cognition perspective, a likely explanation for this dispersion is that attributes of activity—as in collaboration awareness—have been confounded with attributes of cognitive processing—as in peripheral awareness. The boundaries among technology, human, and action have become blurred. Omitting social inference from the analysis may have led to the untenable conclusion that awareness can be almost anything.

The field has also suffered from romanticizing of users’ abilities. For example, Schmidt (2002) and Heath and Luff (1992) admire users’ “highly active and highly skilled” ways of constructing awareness in everyday cooperative settings. Schmidt (2002) concludes his influential editorial as follows: “From a cognitivist point of view, the very notion that an actor is able to pick up and relate to occurrences beyond the scope of his or her line of action and without interrupting that line of action is difficult if not outright impossible to fathom” (p. 293). In light of the present approach, expert interpretation of technological projections can be explained by cognitive skills that allow the inferrer to anticipate, select, filter, elaborate upon, and enrich the cues. This approach may help to explain why novices are unable to reproduce the feat and why even skilled users sometimes fail to achieve meaningful and actionable interpretations. The Social Cognition approach to awareness calls for empirical searching for explanations, not mystification of users’ abilities.

To conclude, the social cognitive approach appears auspicious. It helps us to see essential connections between user interfaces and human action, it brings in a wealth of findings from a field that has studied analogous problems for decades, it opens a plethora of new questions for study, and it can sensitize designers to design issues that were previously not even noted. To produce research hypotheses, one can simply take any empirical finding from a textbook in Social Cognition—typically of the form “Cognitive event C organizes the processing of social setting S so as to make behavior B more probable,” and replace “social setting S” with “technology-mediated cue of social situation S.” Some of the key concepts in Social Cognition, such as attitudes, attribution, social judgment, prototypes, group dynamics, and social identity, will become amenable to investigation in awareness research through this kind of translation. Integrative research into this topic will reveal the limitations and capabilities of technological projections of people. Designers should be aware, however, that the way argumentation works within the Social Cognition approach is not from design to inference but vice versa: one must understand the principles and conditions of social inference in order to understand how to best support it.