Keywords

1 Introduction

The move to include users in an evaluation of robots is a relatively new one, prompted by the goal to build robots for the real world, where robots, through necessity or design, will come in contact with humans. Social Robots are robots that are specifically designed to interact with humans, often with explicit care or service roles. These developments have prompted an unprecedented interest in robot users and in the founding of a new area of research, Human-Robot Interaction (HRI). HRI is a highly challenging, multi-disciplinary field, requiring expertise from a number of different, often radically different, disciplines. Although disciplines that focus on human-machine interactions already exist, such as Human-Computer Interaction (HCI), scholars in the HRI field have shown that the embodied nature of the interaction means the quality of the interactions is different. Interactions with robots has been shown to elicit very different reactions in users than static technological interfaces. Most strikingly, mobile robots are highly likely to be anthropomorphized by people who interact with them, their movement and morphology often implying a much higher degree of intelligence than their actual computational power. Mobile robots in the real world have to adapt to the physical environment and social context in which they operate. Mobile robots also need to sense and respond, to the changing environment [1, 2].

Observations of interactions between robots and users also offer novel insights beyond the scope of the interaction itself, including insights into wider philosophical and anthropological questions regarding the nature of human cognition, the boundaries between humans, objects and the environment. More fundamentally, they offer insights into a more nuanced view of what it means to be human. A study of human-robot interactions foregrounds themes of anthropomorphism, attachment and empathy [3, 4] and raises potential ethical concerns about human relationships with robots, and their potential effect on society in general [5,6,7,8].

A number of HRI scholars have identified the need for a more ‘holistic’ approach to the research [9], and for new methods, methodologies and theoretical approaches [2]. Anthropology, with its commitment to holism and ethnographic research, offers a highly complementary perspective. Indeed, ethnography is now the dominant social science method used in HCI studies [10]. However, while the case for collaboration between the field of robotics and anthropology seems strong, to date there has been limited interaction between the two fields. Recent anthropological accounts of the robotics field tend to focus on the culture of roboticists, rather than the design and evaluation of the robot [3, 11]. Equally, researchers in the AI/Robotics field are more likely to look to psychology and Human-Computer Interaction (HCI) methods to account for the ‘human’ side of the interaction, see [12, 13]. It is likely that the theories and methods used in these fields, which tend to aim for quantitative precision and testable hypotheses, seem more compatible with those of the robotics field. The more exploratory, descriptive and interpretive work characteristic of much of the social sciences can seem ontologically and methodologically incompatible.

This paper proposes a model for evaluation that attempts to reconcile these distinct ontological and epistemological approaches, providing a comprehensive evaluation of user interactions within a framework that remains practicable. This paper describes the theoretical basis for a HRI evaluation model. Further research will need to be carried out to evaluate its efficacy in different empirical settings.

2 Existing HRI Approaches

A number of standardized models and metrics for HRI evaluation have been proposed. However, a standardised approach is far from resolved. Common to all, however, is the complexity of introducing a user into the traditional robot-environment dyad and the heterogeneous nature of the approaches needed. A more straightforward approach, such as [14], is oriented around a specific task in a controlled environment. In this way, the potential lack of predictability in the user’s actions is mitigated. The evaluation metrics proposed by [14] focus on three interacting levels: robot-centric; user focused; and at the system level. Evaluation metrics are suggested across eight dimensions, including navigation, perception, user management, manipulation, social interaction, as well as robot, human and system performance. The strength of the model lies in its in-depth articulation of metrics for evaluating the robot’s performance. The user is evaluated with a focus on psychological processes, such as degree of mental computation. It would therefore be difficult to transfer the model to a more natural setting, which is characterised by a wider context and a lack of predictability.

An alternative approach focuses on the social features of the robot and is based on a distinct ontological perspective: an embodied and situated view of robot intelligence and social context [9, 15]. The stated aim is to enhance and evaluate the explicitly ‘social’ features of the robot, such as the robot’s ability to distinguish between objects and people, and to proactively engage with people [15], as well as facial expressions, body language, spoken language, and rules governing personal space and eye-contact [9]. These models incorporate qualitative and other social scientific methods and approaches, such as conducting exploratory and ‘human-centred’ research [9, 15], and taking steps to simulate more natural environments [14], proposing conversational analysis and observational techniques for investigating interaction [14], using Grounded Theory, contextual design, ethnographic studies and longitudinal field studies [9]. A more streamlined approach relies solely on the use of questionnaires [13] based on the assumption that the performance criteria for service or social robots lies in the satisfaction of their users alone.

Other approaches attempt to cover the full spectrum of disciplines, and their associated methods, such as the ambitious ‘multi-level indicator model’ [16]. This model proposes a combination of quantitative, qualitative and physiological methods, including using standard quantitative measures such as ‘task completion rate’, as well as standardised questionnaires, physiological measures, fieldwork, interviews and focus groups. Additionally, the model includes an evaluation of social impact. This aspect is unique in this model and draws attention to the social and ethical concerns that are highly relevant to AI and robotics research, including potential impact on quality of life, working conditions, education and cultural context. While this provides a comprehensive view of the different methods available, the implementation of such a model and its stated contribution is so vast that it may potentially be unfeasible in all but the largest projects.

As is clear from the above models, bringing the user into the study presents a number of formidable challenges and complexities. Despite attempting to incorporate a number of qualitative and interpretive social scientific methods and theories, the approaches outlined above maintain a commitment to controlled, scientific evaluation and scientifically precise results that necessarily disregard the interpretation necessary for qualitative data. Most recommend a combination of these approaches, often without explicitly examining the distinct philosophical and academic traditions that they are based on. Additionally, there are limitations in adopting approaches and methods from other disciplines, both in terms of the knowledge and skills needed [13] and the lack of appropriateness of applying methods without adapting them to the needs of the field [15].

3 Anthropology and Ethnography - Understanding Humans

Anthropology is, at its core, the study of humankind. Anthropology covers knowledge from the social, physical, biological sciences, as well as the humanities. Within Anthropology, a number of scholars continue to treat Anthropology as a science, subject to the scientific method, which aims at uncovering generalisable, scientific laws. However, it is more common in contemporary Anthropological research to reject the idea of general ‘social’ laws or ‘grand theories’ in favour of theory as sensitizing concepts which will allow for the exploration of the structures and subjectivities of individuals and social formations [17]. This is particularly true of Cultural Anthropology, a subfield of Anthropology dedicated to developing a holistic understanding of human nature and culture.

Ethnography, or ‘fieldwork’, is the core Anthropological research approach and it is now widely applied across many fields including Business, Human-Computer Interaction (HCI) and Organization Studies. At its core, ethnography involves observing social phenomena in a real-world context and producing a descriptive text, rather than a numerical output. Ethnography takes a holistic approach to the study of phenomena, seeking to understand context, environment and the meaning that the people involved give to it. Methods used may include a combination of observation, interviews, statistical data and other descriptions deemed relevant. However, unlike the empiricist method, ethnography does not take place in a closed environment and the researcher is often actively involved in the ongoing activity. Ethnography, as it is currently practiced, does not assume a stable, objective gaze, and instead the researcher is an active participant in the ongoing activity. Far from separating values from truth, an ethnographic approach considers the explanations and perspectives of those involved in the study, not as objective explanations of social phenomena, but as highly relevant data points, that can reveal an individual’s phenomenal experience of the world. The ethnographer takes a reflexive approach to their work, where the role, background, and perspectives of the researcher are included explicitly as a core aspect of the phenomena being studied, as well as the interpretation of the data. The people involved in the study are not viewed as ‘human subjects’ but as informants. More recently, a ‘refunctioned ethnography’ has been proposed, which focuses on ethnographic collaborations with epistemic partners, rather than traditional ethnographic informants [18].

HCI was initially conceived as a fusion of computational techniques and psychological principles, with a particular focus on the cognitive demands of particular tasks [10]. However, since the 1980s there has been a move within HCI towards social scientific methods, the most common of which is ethnography. Although in HCI ethnographic insights are often limited to capturing requirements, HCI scholars have acknowledged that this limits the potential theoretical contribution that such as approach can make [19]. A number of prominent researchers have used social scientific theory in the analysis of technology interaction [20,21,22]. However, the potential for making a theoretical contribution is much greater than simply contributing to theory at the interaction level. Following Brooks [23], contemporary approaches to cognition in robotics research have extended the view of intelligence to include models that consider intelligence to be both embodied and situated. This strand within robotics research draws on phenomenological philosophy [24,25,26], neuroscience [27] and cognitive science [28]. Equally important, from an anthropological and social scientific perspective, is the social nature of cognition. The case for ‘social intelligence’ in the development of AI has been a persistent theme within the works of HRI scholar, Kirstin Dautenhahn [15].

4 A Model for HRI Evaluation

The proposed model evaluates the interaction setting on three levels, see Fig. 1 below. First, the cognitive and social features and functionality of the robot are assessed using a set of standard robot-centred models and functions which are assessed using traditional techniques by expert evaluators. Second, interaction with users is evaluated using a combination of ethnographic techniques, including observation and interviews, to capture both the physical and perceptual experience of the user in the interaction. Finally, an exploratory study of the social implications of the robot are conducted in a workshop setting by all of the stakeholders involved in the study (roboticist, anthropologist, user, etc.), as well as representatives from other relevant groups, including legal experts and citizen representatives.

Fig. 1.
figure 1

A model for collaborative HRI evaluation

Full size image

Robot Level

On the first level, the robot is assessed by an expert evaluator using standardised metrics including an evaluation of global and local navigation (e.g. % of tasks successfully completed; coverage of area), passive and active perception (e.g. % detected, classification accuracy, level of improvement), manipulation (e.g. # of contact errors) and social competency (e.g. responding to social cues, modelling user expressions). This level draws on metrics such as those proposed by [14].

Interaction Level

The next level evaluates the robot’s social competency as experienced in the interaction by the users. This is evaluated using an ethnographic approach, supplanting the need for physiological or psychological methods, and is designed to capture a holistic account of the interaction. For an example of an ethnographic study of the robotic vacuum ‘Roomba’, see [29]. The aim of the evaluation is not to generate quantifiable or numerical data, but to develop a detailed textual description of the interaction. Standard metrics, such as attention, engagement, trust and persuasiveness may be used as heuristics, but do not structure or limit the data that is gathered. The outcomes of this evaluation level are four-fold: first, it generates practical insights for robot design; second, it generates qualitative data which may be used to contribute to theoretical discussions in the fields of cognitive robotics, anthropology, and philosophy; third, the text may subsequently be parsed to develop testable hypothesis and quantifiable metrics. Finally, the collaborative nature of the evaluation and the qualitative data gathered allows for a meaningful contribution to subsequent discussions on social impact.

Social Impact Level

By taking an ethnographic approach to research involving humans, the research realigns the power relations between the researchers and the user. The user is recognised not as a human subject to be studied, but as an epistemic partner whose phenomenal experience of the interaction is an essential input to the study. The researchers, in turn, must also acknowledge and account for their presence in the study, and the effect this has on the interpretation of data. This, in itself, is an ethical move, acknowledging both the validity and the limitations of the perspectives of both the researcher and the participant/user in the assessment of the interaction and its potential impact.

Predicting future impact is a highly speculative and value-laden endeavour [30, 31]. Key methods used include the extrapolation of current trends and the exploration multiple probabilities. In this HRI model, the multiple participants and perspectives involved in the study, as well as the centrality of individual’s phenomenal experiences, are leveraged in a workshop setting into order to explore potential impact. Known issues, such as risks to employment, security and privacy, are used as a heuristic to guide the conversation, but not to limit or direct it. This research is not conclusive, but will uncover issues for future study. As well as a separate roboticist, anthropological and user perspective, it may also be useful to include other relevant perspectives, such as legal, citizens or other domain knowledge.

5 Discussion

This paper provides the theoretical justification for a proposed model for conducting a HRI evaluation based on the inclusion of an Anthropological approach. The choice of methods used in evaluation are often based on distinct ontological views, each with a different perspective of where the user experience lies: whether it is in the robot, in the user, an emergent property of the interaction between them, or something that is culturally and socially contingent. As might be expected, there are established and proven methods for evaluating technology in controlled environments, but these methods are limited when it comes to real-world evaluations or the interpretation of social action. There is a vast body of relevant theoretical and methodological literature in the social sciences, including anthropology. However, this is not widely known within AI/Robotics research and, if known, is often considered incompatible with the rigour of formal research.

This paper aims to bridge this gap. An exploratory, ethnographic study of specific interactions is ideally placed to capture novel insights about individual interactions with specific robotics technologies. It has the potential to contribute significantly to improved robot design. Additionally, an anthropological and ethnographic approach, defined by an experiential understanding of the phenomena, sees the researcher as an active, physical and social presence in the real-world phenomena being studied. Such an approach is ideally placed to collaborate on larger questions of consciousness, embodiment, situatedness and sociality. Despite this, it will continue to be a challenge to relate and translate descriptive and exploratory social insights to codifiable robot functionality and features. Yet I believe that embracing this challenge has the potential to deliver practical, philosophical and ethical contributions for social robotics research.

The HRI model described in this paper is rooted in a phenomenological philosophy, which explicitly views human cognition as embodied, social and situated. This is considered equally true for the models of cognition that are encoded in the robot, as well as the user’s experience of the interaction. This implies that while the robot’s features and functionalities may directly be known and accessed, the user’s motivation and behaviours cannot. The ethnographic method proposed for evaluating the interaction is used because the user’s behaviour is not considered either controllable or alterable.

The model depends on a strong collaboration between roboticists, anthropologists and study participants/users, and is therefore most suitable for application in large research projects. It is envisaged that once its utility has been tested and the model is further refined it should serve as a basis for a more slim line version that might be suitable for use by smaller research projects and robotics students. Additionally, the model must yet be tested in various empirical settings in order to determine its efficacy.

6 Conclusion

There is a need to develop a more standardised approach to evaluating human-robot interactions. Despite a number of models having been proposed, the issue is far from resolved. The field is characterised by the need for an interdisciplinary approach and the complexity of the research object. Including exploratory and interpretivist social scientific approaches is a particular challenge. This paper describes a model that build on existing HRI models and integrates an Anthropological and ethnographic approach that attempts to balance comprehensiveness, rigour and utility.