1 Trust in Human–Robot Teams

The possibility of mature human–robot teams (HRTs) seems within reach with recent advances in unmanned systems, self-driving cars, and similar applications of artificial intelligence [50, 132]. We define an HRT as a team consisting of at least one human and one robot, intelligent agent, and/or other AI or autonomous system. Strictly speaking, a robot is an intelligent system with a physical embodiment, yet in the context of this paper, we choose to use the term human–robot teaming to encompass a broader range of human-autonomy teaming constellations (see Table 1). Even as artificial intelligence and robotics mature to the point of ubiquitous use, the question remains how to create high performing HRTs [4, 12, 31, 32, 55, 98, 108, 117, 124, 137, 148, 154]. A recent study showed that a key predictor of good teamwork is not about having good (technical) capabilities, but about having a way to allow for vulnerable communication in casual and non-work related interactions [37, 39]. Such communication is a major facilitator of positive trust development within the team and ultimately, as the study showed, a major predictor of a team’s success. On the flip side of promoting healthy trust relationships is avoiding unhealthy trust relationships. Decades of industrial psychology, human factors and robotics research have shown that inappropriate or insufficient trust in another team member can have costly consequences [53, 59, 113, 120, 122, 125]. Trusting too much (“overtrust”) can condition operators into complacent states and misuse which can lead to costly disasters with the loss of human lives and destruction of expensive equipment [58, 111, 121, 127]. Trusting too little (“undertrust”) can cause inefficient monitoring and unbalanced workload, leading to disuse of a machine or the avoidance of a person (see Fig. 1).

Table 1 Concept definition
Full size table

Mutual trust is thus a fundamental property and predictor of high performing teams. During collaboration, team members continuously engage in a process of establishing and (re)calibrating trust among each other. We define trust as “the willingness of a party to be vulnerable to the actions of another party based on the expectation that the other will perform a particular action important to the trustor, irrespective of the ability to monitor or control that other party” [96]. Since the establishment and maintenance of trust is crucial for team performance and given the projected dramatic increase in robots that support teamwork it is crucial to understand how the introduction of such systems affects team trust development and maintenance, and ultimately team performance.

1.1 The Research Challenge

While robotic systems that support teamwork have improved tremendously in the last decade, creating functioning social abilities in a robot is one of the most difficult remaining challenges [70]. A key question in the next decade will be how artificial team members can be tightly integrated into the social structure of hitherto human-only teams.

Previous research on trust in HRTs has primarily focused on identifying initial trust states and potential determinants [9, 53, 126]. A new approach is needed to identify what aspects of a robot’s design and behavior determine the adjustment of overtrust and undertrust states over longer periods of time by analyzing the trust process within a broader perspective of longitudinal teaming. By examining trust as a calibration process between team members collaborating in various constellations throughout a range of tasks, we will understand the crucial role of trust calibration for the incremental refinement of task division, communication, and coordination among the team members.

With few exceptions (e.g. [59, 87, 157]), we have little understanding of the temporal dynamics of trust formation and maintenance, nor of how trust increases or decreases over time as a result of moment-to-moment interactions among HRT members. New approaches to understanding trust are therefore needed and especially those that are affectively grounded [73, 128]. To understand how the introduction of social robots in a team might affect trust development and maintenance, we examine trust development in human–human teams [108, 152]. Even though trust between humans and robots may not be tantamount to trust among humans [87], we may still draw insights from human–human trust development frameworks [29]. In this paper, we draw inspiration from the work by Gottman on identifying healthy and unhealthy relationship patterns [67, 68]. Gottman’s research charted the dynamics of trust in couples by analyzing moment-to-moment interactions over longer periods of time, and identified specific trust repair strategies to be used when trust was too low, and trust dampening strategies when trust was too high. We believe a similar approach could be fruitfully applied to HRTs when investigating how relationships with artificial teammates can evolve over longer periods of time. This is particularly relevant given recent evidence that points to similarities in how humans establish relationships with machines via the hormone oxytocin [27].

1.2 Paper Overview

1.2.1 Main Contributions

The conceptual and computational modeling of trust is a research topic that has been thoroughly researched in the past and is currently receiving much attention in various research communities. To distinguish our approach from others and to acknowledge some of the limitations of our approach we describe here what our paper does and does not address. First, and foremost, our model concerns the future autonomous social capabilities of robots. Several other approaches focus on task-specific trust or adaptive trust calibration approaches that measure trust passively and then adjust to the operator. Our approach is unique in the sense that it proposes a future where robots function with social human-like abilities. Second, our model serves as a of meta model to a number of isolated trust process models previously proposed. This has the advantage of allowing for a broader outlook of trust in human–robot teams and allows for a scalable approach that can describe longer-term human robot interactions. Third, although our model as it stands is not ready for computational implementation, it may be amenable to a variety of computational approaches that can be implemented. There are many models available that specify the formulas and computational models required from possible or existing implementation. Fourth, we deliberately did not specify the measurement of trust for this model. Trust is a multi-faceted concept and measurement approaches vary greatly between disciplines. The concepts proposed here are theoretical and could be interpreted and measured in a variety of ways. The strength of this approach is that our model allows for flexible and diverse application and implementation. Lastly, trust in robots involves a number of different variables. The work described in this paper focuses on relationship equity and those variables believed to affect longitudinal trust development. We have excluded a number of other variables that may be important in non-social situations.

Fig. 1
figure 1

Overtrust, undertrust, and calibrated trust as a function of perceived trustworthiness versus actual trustworthiness

Full size image

1.2.2 Overview

We start by presenting the human–robot team (HRT) trust model, to describe how iterative collaboration helps team mates to incrementally construct accurate models of one another’s ability, integrity, and benevolence, and how trust calibration can contribute to this process. Thereafter, throughout the rest of this paper, we go through the various elements in the HRT trust model to describe a thorough and integrated theory of how trust develops over time as a result of a series of one-shot interactions. During each one-shot interaction, the robot determines whether or not it might cause a trust violation, i.e. behave in a way that is not in line with its team member’s trust stance. This allows the robot to engage in active trust calibration by using a social signal detection theory. Through the application of our presented design guidelines, the designer of the robot may determine what type of behaviour the robot should use to calibrate trust either in advance of the potential trust violation, or afterwards in case of a false detection or a miss. Long-term interaction is described as a repeated series of one-shot interactions, the outcomes of which are stored in a relationship equity bank that builds up (or breaks down) depending on whether trust is either violated or complied to. The theory presented in this paper allows for a range of propositions that can be tested and validated by implementing the proposed model and investigating the effects observed when humans team with trust-calibrating robots. We state each of these propositions at the end of each described theoretical component of the model.

2 A Model for Longitudinal Trust Development in Human–Robot Teams

Figure 2 presents a new model explaining the role and process of establishing longitudinal social trust calibration throughout the life cycle of an HRT. The HRT Trust Model describes the development and role of trust calibration in HRT collaboration. HRT consists of four parts including 1) Relationship Equity, 2) Social Collaborative Processes, 3) Perceptions of Team Partner, and 4) Perceptions of Self.

Fig. 2
figure 2

The Human–robot team (HRT) trust model. The collaboration itself is represented in the middle of the figure, describing how each action from either of the team members adds to or takes from the relationship equity bank, and how the level of this bank influences the preferred way of collaboration, i.e. through informal and implicit agreements, or through formal and explicit agreements. The blue-grey boxes represent the passive trust calibration process, whereas the yellow boxes describe the active trust calibration process

Full size image

2.1 Relationship Equity (Light Blue)

Central to our model is the idea of relationship equity which describes the cumulative result of the cost and benefit relationship acts that are exchanged during repeated collaborative experiences (including social and/or emotional interactions) between two actors. The concept is somewhat similar to the notion of social capital [11] and goodwill [41]. It is also somewhat inspired by equity theory as part of social exchange and interdependence theory [61]. While our concept of relationship equity is primarily the difference between the cumulative costs and benefits between two partners, equity in this theory refers to whether the ratio of relationship outcomes and contributions is equal between partners. Unbalanced ratios cause relationship distress.

2.2 Social Collaborative Processes (Red and Green)

The middle part of the model describes the collaborative task performance between the teammates. Together, they perform a joint activity with the purpose of achieving a common goal. Collaboration is risky: actions may fail and circumstances may change. Therefore, the individual actors monitor the behavior and collaboration of themselves and their teammates. Based on their observations, they aim to establish appropriate trust stances towards one another, so as to mitigate the potential risks involved in accomplishing the joint task (also see Sect. 2.3). This trust stance allows actors (both human and robot) to decide on safe and effective ways to collaborate on the current task with the current team constellation. Based on the trust stance, a teammate may decide to rely on a combination of formal, explicit work agreements (especially in cases where relationship equity is low) and informal, implicit collaborative agreements (especially in cases where the relationship equity is high). Both types of collaborative agreements aim to improve the team performance, for instance, by mitigating risk, compensating for one another’s limitations, coordinating parallel activities, or communicating information relevant to the team [20, 21, 84, 92, 150].

2.3 Perceptions of Team Partner (Grey)

The blue-grey boxes indicate the passive trust calibration process: Based on team members’ perceptions of one another, actors predict one another’s trustworthiness. Taking into account their current formal work agreements and informal way of collaboration, they then (sub)consciously assess the risk involved in the collaboration as it currently is, and decide upon a trust stance towards one another [16, 90]. They then may decide to adjust their collaboration to mitigate the assessed risks, for example by proposing formal work agreements or by relaxing the existing work agreements. During the next collaborative occasion, the actors obtain additional information concerning their team member’s trustworthiness. This information may deviate from the original prediction, resulting in a prediction error, or miscalibration.

Adequately calibrated trust stances among the team members lead to more effective collaboration: Overtrust can condition team members into complacent states and misuse, whereas undertrust can cause inefficient monitoring and unbalanced workload. In other words, trust calibration is crucial for optimal team performance. Through the feedback loops described in the model, the HRT trust process leads to continuous incremental updates of the team members’ trust stances towards one another and an overall reduction of miscalibrations. We assume that, for team members that are benevolent and sincere, the development of appropriate trust stances will benefit their collaborative efforts; team members can compensate for each others’ flaws, while relying on each others’ strengths.

2.4 Perceptions of Self (Yellow)

The yellow boxes indicate the active trust calibration process: This process is based on an actor’s awareness concerning their involvement in team trust calibration. This awareness enables both actors to engage in deliberate attempts to influence, aid, or hamper the trust calibration process. This is achieved first and foremost through the formation of a theory of mind, allowing an actor to reason about the other actors’ mental models. If the actor concludes, based on their self-confidence and their theory of mind, that another team member may be mistaken about their performance level, the actor may decide to actively intervene in the trust calibration process, through a relationship regulation act, such as an explanation or an apology.

The next few sections describe the various parts presented by the HRT trust model in more detail.

3 Relationship Equity: Benefits of Building Trust Over Time

Relationship equity represents the interaction history between two actors and is the cumulative positive or negative assessment with respect to the relationship between the actors. Relationship equity affects future perceptions of trustworthiness and the trust stance by functioning as a lens through which future interactions are perceived and interpreted. Relationships that have accumulated a lot of positive equity may be able to absorb trust violations, without stirring the relationship equity all that much. Alternatively, relationships showing negative equity may be rattled even by small trust violations compared to relations that have positive equity. We believe relationship equity is a critical construct that is needed to predict long-term human–robot interaction. The relationship equity between team mates is influenced not only by the collaborative experience itself but can also be actively and deliberately affected through relationship regulation acts, as we will see during the last two steps in the HRT trust model.

The feedback loops (passive and active trust calibration) presented in Fig. 2 occur continuously while interacting with other actors. The feedback obtained from these loops is remembered and stored in what we propose as a new construct known as relationship equity (also see Fig. 3).

Fig. 3
figure 3

The core of the HRT trust model is the relationship equity bank, which accumulates the net result of repeated interactions over time

Full size image

The first part of this section focuses on healthy human relationships that are maintained by partners who actively engage in relationship regulation acts to contribute to relationship equity [67, 68]. Research on human–robot interaction has applied some of the core concepts related to this work to HRTs to explore the similarities and differences [24,25,26,27].

3.1 Emotion Regulation as the Key Activity to Build Relationship Equity in Human Relationships

Emotion and emotion regulation play an important role in the formation of trust [33, 151]. Expressions of emotion are a crucial mechanism by which people determine how to relate to each other and whether to trust each other. Social functional accounts of emotion highlight this important role of emotional expressions by conceptualizing them as “interpersonal communication systems that help individuals navigate the basic problems that arise in dyadic and group relations” [103]. Interactions thus involve a constant coordination on affect as participants of an interaction jointly determine how behavior should be interpreted and responded to [73]. Some emotional expressions increase interpersonal trust while others reduce it. For example, negative and especially hostile expressions, such as anger and contempt, have been found to impair trust formation [2, 103], whereas positive expressions, e.g. expressions of embarrassment, have been found to facilitate trust formation [83].

Fig. 4
figure 4

Regulated and non-regulated point graphs

Full size image

Researchers on emotion expression in couples suggest that in order to understand trust formation and maintenance, we need to take the temporal dynamics of emotion expression into account [66, 68]. This important role of temporal dynamics for our understanding of trust formation has also recently been highlighted in the area of human–automation interaction, e.g. [157]. A point graph method can be used to account for different temporal dynamics of emotion expressions [48]. Point graphs plot the cumulative sum of positive minus negative expressions over time (see Fig. 4), describing emotional interaction dynamics over time. An upwards directed point graph (regulated) indicates an interaction in which participants are able to shape the emotional dynamics in such a way that more positive than negative expressions are consistently produced, whereas a downwards directed point graph (non-regulated) indicates an inability to do so. While interactions in couples exhibit different levels of intensity, research shows that as long as a surplus of positive over negative behavior can be maintained (regulated interaction) it has positive consequences for short and long-term outcomes.

Evidence from several studies suggests that the ratio of positive to negative behaviors assessed through the point graphs generalizes as a predictor of outcomes from couples to teams. For example, one study showed that this ratio predicted satisfaction with group membership and team performance [75]. Further studies [43, 72, 89] all demonstrate the importance of the ratio of positive to negative expressions for team performance.

The point graphs also highlight that each positive or negative behavior exhibited by participants of an interaction has a cumulative impact on the relationship. An interaction that is characterized by mostly positive behaviors is likely more resilient towards occasional negative or hostile behaviors than an interaction that has been less positive over time. We use the term relationship equity to refer to the idea that trust is built up through moment-to-moment exchanges of positive and negative behaviors that accumulate over time. The term “equity” highlights that the impact one negative behavior has on the relationship depends on the equity accumulated throughout prior exchanges. Operationally, as suggested also by [157], relationship equity is best understood as the area under the curve of a point graph as shown in Fig. 4 [157].

When integrating robots into teams it is crucial to understand how their presence and behavior influences a team’s exchange of positive and negative behaviors and, through that, overall trust formation. Currently, only little is understood about how robots influence the dynamics of the teams they are embedded in [74, 76]. Recent work, however, has shown that robots can actively shape interaction dynamics in teams through repair of negative behaviors [75], through the expression of vulnerable behavior [135], or through the expression of group based emotion [19]. It is thus important to understand not only how robots influence the formation of a team’s relationship equity through their behavior, but also how robots might be used to actively regulate interpersonal exchanges to promote relationship equity buildup and trust formation.

Proposition 1

When designing a robotic team partner, it should have access to a relationship equity bank that allows the robot to maintain an understanding of the current relationship equity, rising with each positive relationship act, and falling with each negative one.

3.2 Relationship Equity as a Predictor for Team processes that Manage Risk

3.2.1 Formal and Informal Work Agreements

Based on their relationship equity and current knowledge of one another’s capabilities, the team members may jointly define a series of work agreements [102, 107]. Work agreements ensure smooth collaboration between team members by explicitly defining collaborative agreements, such as communication, coordination, and task allocation between the team members. Work agreements evolve over time as they are either learned implicitly through training or collaboration, or formed explicitly through formalized rules in the form of obligations and permissions/prohibitions. Scientific research on the formalization of work agreements (in the literature often referred to as “social commitments”) comes form the field of normative multi-agent systems, where work agreements are used to support robustness and flexibility [17]. According to this framework, a work agreement is an explicit agreement made between two parties, denoted as a four-place relation: \(\langle debtor,creditor, antecedent, consequent \rangle \), where the debtor owes it to the creditor to effectuate the consequent once the antecedent is valid. Work agreements can, for instance, be used to specify the extent to which team members monitor and check one another’s work and/or ask for permission to continue with their next activity or task before doing so. In sum, work agreements can be used to mitigate the risks involved in collaboration (assessed based on the actor’s current knowledge of its team members’ capabilities), by introducing rules that restrict the team members in their autonomy, especially when it comes to task allocation, assessment, and completion. For more information, please refer to related works, such as [22, 82, 102].

Proposition 2

Relationship equity will negatively predict the degree to which formal work agreements will be constructed as a method for reducing risk. Lower degrees of relationship equity will predict more formal work agreements.

3.2.2 Collaboration

The team proceeds to collaborate in compliance with the work agreements. During collaboration, the actors observe one another’s capabilities and inspect one another’s deliverables and performance. Based on their observations, the agents continuously update their mental models and corresponding trust stances. This potentially leads to revisions of the work agreements.

Collaboration does not merely entail the team’s ability to adequately perform the task. Collaboration is also characterized by social interaction between team members, i.e. non-task related peripheral interactions with team members (jokes, humor, being able to honestly call each other out, informal things). Being able to show vulnerability at work, especially during non-work related interactions, has been shown to increase team effectiveness, as it facilitates positive trust development [37, 39].

Proposition 3

Relationship equity will positively predict the degree to which informal collaboration will occur as the primary manner of interaction between team members. In a team with a high relationship equity (through social interaction and successful past team performance) the need for regulative formal agreements and corrections will decrease, leading to a less controlling atmosphere in the team environment.

3.2.3 Team Trust Dynamics

Unique trust effects can occur within a team as a result of feedback loops. For example, ripple effects can occur when one behavior of a team member is copied by another. In a striking example of this type of effect, a robot expressing vulnerability caused other human team members in the group to express vulnerability as well [135]. Another example are spiraling effects when negative behaviors and especially hostile behaviors have a tendency to be reciprocated and trigger a spiral of increasing negativity with detrimental consequences for trust [2]. Finally, maladaptive feedback loops can occur when team members are simply out of sync with one another. For example, a teammate A may try to compensate for a perceived failure that teammate B is not aware of. The lack of reciprocation by teammate B may cause frustration by teammate A that leads to confusion in teammate B. The lack of empathy by teammate B may inspire even more frustration in teammate A. What started as a minor miscommunication and ensuing adaptation strategy can cause maladaptive feedback that further escalates the situation.

Proposition 4

In our model, ripple and negative spiral effects can occur as a result of subsequent reactive tit-for-tat behaviors. We predict that these will be positive with higher relationship equity and negative with lower relationship equity. Maladaptive feedback loops may occur when there is sustained miscalibration between the two actors.

4 Minimizing Social Calibration Errors as a Way to Build Relationship Equity

This section presents a signal detection approach to social trust calibration, proposing that trust violations can either be correctly anticipated (hit), incorrectly anticipated (false alarm), incorrectly unanticipated (miss), or correctly unanticipated (correct rejection).

While many of the terms described in our team trust model have been researched extensively, the proposed concept of social trust calibration is new. The focus of the current section is to describe a signal detection approach for social trust calibration with the goal to create intelligent systems that can recognize when their social behavior may cause a trust violation, e.g. when a machine performs unlike its usual performance standards, or when it fails to meet others’ expectations.

Fig. 5
figure 5

Passive trust calibration between team partners

Full size image

4.1 Social Trust Calibration

When teammates collaborate, they engage in a constant process of social trust calibration (see Fig. 1). Calibrated trust between team members is defined such that someone’s perceived trustworthiness of a teammember matches that teammember’s actual trustworthiness. When looking at the passive trust clibration part of our model, depicted in Fig. 5, perfectly calibrated trust would mean that the prediction error (miscalibration) is 0. Undertrust is defined as the situation in which the trustor has lower trust in the trustee than the trustee deserves. In situations of undertrust, the trustor fails to take full advantage of the trustee’s capabilities. In teams, situations of undertrust can result in suboptimal solutions to problems, a lack of communication, and increased workload for individual members (as opposed to distributed workload). Undertrust can be detrimental to the effectiveness and efficiency of the HRT, as it may lead to disuse or micro-management. Actors can also trust each other too much. Overtrust is defined as the situation in which the trustor trusts the trustee to a greater extent than deserved given the trustee’s true capabilities. In situations of overtrust, the trustor allows the trustee to act autonomously, even in situations where the trustee is not capable of performing the task adequately. Overtrust is a dangerous condition that should be avoided for all critical systems because it can lead to disastrous outcomes. Many examples exist of accidents caused by overtrust [6, 45, 111, 133]. Overtrust can be hazardous as it may lead to a lack of guidance and control for systems not fully capable of performing a given task.

Proposition 5

Socially calibrated trust will increase relationship equity. Socially miscalibrated trust will decrease relationship equity.

Fig. 6
figure 6

Active trust calibration between team partners

Full size image
Fig. 7
figure 7

Over time, an actor can use its predictions about itself and its impact on the other actor’s trust stance to dampen or repair trust, or to prevent breaking trust. Credit for illustration: USC Viterbi School of Engineering, with permission from Prof. M. Matarić [49]

Full size image

4.2 A Signal Detection Model for Social Trust Calibration

Trust (mis-)calibration may occur in any team situation with a great potential for frustration. Since trust violations have a detrimental effect on trust, it is vital to minimize their overall impact. Early detection and accurate awareness of potential trust violations may allow team partners to engage in active trust calibration (see Fig. 6), so as to prevent escalation of minor issues into larger problems within the HRT. Accordingly, we created a simple model of anticipated and unanticipated trust violations with the use of trust repair, dampening, and transparency methods. We may imagine four situations depending on whether a trust violation is anticipated and whether a trust violation occurred using a signal detection classification approach (see Fig. 7). A “Hit” situation may be one where a trust violation is anticipated prior to the occurrence of the trust violation and a trust violation also occurs. A system may actively attempt to lower expectations in anticipation of expected failures prior to the attempted action. Such an action would effectively dampen trust and may more appropriately calibrate trust. When the trust violation occurs, the robot can refer back to the lowered initial expectations. A “False Alarm” situation is one where a trust violation is anticipated, but does not occur. It may not be bad overall to dampen expectations that are not borne out in an actual trust violation. Lowering expectations may alert the operator to guide the interaction more. However, dampening without the trust violation occurring may result in the system downgrading itself unnecessarily which may lead to reduced trust over time. A “Miss” situation is one where no trust violation was anticipated, but one occurs. This is the situation where trust repair activities are most needed since a trust violation was not anticipated. This may also be the situation that occurs most frequently in team interactions. The “Correct Rejection” situation is one where no trust violation is anticipated and no trust violation occurs. No specific action is required. General transparency methods can enhance this baseline situation in a passive, but proactive manner.

The signal detection model focuses on single-shot interactions, yet when looking at longitudinal teaming, one should actually be looking at a series of single-shot interactions, that accumulate over time, as outlined in Sect. 3.1. Figure 4 displays how relationship equity may be constructed (or broken down) as a result of repeated interactions. Ultimately, the relationship equity bank, described in Sect. 3.1, serves as a lens through which each subsequent interaction is reviewed, and thereby mediates trust calibration and mental model updating, as described by the HRT trust model presented in Fig. 2.

Proposition 6

Social hits and correct rejections will decrease the probability of mis-calibration. Social false alarms and misses will increase the probability of mis-calibration.

5 Methods for Building Relationship Equity through Social Trust Calibration

In the previous sections, we have outlined components of a theory that describes social trust calibration in human–robot teams. The theory assumes a number of calibration methods, such as trust repair, trust dampening, transparency and explanation. We describe each method in detail in Table 2 as well as in the following subsections.

Recognizing that a potential trust violation is going to happen or that an actual trust violation has occurred is important to determine whether trust calibrating acts are appropriate. Please note that a trust violation can go both ways: either an actor is lucky and performs unusually well, or it fails and performs substantially worse than it normally would. In both cases, trust calibrating acts are warranted. This section introduces a range of actions and utterances an actor can perform to calibrate trust either by mitigating a state of undertrust through trust repair or a state of overtrust through trust dampening. In addition we discuss methods of transparency and explanation as viable options for trust calibration.

5.1 Methods for Trust Repair

Trust repair is a reactive approach to restore undertrust after the machine has made a mistake, caused trouble, or displayed unexpected or inappropriate behavior [7, 29, 94]. Trust repair seeks to repair situations where trust is broken or where there is an initial distrust bias, e.g. by explaining the cause and/or situational nature of a mistake, or making promises about future behavior.

Proposition 7

Trust repair activities will help to increase trust if they are appropriately timed and commensurate with the degree of trust violation.

Table 2 Methods for trust repair, trust dampening, transparency, and explanation
Full size table

5.2 Methods for Dampening Trust

Trust dampening is a reactive approach to quell overtrust after a machine has made a lucky guess, or when a machine makes a mistake that has not been noted by its collaborators or users. Trust dampening approaches seek to lower expectations when too much faith has been put into a machine. Dampening methods may include showing a user what a system failure looks like, showing a history of performance, and providing likelihood alarms [88]. Dampening approaches may need to be applied especially in the beginning of interacting with a new machine. Often, people tend to have high expectations of machines and robots known as automation bias [38]. When actual robotic behavior is observed, people may punish machines more deeply than their human counter parts [47].

Proposition 8

Trust dampening activities will help to stimulate trust resilience by appropriately adjusting expectations in the face of anticipated errors.

5.3 Methods for Transparency

Some methods apply to facilitate both processes of repair and dampening, such as making the internal processes and processing steps more transparent or inspectable for other team members. These methods revolve around the central method of increasing the transparency of a system [13], such as its performance, its process, and its intent and purpose [3, 13, 14, 99]. The benefits of increasing transparency have been demonstrated through empirical research. Previous work has emphasized design approaches to increase the transparency of the system to a user to promote trust calibration. Transparency design methods focus on conveying trust cues which convey information about an agent’s uncertainty, dependence, and vulnerability [25, 136].

Proposition 9

Transparency activities will help to calibrate trust by providing accurate meta-information about the robotic partner.

5.4 Methods for Explanation

Recently, the research on explainable AI (XAI) has expanded rapidly (e.g. see [23, 51, 104]). XAI refers to (1) the ability to offer a meaningful explanation for a specific human actor when needed, and (2) the ability to ask for an interpretable explanation from a specific human actor when needed. So far, research centered primarily on the first type of ability, often with a focus on a specific (classification) task or machine learning model. However, more integrative methods are evolving, which include both bottom-up data-driven (perceptual) processes and top-down model-based (cognitive) processes ([106]; cf. dual process theories [40, 79]). Such methods could help to assess the trustworthiness of AI output (i.e. robot’s own task performance) and, subsequently, explain the foundation and reasons of this performance to establish an adequate trust stance.

At the perceptual level, the provision of an Intuitive Confidence Measure (ICM) enhances human judgment of the quality of the data processing and corresponding inferences [140, 141]. The ICM explains the likelihood of a correct single prediction (output) based on similarity and previous experiences (e.g. “I am reasonably certain that there is a victim at location A”). At the cognitive level, available models of the user’s goals, beliefs, and emotions can be used to provide explanations that provide the reasons of specific output (e.g. advice) and behaviors (e.g. “It is important to drive around this area, because there is an explosion risk”; cf. [81]). Personalization of these explanations is crucial to accommodate a user’s goal and emotional state [80]. At the perceptual-cognitive level, contrastive explanations provide the reasons of a specific output (the “fact”) in relation to an alternative output that is of interest (the “foil”) [142, 143]. Humans often use this type of explanation. Contrastive explanations narrow down the amount of features that are included in the explanation, making it more easy to interpret [101].

For the construction of meaningful explanations, challenges are to establish at run-time (1) an adequate level of detail, specificity and extensiveness, (2) an effective dynamic adaptation to the human and context, and (3) an appropriate choice for allocating the initiative of an explanation dialogue. The development of Ontology Design Patterns can help to meet these challenges, particularly for an artificial actor’s reasoning and communication [130]. Interaction Design Patterns are being constructed for shaping mixed-initiative communicative acts of explanation [106]. Developing the ability for a robot to ask for an interpretable explanation from a specific human actor when needed, is yet a rather unexplored area.

Proposition 10

Explanation activities will help to calibrate trust by providing accurate meta-information about the robotic partner.

6 Implications of the Framework

The HRT trust model describes the role and purpose of trust calibrating acts, i.e. trust repair and trust dampening, in improving HRT collaboration over longer periods of time. Our model assumes that trust calibration benefits all actors as long as they are sincere and benevolent in their collaboration. If accurately executed, trust calibration results in optimized collaboration through the adoption of implicit and explicit work agreements that appropriately benefit from the strengths of the actors involved, while mitigating risk by compensating for team members’ shortcomings and/or limitations.

From our model, there are a number of different research directions that can be explored given the longitudinal interaction between humans and artificial team members (e.g. robots, agents, and other AI-based systems) that actively employ trust calibration methods. Some of these topics have been raised in other publications [27], each of which could merit its own research program with a set of experiments. We discuss several of these research directions as well as the implications of our framework next.

6.1 A Common Framework for Mixed Human–Robot Teams

The first implication of our model is that it presents a flexible and common way of modeling relationship equity as it relates to trust calibration in human–human, human–robot or robot–robot teams. This is useful because currently the social capabilities of robots are still limited, but expected to greatly improve in the next few decades. For example, if a robot can function as a human in this type of relationship, the robot would be expected to have models and an understanding of its own behavior, its teammate, and the process of collaboration itself. Progress indeed has been made in each of these areas, but work remains to build the types of teams that can regulate emotion and manage relationships in mixed human–robot teams. Currently, these relationships are asymmetrical where the human compensates for the lack of a robot’s social abilities. For longer-term interaction to be sustained, those deficiencies will have to be resolved.

6.2 Adaptive Trust Calibration Systems for Mixed Human–Robot Teams

The second implication of our work is that it provides a step in the direction of mutually adaptive trust calibration systems (ATCS). These systems are a special form of adaptive automation [10, 42, 63, 78, 129] that can measure the trust state of a human and then adapt their behavior accordingly to provide a positive impact on team performance. When implemented well, these systems have the potential to calibrate trust on the fly and provide immediate benefits for human–robot team performance.

In recent pioneering work that exemplifies ATCS, researchers designed and developed a robot that calibrates trust through automatically generated explanations [149]. Especially when the robot had low capabilities, its explanations led to improved transparency, trust, and team performance. This important work shows how the impact of expected trust violations can be mediated with the use of a trust dampening strategy, i.e. explanations. Other pioneering work demonstrating the utility of ATCS has used various modeling techniques to incorporate trust measures and adapt team performance. For instance, Chen et al. (2018) [15] used trust-POMDP modeling to infer a human teammate’s trust and only engage in moving a critical object when a human teammate has built up enough trust in a robot’s arm’s ability to move objects carefully.

With the use of our model, researchers have a means to place this work in a larger context of challenges related to trust development in HRTs. This framework may therefore serve as a guide by providing an overview of what parts the research community currently understands and/or is able to successfully implement in an artificial team mate, and which parts still require additional research.

6.3 Towards Long-Term Interaction for Human–Robot Teams

There is much to be learned about effective team behavior in general, both in human-only teams and teams with a mix of humans and artificial team members. By achieving a better understanding of trust dynamics, researchers may learn a great deal about the impact of particular team behaviors on the trust relationships within a team, e.g. what behaviors contribute to team effectiveness and what behaviors are detrimental.

More importantly, one might be able to recognize negative team behavior patterns, such as the ripple effect or downward spiraling mentioned in Sect. 3.2.3. In this way, artificial actors can be designed with a range of individual behaviors serving as subtle interventions that help steer away from such destructive patterns and simultaneously promote healthy, emotionally regulated, and effective team behavior patterns [144].

In addition, the issues below are primary research challenges that researchers could address in the future.

6.3.1 Ontology and Model Development

There is a great need to develop a method that allows for reasoning about humans, agents, and robots alike. For this purpose, we are developing an ontology, that provides the vocabulary and semantics of multi-modal HRT communication and the foundation for automated reasoning [5, 77, 114,115,116]. Furthermore, a computational model of trust development and repair -providing quantitative predictions of potential trust violations and their prospective impact- would help to focus and integrate research within the HRI community. Progress towards this goal has already been made by research that has indicated how trust can modeled to extend across tasks [134].

6.3.2 Trust Measurement and Modeling Development

Trust measurement remains a key issue for HRT interaction [34, 35, 44, 53]. It is important to know how trust is initiated, how it develops, how it breaks down, and how it recovers. This requires a convergence of behavioral, self-report, observational, and neuro-physiological measures and correlates [28, 53], as well as the development and validation of new measurements specific to the process of HRT trust. Recent work has demonstrated the multi-faceted nature of trust as it relates to delegation decisions [155] as well as the ability to model trust with dynamic Baysian models [156].

6.3.3 Implementation and Experimentation

In addition to the conceptual and computational models, it is important to develop software modules for robots and other artificial team members enabling them to compute and/or reason about trust and engage in trust calibrating acts where needed or appropriate [138, 139, 145]. This will also allow for hypothesis testing based on predictions and prescriptions provided by our model.

6.3.4 Validation and Verification

Endeavors of implementation and experimentation will facilitate theory development and refinement as we gather empirical data on the actual effectiveness of trust calibrating acts, as well as relationship equity models and predictions of potential trust violations, all situated in field exercises with prospective end users [91].

7 Conclusion

Future societies will rely substantially on human–robot teams (HRTs) in a wide variety of constellations. Enabling such teams to effectively work together towards the achievement of shared goals will be paramount to their success. The theory, models, methods, and research agenda presented in this paper will contribute to this endeavor, and may lead to the design and implementation of artificial team members and corresponding team behaviors that support healthy trust development, in turn contributing to high performing HRTs.