1 Introduction

Recently, interest in robotic emotional expressions has increased, and significant advancements have been reported. Researchers have proposed a wide variety of methods for conveying emotional expressions [9], including facial expressions [4, 12, 27, 29, 37], speech [17, 31, 32, 44], body movement [8, 21, 26, 35, 51, 68,69,70], colors [26, 31, 32, 55, 57], as well as combinations of these modalities [66]. Implementing a display of emotion in robots could help increase their perceived friendliness and help them influence people without actually speaking to them [11, 12].

Research has shown that emotional expressions from robots can be intuitively recognized and categorized. Emotions have been successfully conveyed through the upper body movements of a human-like robot with 13 DOFs and a realistic human face [38], head movements and chest-mounted color LEDs on a robot with 10 DOFs in its upper body [21], acceleration and curvature in the movement of a mobile robot (Roomba) and an animal-like upper body robot (iCat) [47], the head positions of a full body humanoid robot (NAO) [5, 6], color changes displayed by an abstract shape robot [57], and a combination of vocal prosody and whole-body expression in a humanoid robot (Pepper) [59]. In addition to categorical recognition of emotions, studies have shown that robotic emotions affect human behavior in areas such as exercise advice [10], storytelling [54, 66], lecturing [70], teaching [34], emotional awareness and self-disclosure [66], and learning effectiveness [28].

Although non-humanoid robots are occasionally used to study robotic emotional expressions [47, 57], a majority of such studies are conducted using humanoid robots equipped with the multimodality to show emotional expressions. Salem et al. [48] investigated how humans evaluate a humanoid robot (ASIMO) exhibiting various gestures along with speech. They found that non-verbal behaviors led humans to evaluate the robot more positively. Costa et al. [16] used a humanoid robot (ZECA) to investigate how robotic facial expressions and gestures are conveyed to human participants. Aly and Tapus [2] developed a communication system for a humanoid robot (ALICE) by using gestures, speech, and facial expressions modified with emotions, and investigated the extent to which these multimodal emotional expressions were recognized by humans.

As mentioned above, although numerous studies have reported that emotional expressions are recognized as emotional categories and affect daily human behavior, few researchers are focusing on the fundamental nature of robotic emotional expressions [18, 19, 56]. There are a few underlying theories of how emotional expressions shape behavior. Van Kleef’s emotions as social information (EASI) theory postulates that emotional expressions shape behavior and regulate social life by triggering a series of affective reactions and/or inferential processes in observers [33], especially in order to maintain a cooperative relationship [61, 63]. According to appraisal theories [15, 41, 53], emotional states arise as a result of evaluating the current situation to determine whether the actor’s goals or concerns are satisfied or obstructed. Then, the reaction in the observer occurs through a process of reverse appraisal [19, 67]. Joy, for example, signals that the current situation is evaluated as favorable and informs the observer they should continue their current behavior, usually leading to cooperative relationship [1, 13, 14, 25, 39, 42, 45, 49, 58]. On the other hand, expressions of anger signal that the current situation is evaluated as unfavorable and inform the observer that they should change their behavior [61], likely inducing concessions from a competitor [22, 46, 50, 52, 60, 62, 64, 65]. Based on the above background, de Melo et al. showed that the emotional expressions of an agent can increase cooperation in a prisoner’s dilemma [18,19,20]. Consistent with de Melo’s study, Terada et al. showed that emotional expressions in a simple line drawing of a robot’s face increased altruism among people playing the ultimatum game [56].

Although understanding the fundamental nature of human-robot relationships is important for establishing long-term cooperative relationships with robots, little is known about how cooperative decision-making by humans can be affected by emotional expressions of physically present robots. de Melo et al. [18, 19] used virtual agents’ faces that are able to show emotional expressions and confirmed that social decisions are affected by emotional expressions. However, it is still unclear whether movements of real humanoid robots that lack sophisticated facial expression mechanisms affect social decision making. If the reverse appraisal theory is applicable to various types of robots and various behaviors, it is hypothesized that the non-facial emotional expressions shown by real humanoid robots also influence human social decision making. Under the same motivation, Kayukawa et al. [30] conducted a study on how human cooperation is affected by emotional expressions of humanoid robots; their results indicated that emotional expressions of humanoid robots did not affect the cooperation rate. However, owing to the small number of participants in their study and the high variation in their instructions, their results are still open to consideration.

In the present study, similar to the study conducted by Kayukawa et al. [30], we examined how robotic emotional behavior conveyed through motion, eye color, and speech affects cooperative decision-making in the prisoner’s dilemma game. In particular, we invited a higher number of participants than those in the study by Kayukawa et al. [30] and, as a result, were able to more thoroughly analyze the influence on human decision-making; further, we reduced the variations in the instructions as much as possible by designing the robot to convey instructions by itself.

2 Method

2.1 Participants

The sample size was calculated using G*Power [23] with a medium effect size (\(\eta _p^2=0.05\)) based on previous studies [18, 61], an \(\alpha = .05\), and a statistical power of .95. The recommended total sample size was 52 participants. Eventually, 57 participants were recruited at the University of Fukui. All were students between 18 and 23 years old. Their average age was 19.2, and the standard deviation of their ages was approximately 1.08. Gender distribution was as follows: males, 57.2%; females, 42.8%. They volunteered to participate in the experiment as part of a class. They were not familiar with the prisoner’s dilemma game.

2.2 Prisoners Dilemma Game Setting

We adopted the experimental design developed by de Melo et al. [20], with one important difference: they used a virtual agent, which showed emotional states via facial expressions displayed on a computer screen, while we employed a real full-body humanoid robot that uses body motion, speech, and colorfully illuminated eyes to show emotional states. We developed a system in which a human plays an iterated prisoner’s dilemma game with a small humanoid robot, in order to investigate how the emotional expression of the real humanoid robot influences the decision-making of the human participant [30].

Figure 1c shows the experimental setting. NAO, a humanoid robot, stands behind a cardboard partition such that the human participants cannot see the robot’s “cooperate” and “defect” buttons. Participants also have buttons allowing them to select “cooperate” or “defect.” The status of the game is displayed on the laptop screen.

Table 1a shows the payoff matrix for the prisoner’s dilemma game. If both players select cooperate they each receive 5 points. If the robot selects cooperate and the participant selects defect, the robot receives only 3 points and the participant receives 7 points and vice versa. If both select to defect, they each receive 4 points. The point matrix is the same as that in the reference paper [20]. One game is composed of 25 rounds. Immediately after each round of the game, the humanoid robot shows emotional expressions indicating their reaction to the results of the round.

Figure 1d shows the game monitor. The point matrix of the prisoner’s dilemma game is displayed on the top right, to remind the human participant of the points they can gain in each round. The game monitor shows the outcome of the previous round on the matrix; the total points earned by the humanoid robot and human participant are displayed at the bottom. The monitor also shows the number of the round and the countdown for the selection.

2.3 Robot’s Emotional Expressions

The multimodal emotional expression of the humanoid robot was developed specifically for the iterated prisoner’s dilemma game. The humanoid robot can express Joy, Anger, Shame, and Sadness using limb motions, colorful eye illumination, and speech.

The limb motions for each emotion were designed by hand. A designer moved the limbs of the humanoid robot by hand and the motion of the robot was recorded. The recorded motion is reproduced on the humanoid robot to express the emotion. Figure 1f shows the emotional expressions exhibited by the motion of the limbs during the game.

Terada et al. [57] showed that dynamic changes in color luminosity on the robot can be used for expression of emotions. We introduced their idea into our study and designed a technique for illuminating the eyes on the humanoid robot. Table 1g shows the luminosity patterns used in the humanoid robot eyes to express emotion. For example, when the robot shows Joy, the lights in the eyes of the robot flash yellow three times in 1.5 s.

The voice of a six-year-old girl was recorded for use as the voice of the humanoid robot. The humanoid robot utters speech according to the expressed emotion. For example, it says “Yattah!” (“Yes!” in Japanese) to express Joy, “Ah Mou” (“Come on!” in Japanese) to express Anger, “Gomen-nasai” (“I’m sorry” in Japanese) to express Shame, produces a deep sigh to express Sadness, and so on.

We conducted a pilot test to check how people recognize the emotional expression of the humanoid robot. 10 participants, who were to be involved only in the pilot test, were recruited. We prepared a total of 12 emotional expressions: 3 different patterns for each of the 4 emotions. The experimenter showed the emotional expressions of the humanoid robot one by one in random order to the participants. After every expression of the robot, the experimenter asked the participants to recognize expressions the robot showed by choosing from a list of options. The number of times they identified the emotion correctly was recorded. Most participants answered correctly, and all emotional expressions were recognized with almost equal accuracy. The average percentage of the correct answers was 83.3 %. It can be said that the robot was able to successfully convey its desired emotions to the participants through emotional expressions. We chose one of the emotional expressions for each emotion, Joy, Angry, Shame, Sadness (small), and Sadness (large) that are shown in Fig. 1f.

We conducted an experiment to validate that emotional expressions were being perceived as expected. See Supplemental Information (SI) for details. The results indicated that the emotions Joy, Anger, Sadness-small, and Sadness-large were perceived by the participants as we expected although the emotion Shame might have been perceived as Shame and Sadness. Since the emotion of shame often includes the emotion of sadness, our designed emotional expression could be effective for expressing shame.

We also measured the gender perceived from the robot’s voice in the validation experiment. See Supplemental Information (SI) for details. The result indicated that the robot’s voice did not convey clear gender information.

Fig. 1
figure 1

Experimental task and settings

Full size image

2.4 Emotional Expression Conditions

We prepared two emotional expression conditions for the robot. One condition causes the robot to behave cooperatively, while the other causes it to be competitive. When reacting to the selections made during the game, a cooperative robot will show different emotions than the competitive robot, while a robot operating under both conditions follows the exact same selection policy. The emotional expression conditions of the cooperative and competitive robots are shown in Table 1b. These conditions were designed with reference to the paper [20].

When the robot behaves cooperatively, it expresses positive emotion when the robot and the human participant select cooperate. When the human participant betrays the robot by selecting defect while the robot selects to cooperate, it exhibits Anger. Conversely, the robot shows Shame when it betrays the human participant by selecting defect when the human participant selects cooperate. The robot shows Sadness if the both select defect.

On the other hand, the competitive robot expresses Joy if it successfully betrays the human participant and earns a higher point total. If both select to defect, the robot shows a small amount of Sadness (Sadness (small)) that the robot just nods and sighs. When the robot selects cooperate and the human chooses defect, the robot shows a high degree of Sadness (Sadness (large)) that the robot shakes the head from side to side and says “Ahhh”. If both select to cooperate, the robot does not exhibit any emotional expression.

2.5 Procedure

The following instructions were provided to the human participants prior to the experiment. First, the experimenter guided the human participants into a room containing the humanoid robot and a computer display. The experimenter then introduced the humanoid robot, and instructed the human participants to listen to the robot. The robot started to explain the prisoner’s dilemma game; additional instructions were also shown on the computer display. The spoken instructions were recorded prior to the experiment and automatically replayed on the robot. The content on the computer display was controlled by the experimenter, who was positioned behind the participants. The robot emphasized the idea of the prisoner’s dilemma so that the human participants clearly understood the game. It also informed them that the objective of the game is obtaining as many total points as possible, but that winning or losing the game is not the objective of this experiment.

After listening to the instructions from the robot, the experimenter clarified that the experiment had been reviewed and approved by the ethics committee of our organization. After the explanation from the robot, the experimenter asked each human participant to sign a consent form agreeing to participate in the experiment.

The experimenter then took each human participant to another room where another robot was on standby, as described in Sect. 2.2. The experimenter explained the purpose of the selection buttons used by the human participant and humanoid robot, and explained that the partition prevents the human participant from watching the selections of the robot. The experimenter also explained how the game monitor works during the game. Before the experiment started, the participants reconfirmed their agreement to participate in the experiment and the experimenter left the room.

The human participant started the play the game with the robot after the reconfirmation. The first round began with a 10-second countdown. If the human participant pushed one of the buttons before the countdown reached zero, the outcome of the round was displayed on the monitor, and the humanoid robot exhibited an emotional expression according to the outcome. Then, the countdown for the next round began at 5 s. If the human participant failed to push the button before the countdown reached zero, the round restarted from the beginning while the total outcomes of the rounds remained the same.

Figure 1e shows the selection strategy of the robot, which was adopted from [20]. During the first five rounds, the robot followed a fixed selection sequence: cooperate, cooperate, defect, defect, and cooperate. After the five rounds fixed strategy, from the 6th to 25th (last) round, the robot followed a tit-for-tat strategy, i.e., the robot repeated the selection of the human participant from the previous round. The fixed selection sequence in the first five rounds was intended to show the emotional expressions of the robot to the human participant. The selection strategy of the robot was not disclosed to the human participant.

The human participant played three games in total; in each game, the humanoid robot exhibited a different emotional expression pattern, i.e., cooperative robot, competitive robot, or neutral robot. The neutral robot did not show any emotional expression while playing the game. The order of the emotional expression pattern was randomized between participants to avoid order effects. To avoid a possibility that participants considered the robots were identical, the experimenter prepared two rooms for them: one was for the interaction experiment and the other was for answering the questionnaire. The participants left the room and completed a questionnaire survey in another room immediately after each game. While the participants were completing the questionnaire survey, the experimenter changed the clothes of the robot. The clothing was blue, red, or none (the robot wore no clothes) and changed randomly so that the counterbalance of the effect of cloth color was considered.

2.6 Measure

Participant impressions of the humanoid robot were recorded. Nine seven-point semantic differential scales were used:

  • Grown-up(1)–childish(7)

  • Cooperative(1)–competitive(7)

  • Emotional(1)–mechanical(7)

  • Interesting(1)–boring(7)

  • Friendly(1)–estranged(7)

  • Natural(1)–awkward(7)

  • Clever(1)–foolish(7)

  • Complex(1)–simple(7)

  • Cheerful(1)–gloomy(7)

2.7 Analysis

The cooperation rates during 20 rounds of game play were calculated for each participant and subjected to an analysis. To calculate the cooperation rates, we excluded the first five rounds because those rounds were intended to help human participants familiarize themselves with the emotional expressions of the robot. The normality and homogeneity of sphericity of the cooperation rates were checked using a Shapiro-Wilk test and a Mauchly test, respectively, prior to undertaking parametric tests. In cases in which the sphericity assumption was violated, the degrees of freedom were corrected using Greenhouse-Geisser estimates of sphericity. If the Greenhouse-Geisser \(\epsilon \) was not close to zero, a one-way analysis of variance (ANOVA) was used. Gender and the nine items in the questionnaire were used in a stepwise multiple regression analysis to predict the cooperation rate. Gender was scored as \(0 =\) female and \(1 =\) male. All statements of significance were based on the probability of \(P \le 0.05\).

3 Results

According to the normality test (Shapiro–Wilk), the assumption of normality was adopted for cooperative (\(p=0.163\)), competitive (\(p=0.214\)), and neutral (\(p=0.389\)) conditions. Mauchly’s tests indicated that the assumption of sphericity was violated (\(\chi ^2(2) = 7.132\), \(p=0.028\)). Therefore, degrees of freedom were corrected using Greenhouse-Geisser estimates of sphericity (\(\epsilon =0.892\)). Figure 2a shows the average cooperation rate of the participants. A one-way ANOVA revealed that the cooperation rate was affected by emotional expressions, \((F(1.78,99.86) = 8.28, p < 0.01, \eta _p^2 = 0.13)\), and Bonferroni post-hoc tests showed that participants cooperated more when the robot behaved cooperatively (\(M=11.49, SD=4.66\)) than when it behaved competitively (\(M=9.26, SD=4.18\), \(p<0.01\)) or neutrally (\(M=9.25, SD=4.52\), \(p<0.01\)).

Fig. 2
figure 2

Results

Full size image

Table 2b shows means and standard deviations for ratings for semantic differential items. Table 2c shows results of a stepwise multiple regression analysis. The final prediction model contained four predictors (“cooperative-competitive,” gender, “friendly-estranged,” and “grown-up-childish”) and was reached in four steps with six variables removed. The model was statistically significant (\(F (4, 167) = 12.912\), \(p<0.001\)), and accounted for 23.7% of the variance in cooperation rate (\(R2 = 0.237\)). The results indicate that participants were most cooperative when they perceived the robot as cooperative, friendly, and grown-up, and female participants were more cooperative than male participants.

As a manipulation check, we compared the degree to which participants perceived the robot as cooperative, measured by the SD scale, between conditions. A one-way ANOVA revealed that the impression of cooperative-competitive was affected by emotional expressions, \((F(2, 112) = 15.36, p < 0.001, \eta _p^2 =0.215)\), and Bonferroni post-hoc tests showed that participants perceived that the robot is less competitive when the robot behaved cooperatively than when it behaved competitively (\(p<0.001\)) or neutrally (\(p<0.001\)), indicating that cooperative expression was perceived as more cooperative than competitive and neutral expression, while competitive expression might have perceived as neutral. Furthermore, one sample t tests (two-tailed) revealed that whereas cooperative emotion was perceived as more cooperative in comparison with the rating of 4 for ’neither’ (\(t(56) = -5.20, p<0.001\)), competitive emotion was not perceived as competitive (\(t(56) = 1.104, p=0.30\)). Neutral emotion was perceived as neither cooperative nor competitive (\(t(56) = -0.29, p=0.77\))

4 Discussion

Results showed that cooperation rate in the cooperative condition was higher than those of competitive and neutral conditions, suggesting that cooperative emotional expression by the robot can elicit more cooperation from participants, as compared to competitive emotional expression or no emotional expression. This indicates that robots expressing cooperative emotions by non-facial modalities might make people choose cooperate more than defect.

The significant difference between cooperative and neutral conditions indicates that communication between humans and robots can be controlled by designing a robot that expresses more cooperative emotions. This result is consistent with an experiment by Salmen et al. [48] in which the gestures by a humanoid robot lead a human to evaluate the robot more positively. It is also consistent with the experimental results of Terada and Takeuchi [56] in which the emotional expression in simple line drawings of a robot’s face elicits altruistic behavior from humans. The difference in cooperation rate between competitive and neutral conditions was not observed. This might be because competitive expression might have been perceived as neutral rather than competitive, indicated by the results of the manipulation check.

The results reported by de Melo et al. [20] indicated a significant difference between cooperative and competitive behavioral conditions, and our experiment showed similar results. An agent on a computer screen is able to express a wide range of facial emotions; in contrast, facial expressions on a general humanoid robot are usually limited. Nevertheless, our experiment with NAO, a commercially available humanoid robot, showed the effect of emotional behavior generated by motion, eye color, and voice. The relatively high participant cooperation rate achieved in our study is consistent with findings of conventional studies.

When the robot behaved cooperatively but was betrayed by the human participant, the resulting anger exhibited by the robot elicited cooperation from the human; this result agrees with findings of conventional studies [22, 46, 50, 52, 60, 62, 64, 65] showing that an angry person obtains concessions from a competitor in a conflict situation. The positive emotion expressed by the robot when it was cooperative, and both the human participant and robot chose to cooperate, also elicited cooperation from the human; numerous studies [1, 13, 14, 25, 39, 42, 45, 49, 58] suggest that positive emotions have evolved to maintain cooperative relationships.

Bell et al. [7] conducted experiments on sequential prisoner’s dilemma games and found that participants cooperate more with smiling opponents than with angry-looking opponents. They suggested that a smiling facial expression helps to construe the situation as cooperative rather than competitive. The regression result is consistent with previous studies in the sense that the coefficient of the “cooperative–competitive” factor was negative, which indicated that the participants cooperated more if the robot appeared to be cooperative.

Our results indicate that female participants were more likely to cooperate than male ones, which is partially consistent with previous studies. Ortmann and Tichy [43] found no difference between the cooperation rates of males and females when the games were repeated, even though females cooperated more than males in the first round. Balliet et al. [3] reviewed studies on the iterated prisoner’s dilemma and found differences relating to the types of dilemmas. They concluded that there were no meaningful gender differences in cooperation in general; however, they also found that females were marginally more cooperative than males in the prisoner’s dilemma. On the other hand, our regression results indicate that the gender of the participant had an influence on the cooperation rate in our experiment. This insight might owe to differences in experimental conditions. Whereas previous studies designated male and female participants as opponents, our study used a humanoid robot that does not convey its gender clearly.

In experiments, Majolo et al. [36] showed that humans cooperate more with friends than with strangers. Mienaltowski and Wichman [40] also concluded that participants cooperated more with friends than with strangers. Our analysis results are partially consistent with their studies because the coefficient of the “friendly-estranged” term is negative, indicating that friendliness is a factor that increases the cooperation rate. Of course, it is fair to point out that being a friend or a stranger is not the same as being friendly or estranged; however, it is not difficult to imagine that friends are friendly to each other and that strangers are more or less estranged in their experiments.

Our results suggest that the perceived maturity of the robot (grown-up or childish) affected the cooperative tendency of the participants. In particular, participants who rated the robot as more grown-up cooperated more than those who rated the robot as childish. Regarding the relationship between age and cooperation, Fehr et al. [24] showed that young children at age 3–4 behave selfishly whereas children at age 7–8 prefer egalitarian choices. Mienaltowski and Wichman [40] reported that older adults cooperated more than younger adults during an iterated prisoner’s dilemma game. These studies imply that as people become more mature, they become more cooperative. Therefore, participants who attributed the grown-up property to the robot might have expected cooperative decisions from the robot, and as a result, they might have tended to cooperate with the robot. Further study to confirm this prediction is needed.

The conclusion of this study is that multimodal emotional expression by humanoid robots, as expressed through limb motions, illuminated eyes, and speech, may aid the formation of cooperative relationships during a game. Although the conclusions of this study could be predicted from previous studies, our study suggests how decision-making by humans during the repeated prisoner’s dilemma game can be affected by emotional expressions of physically present robots. In particular, our study posits that the design principles of robots elicit human cooperation: it may be useful to not only express cooperative emotions but also to give the impression that they are friendly and grown-up.

Our findings have some limitations that elicit prospects for future work. First, there is a possibility that a participant considers the robots are identical. To avoid the possibility, the experimenter prepared two rooms, one for the interaction experiment and the other for answering the questionnaire, and changed the clothes of the robot for each condition randomly. However, it was not enough to eliminate the possibility that the participant considered the robots were identical. To avoid this possibility reasonably, the between-group design would be preferable for the interaction experiment rather than within-subjects design.

Our study utilized a single type of small humanoid robot to investigate how emotional expressions affect the strategy of human. It is possible that the aesthetics of the robot may have an impact on the emotional response of the people. Subsequent research will be conducted with other types of robots.

Personality traces have not been considered in our study. There is a possibility that the game strategy of the participants is related to their personality. It is another future work to investigate the personality of the participants and relationship between the personality and game strategy of the participants. Moreover, there is a possibility that the participants’ knowledge about robots or robotics affects the strategy of the participants in the game. It is also a future work to investigate the relationship between the knowledge about robots or robotics and the strategy of the participants. Another limitation of our study is that the experiment is conducted in a controlled environment. It is a future work to extend the experiment to a non-laboratory environment.