1 Introduction

With rapid development of modern human–computer interaction technology, computers or robots are required to help people complete extensive and complicated works [1, 2]. Perception, logic and implementation capacities are enhanced to form more and more harmonious human–computer interface. People have natural affective interactions. Computers are hoped to affectively interact with human beings, thus forming more harmonious relationship through self-analysis, self-regulation and self-improvement [3]. To a large extent, it depends on robots and computers’ affective interaction abilities including emotion recognition, understanding and expression.

2 Literature review

Before 1960s, most scholars considered that emotion is just a by-product of physical activity. Until late 1960s, people gradually accepted the unique mechanism and function of affective process through effective promotion of Tomkins and Izard [4]. The function consisted of adaptability, motivation, organization and communication. Recently, the institute for scientific research proved that emotion is an important part of intelligence. In 1994, Damasio [5] found that people with a synthesis between pallium and limbic have large decision barrier and good logical thinking ability. This research showed that human intelligence consists of rational thinking, logical reasoning and affective intelligence.

In fact, Minsky [6], as the founder of human intelligence field, mentioned the similar opinion that “intelligent robots have feelings” in The Society of Mind. This opinion is constantly confirmed in the development process of human intelligence. Computers with only logical reasoning abilities are more excellent in storage and operation than human brains. However, shortage of affective intelligence results in blind search in huge solution space for decision. Therefore, computers cannot make rapid and optimal decisions as human brains. The lack of affective intelligence affects computerized decision ability, harmonious interaction between people and computers, thus reducing environmental adaptability and problem-solving ability. Reeves and Nass [7] found that human–computer interaction has similar basic principle with human–human interaction. In the process of human–computer interaction, if computers cannot feel and respond to human emotions, then negative mental state will be brought about.

If computers do not perceive and respond to user emotions, people will feel frustrated, angry and disregarded in human–computer interaction. Therefore, perception and feedback of computers to user emotions become more and more important.

At present, some scholars focus on achieving affective understanding and expression between person and computers. Picard defined affective computing as the calculation of affective expression, induction and influence in Affective Computing [8]. MIT developed sensors for mental signal measure. The sensors consist of GSR, BVP, EMG and The Respiration Sensor, etc. [9]. Japanese scholars developed a variety of “personal robot” product series based on research of kansei engineering [10]. In addition, IBM proposed “the Blue Eye Program” to make robots feel human thought. IBM also developed emotion mouse products feeling human emotion by touch of hand and mouse [11]. Affective group led by Klaus Scherer and affective robot lab by Canamero were famous research institutes [12].

After the 1st Chinese Affective Computing and Intelligent Interaction Conference, the research on affective computing was becoming more and more extensive. The conference received theses on affective computing, psychology & cognition, face processing, language and speech, human–computer interaction, etc. The hotpots of Chinese affective computing focus on unit theory and implementation of unit technology. Wang firstly proposed the concept of emotion to determine theoretical framework of emotional psychology, thus realizing simulation of human emotion activities [13]. Based on this research result, emotional robots are developed to achieve good application in consumer good selection system. Further researches are conducted on affective modeling, human–computer interaction and emotional robots. Gao realized researches of multifunctional perceptrons combined with artificial intelligence technology and parallel processing method. Related software and hardware platforms were developed by using artificial intelligence, modulus operation, parallel processing, real-time operation and emotion recognition [14, 15]. Besides, Wang discussed robot control system framework based on artificial psychology by research of virtual human with expressions and actions [16, 17]. Based on fictional character and emotion space structure in E-Teatrix, Liu et al. [18] proposed original ideas in the direction of artificial and emotional psychology. Fu et al. used PAD basic theory to propose a facial emotion recognition model, thus expanding the application of interaction between cognition and emotion [19]. Wang et al. firstly analyzed the influence of cognitive emotion on driving decision in driving assistance system [20].

As an interdisciplinary with wide application, affective computing involves psychology, physiological sciences, cognitive and computer technology [21]. The application of affective computing is as follows. Firstly, it is used for driver pressure or fatigue monitoring. Secondly, emotional ornament can monitor human indicators and emotions by embedded sensor. Thirdly, emotional doll is produced to reflect the owner’s emotional state. Fourthly, it can also be used as learning partner, changing guide contents with emotions. Fifthly, it can provide special medical services including monitoring intellectual disabilities and autistic children.

As the base of affective computing, the research of affective model is especially important. It is of theoretical significance and application value to determine suitable affective model for artificial emotion expression of computers in the research of human–computer interaction and affective robots. In recent years, psychologists and computational science specialists have proposed affective models based on cognition, probability and multi-layer. Ortony et al. [22] divided emotion into 22 kinds by considering emotion-induced cognition factors based on OCC (Ortony, Clore, Collins) model. Elliott used the method in Ref. [22] to expand OCC model, thus obtaining 26 kinds of emotion [23]. In 1995, Picard of MIT proposed that HMM (hidden Markov model) can be applied in affective modeling [8]. Combined with relationship among character, mood and emotion, Kshirsagar [24] realized synthesis of facial expression for virtual human by proposing affective model based on multi-layer modeling technology. This modeling method simply divides mood into good, poor and neutral. Without considering the relation between character and emotion, this method cannot express complex emotion of human beings. Guoliang et al. [25] discussed the influence of mood to emotion by considering mapping relationship among mood, character and emotion. Based on artificial psychology theory, an affective model is proved to be effective by experiments. However, volatility transfer of complex emotions cannot be realized, presenting poor practicability.

Based on previous research results, the work proposed multi-layer affective computing model of character, mood and emotion spaces by quantifying the relationship among emotion factors. The model discussed the influence of external stimulation, factor desalination and emotion induction to emotional changes. It also presented continuous change of emotion. Simulation and experiment results show that this model can reflect characters of human emotions.

3 Emotional psychology theory

3.1 Definition of affective model

Affective computing model in the work involves the relationship among three factors and the time response to external stimulation signals. Therefore, relevant definition and quantization processing technology are firstly discussed in the work.

3.1.1 Character space

In emotional psychology, character is defined as the whole spirit of one person—the sum of stable mental characteristics formed in certain social conditions. A person’s character is determined by parents and surrounding environments. In the work, character is considered to be constant. The character model is determined by FFM (five factor model) [26]. Therefore, a five-dimensional vector can be used to describe character as follows.

$$C = [{\text{c}}_{1} ,{\text{c}}_{2} ,{\text{c}}_{3} ,{\text{c}}_{4} ,{\text{c}}_{5} ]^{T}$$

where T is transposition; ∀c i  ∊ [0, 1], i = 1, 2, …, 5 are Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism in character, respectively.

3.1.2 Emotion space

Emotion is a person’s feeling about surrounding environment, reflecting demand satisfaction degree in emotional process and character. There are many ways to classify emotion. The work applies classification methods of 6 emotions (joy, sadness, anger, surprise, fear and disgust) in Ref. [27] to define affective variable as follows.

$$E = [e_{joy} ,e_{sadness} ,e_{anger} ,e_{surprise} ,e_{fear} ,e_{disgust} ]^{T}$$

where ∀e j  ∊ [0, 1], j = joy, sadness, anger, sup rise, fear, disgust. Compared with other two factors, emotion space has larger variation speed. It changes fastest with external stimulation signals.

3.1.3 Mood space

With weak signal, mood is a durable emotion state affecting a person’s whole spirit. The main characteristic of mood is diffusivity aiming at the whole objects rather than certain experience of specific thing. Mood is a time variable compared with character. It can be defined as good, bad and normal states.

In psychology, there is no uniform definition about dimensions of mood. The work defines mood variable as follows by using PAD (pleasure-displeasure, arousal-nonarousal, dominance-submissiveness) 3D mood space model [28].

$$M = [m_{P} ,m_{A} ,m_{D} ]^{T}$$

where − 1 ≤ m P , m A , m D  ≤ 1. If the mood is in a calm state, then M = [0, 0, 0]T.

3.2 Induction and reduction of mood and emotion

3.2.1 Induction of emotion

Common external stimulation signals consist of facial expressions, sounds, gestures, postures and all kinds of physiological signals. All emotions are induced by stimulation signals. Affective factor enters the brain as input signal, stimulating people’s mood and emotion changes. Emotion-induced variable A can be determined by evaluation system of emotional cognition. It is assumed that A can be expressed as follows [29].

$$y = \left\{ {\begin{array}{*{20}l} {a_{1} t^{{b_{1} }} } \hfill \\ {a_{1} e^{{ - b_{2} (t - t_{1} )}} } \hfill \\ \end{array} } \right.,{\kern 1pt} \quad 0 \le t \le t_{1}$$
(1)

where y is external stimulation signal strength at t; b 2 controls response decay rate; a 1 and b 1 control response amplitude. Emotion-induced variable is defined as follows.

$$A = [y_{1} ,y_{2} ,y_{3} ,y_{4} ,y_{5} ,y_{6} ]^{T}$$

where y i , i = 1, 2, 3, 4, 5, 6 are affective information variables of emotions including joy, sadness, anger, surprise, fear and disgust.

3.2.2 Decay of emotion

With the disappearance of the stimulation, the emotion fades; the strength of the emotion decreases; the emotional state eventually tends to be stable. In the above process, character will be the only influence factor of emotion if there is no stimulation. Emotional decay process is defined as Eq. (2).

$$\frac{dE(t)}{dt} = \alpha [E_{0} - E(t)]$$
(2)

where E 0 is the emotion in ideal state; α the affective dilution coefficient determined by the strength of character.

3.2.3 Decay of mood

With the disappearance of the stimulation, the mood fades; the strength of mood gradually weakens. The process of mood decay is defined as Eq. (3).

$$\frac{dM(t)}{dt} = \beta [M_{0} - M(t)]$$
(3)

where β is mood decay factor related to character. Equation (3) shows the following conclusions. If current mood state is not ideal, then it will migrate to ideal state based on positive mood fluctuation rate. If current mood state has a higher dimension value than ideal state, then it will migrate to ideal state based on negative mood fluctuation rate.

3.3 Mapping among character, mood and emotion

Character is a constant value. Under the stimulation of external signals, individuals with different personalities have large fluctuation difference between mood and emotion.

3.3.1 Mapping between character and mood spaces

Mood is 3D vector. Therefore, mood space consists of 8 subspaces (See Table 1).

Table 1 Description of subspaces in PAD space
Full size table

In Table 1, +P stands for Pleasure > 0; −P for Pleasure ≤ 0. Therefore, character space is a 6D vector. Mapping relation between character and mood is established as Eq. (4).

$$M = K*P^{T}$$
(4)

where M is the mood strength; P the character; K the mood transfer matrix. K is defined as follows.

$$K = \left[ {\begin{array}{*{20}c} 0 & 0 & {0.25} & {0.59} & {0.19} \\ {0.16} & 0 & 0 & {0.42} & { - 0.54} \\ 0 & {0.21} & {0.77} & { - 0.27} & 0 \\ \end{array} } \right]$$

3.3.2 Mapping between mood and emotion spaces

When stimulation signal, character and mood are different, emotion will change. It is assumed that mood is a 1D vector. Therefore, we establish linear mapping relation between mood and emotion spaces. Table 2 shows the relation between PAD and emotion spaces based on the above emotion state.

Table 2 Mapping relation between PAD and emotion spaces
Full size table

Based on Table 2, mapping matrix between mood and emotion spaces is expressed as Eq. (5).

$$\begin{aligned} L & = [PAD_{joy} ,PAD_{sadness} ,PAD_{anger} ,PAD_{\sup rise} ,PAD_{fear} ,PAD_{disgust} ]^{T} \\ & = \left[ {\begin{array}{*{20}c} {0.4} & { - 0.4} & { - 0.51} & {0.2} & { - 0.64} & { - 0.4} \\ {0.2} & { - 0.2} & {0.59} & {0.1} & { - 0.6} & {0.2} \\ {0.15} & { - 0.5} & {0.25} & {0.1} & { - 0.43} & {0.1} \\ \end{array} } \right] \\ \end{aligned}$$
(5)

Mapping relation between mood and emotion is as follows.

$$E = f(M,PAD) = \frac{D}{{\sum\nolimits_{i = 1}^{6} {d_{i} } }}$$

where D = [d 1, d 2, d 3, d 4, d 5, d 6]; \(d_{i} = [(M - PAD_{i} )^{T} (M - PAD_{i} )]^{{\frac{1}{2}}}\) (i = 1, 2, …, 6). Parameters i stand for 6 emotions including joy, sadness, anger, surprise, fear and disgust.

4 Multi-layer affective computing model

The establishment of model can be decomposed into emotion and mood models because of the influence of mood to the change of emotion. Mood is updated by considering external stimulation, emotion and mood decay. Besides, emotion strength is updated by considering the interaction among external stimulation, mood, character and emotion.

4.1 Mood update equation

The fluctuation of mood state is related to the external stimulation and character. It can also be diluted. If the mood state is M t at t, then mood update equation will be expressed as follows.

$$M_{t} = M_{t - 1} + K*C_{t} + h(C_{t} ,M_{t - 1} ) + r(A_{t} ) - T_{M}$$
(6)

where M t−1 is the mood state at t − 1; K * C t the influence component of Character C t to mood state; h(C t , M t−1) the dilution component of mood restraining transfer of mood state; r(A t ) the influence degree of A to mood; T M the mood change threshold. If M t  > T M , then mood state will transfer; otherwise, it will not transfer. Equation (3) is dispersed to obtain Eq. (7).

$$\begin{aligned} M_{t}^{\prime } & = \frac{{M_{t - 1} + \beta M_{0} }}{1 + \beta } \\ h(C_{t} ,M_{t - 1} ) & = M_{t}^{\prime } - M_{t - 1}^{\prime } = \frac{{\beta (M_{t - 1} - M_{0} )}}{1 + \beta } \\ r(A_{t} ) & = \lambda_{M} *PAD*A_{t} \\ \end{aligned}$$
(7)

where λ M is inducer of external stimulation to mood determined by character.

4.2 Emotion update equation

If there is external stimulation, then emotion will transfer. Transfer of emotion is affected by factors including self-dilution, stimulation, mood and emotion. When there is stimulation, emotion update equation (multi-layer affective model) can be expressed as Eq. (8).

$$E_{t} = E_{t - 1} + f(M_{t} ,PAD) + \varphi (C_{t} ,E_{t - 1} ) + \theta (A_{t} ,C_{t} ) - T_{E}$$
(8)

where E t−1 is emotion state variable at t − 1; f(M t , PAD) the influence of mood to emotion state; φ(C t , E t−1) the emotion decay component restraining emotion transfer; θ(A t , C t ) the influence of A to M; T E the emotion change threshold. If E t  > T E , then emotion will transfer.

$$\begin{aligned} E_{t}^{\prime } & = \frac{{E_{t - 1} + \alpha E_{0} }}{1 + \alpha } \\ \varphi (C_{t} ,E_{t - 1} ) & = E_{t}^{\prime } - E_{t - 1}^{\prime } = \frac{{\alpha (E_{t - 1} - E_{0} )}}{1 + \alpha } \\ \theta (A_{t} ,C_{t} ) & = \lambda_{E} A_{t} \\ \end{aligned}$$
(9)

where λ E is emotion inducer relevant to character.

4.3 Multi-layer affective model

Figure 1 shows multi-layer affective model. External stimulation is transformed into affective variable A by perception and evaluation. A acts on mood and emotion spaces, respectively. In the relationships with character space, the two spaces affect the changes of emotion and mood. At last, expression synthesization is output.

Fig. 1
figure 1

Multi-layer affective model

Full size image

Thereinto, facial expression synthesization is emotional expression. Requirement expression can be obtained based on multi-layer affective model.

5 Example simulation

Induced by external stimulation, mood and emotion will change. Test objects with different characters have different fluctuation amplitude and frequency. If stimulation disappears, the corresponding emotion and mood states will tend to stable under emotional control.

In the work, relevant parameters are set to simulate transfer of emotion and mood based on external signal at quiet condition ([0, 0, 0, 0, 0, 0]T, [0, 0, 0]T). After that, we obtain transfer process and facial expression output of emotion and mood states under different characters, single and multiple attribute stimulation signals.

When stimulation signals of emotion-induced variables are anger and disgust, we simulate emotion states of introvert (C = [0.7, 0.3, 0.1, 0.4, 0.2]T) and extraversion (C = [0.7, 0.3, 0.7, 0.4, 0.2]T). Figures 2 and 3 show state strength curves, respectively.

Fig. 2
figure 2

State strength of anger with introvert and extraversion

Full size image
Fig. 3
figure 3

State strength of disgust with introvert and extraversion

Full size image

In Figs. 2 and 3, emotion strength rapidly rises to the maximum under the simple attribute simulation signal. With the disappearance of simulation signal, emotion strength gradually decreases, and finally tends to be stable. Meanwhile, introvert and extraversion characters are compared. The extrovert has quick emotion change. With the increase of the maximum, the recovery of mood becomes faster. The extrovert has higher output expression strength.

In the work, multi-layer affective computing model is applied to predict emotion states of 400 selected test objects. System prediction accuracy rate δ is defined as follows.

$$\delta = \frac{\upsilon }{N} \times 100\%$$

where υ is the time of accurate predictions; N the total time of system predictions. The model in the work is compared with the method in Ref. [30]. We select prediction accuracy rates of 50 test objects for drawing (See Fig. 4).

Fig. 4
figure 4

Forecast accuracy of multi-layer affective model and HMM model

Full size image

Figure 4 shows that prediction accuracy fluctuates in prediction process. However, multi-layer affective computing model in the work is better than HMM in Ref. [31]. The maximum prediction accuracy rates are 84 and 72%, respectively.

6 Website sales behavior decision based on multi-layer affective model

Research shows that consumption psychology is affected by customer’s age, gender, race, religion, commodity attribute (such as color and origin), other customers’ situation of purchase and use. In recommendation system, service quality is improved by semantic recommendation service according to emotion [32, 33]. Therefore, it is of great significance to explore the emotion between commodity and consumer for recommendation. Based on multi-layer affective model proposed in the work, we build a model perceiving, identifying and understanding people. The model can make intelligent, sensitive and friendly responses to people’s emotions. In E-commerce field, customer’s requirements are met by analyzing customer’s emotion state and change to commodity, thus providing satisfied service for the customer. Therefore, the income is increased by improving business management mode.

Based on multi-layer affective model, online shopping emotions of 10,000 customers are predicted to adjust marketing strategy. Before and after the application of affective model, sale activities of customers with different types are conducted with statistical analysis. Effect of sale strategy In is introduced to express purchase decision of customers with different attributes while using affective model.

$$In = \frac{{R^{'} - R}}{R}$$
(10)

where R and R′ are customers’ purchase rates before and after using affective model. After statistic analysis, we obtain purchase rate change of customers with different types (See Table 3; Fig. 5). Figure 6 shows the change of purchase rate.

Table 3 Purchase rate of customers with different types before and after using affective model (%)
Full size table
Fig. 5
figure 5

Purchase rate of customers with different types before and after using affective model

Full size image
Fig. 6
figure 6

Purchase rate change of customers with different types before and after using affective model

Full size image

Purchase rate of customers with different types increases after using affective model (See solid line in Fig. 5). For customers with different types, the changes of rate are slightly different. Figure 6 shows that purchase rate of female customers obviously increases. This is because female customers are more sensibility, easily affected by emotion. In customers of different ages, the young are more susceptible to emotional impact. The higher the income, the larger the emotional impact is. The result accords with actual situation.

7 Conclusions

Based on the basic theory of emotional psychology, the relevant definitions and quantification are given. A multi-layer affective computing model is proposed by building mapping relationship among three spaces. This model can reasonably reflect the relationship among external stimulation, character, mood and emotion. Result shows that the change process of emotion and mood basically accords with the law of human emotion change. Consequently, practicality and effectiveness of the model are verified by experiment, thus providing a new method for affective automatic generation and computing model.