1 Introduction

Computational methods are more and more used to get more insight in human functioning and dysfunctioning. By designing a human-like computational model for certain normal functioning of certain mental and/or social processes, it can be found out what alterations make the model show dysfunctional behavior, and verify how that relates to the empirical literature. Such a computational model can be a basis for a so-called virtual patient model. An important source of knowledge for the design of a human-like computational model can be found in the fields of Cognitive and Social Neuroscience, and in what is encountered in the practice of medical clinics. The work reported in this paper results from a cooperation between researchers in AI and in medical practice.

The focus here is on social functioning and dysfunctioning resulting from a certain type of dementia, in particular the behavioral variant of frontotemporal dementia (bvFTD); see (Piguet et al. 2011). As will be explained in Sect. 2 in more detail, one of the problems encountered is difficulty in recognizing emotions of others, in particular the negative ones, even while emotion contagion still can function properly.

To model such human processes in a way that is justifiable from a neuroscientific perspective, knowledge of the underlying mechanisms in the brain is needed. Usually dynamics and cyclic connections play an important role in such mechanisms, and therefore a modeling approach is needed that can handle such cyclic dynamic processes well. The Network-Oriented Modeling approach based on temporal-causal networks used here is indeed able to do so (Treur 2016b; 2018).

In the paper, first in Sect. 2 background knowledge is described on the processes addressed. In Sect. 3 the temporal-causal network model is introduced. Section 4 describes simulation experiments for the addressed case, both for normal functioning and for dysfunctioning. Finally, Sect. 5 is a discussion.

2 Neuropsychiatric Background

The behavioral variant of frontotemporal dementia (bvFTD) is a neurodegenerative disorder associated with progressive degeneration of the frontal lobes, anterior temporal lobes, or both (Piguet et al. 2011). The disease is a leading cause of early-onset dementia and the third most common form of dementia across all age groups (Ratnavalli et al. 2002). Alterations in social cognition represent the earliest and core symptoms of bvFTD resulting in emotional disengagement and socially inappropriate responses or activities (Ibanez and Manes 2012; Kumfor and Hodges 2017). As part of their impaired social cognition, bvFTD patients are often unconcerned about their relatives, unable to adjust to their environment, lacking usual social inhibitions, and unable to recognize and attribute mental states to self and others. Consequently, dissolution of social attachment can be profound and the implications on patients’ life and their relatives are far-reaching (Diehl-Schmid et al. 2007; Riedijk et al. 2006). In this paper, the following case is used as an illustration.

A 55 year old man who was recently diagnosed with bvFTD visited our outpatient clinic with his wife. While explaining the difficulties she met in the home situation, she started crying. The patient followed the conversation, and at this point he looked at her, his own eyes got watery, but he looked dazzled. Upon the question how he thought his wife was feeling, he answered that his wife was probably feeling happy. On the Ekman 60 faces test, he scored 43 out of 60 items correctly, which is below the cutoff of 46. His subscores were: Anger 8/10, Disgust 9/10, Anxiousness 8/10, Happiness 8/10, Sadness 5/10, Surprise 5/10.

This case illustrates that in bvFTD there may be a dissociation between emotion contagion and facial emotion recognition, in this case in particular the recognition of sadness.

Studies on social cognition in bvFTD have shown that facial emotion recognition is severely disturbed, with the exception of happiness. In particular, impaired recognition of negative emotions such as anger and disgust have been reported (Gossink et al. 2018). Applying the animal model of empathy of Frans de Waal, emotional contagiousness is the most inner layer, present from early evolution in most vertebrate animals (de Waal 2009). Following the hypothesis that empathy in humans, and more specific in bvFTD, will exhibit a ‘Recapitulati in reverse’, the outlayers of the Russian Doll, symbolic for more advanced evolutionary social cognitive abilities will be lost first and the inner layer of emotion contagion will be preserved and even more prominent in advanced dementia: ‘Heightened emotional contagion in mild cognitive impairment and Alzheimer’s disease is associated with temporal lobe degeneration’, by (Sturm 2013), as is illustrated in our case.

3 The Temporal-Causal Network Model

This section describes the temporal-causal network model for interpretation of emotions. The model describes how interpretation of emotions takes place, with a focus on recognizing emotions showed by others. Patients with frontotemporal dementia (bvFTD) show emotional disengagement and social responses or activities that are not suitable. In particular, this model focuses on the part where people with bvFTD are unable to recognize and attribute emotional states to self and others. This can lead to the effect that emotions are misinterpreted or even not recognized at all. The model can both show how the process of recognizing and attributing emotional states works regularly and when it is affected by bvFTD.

A conceptual representation of a temporal-causal network model represents in a declarative manner states and connections between them that indicate (causal) impacts of states on each other, as assumed to hold for the application domain addressed. The states have (activation) levels that vary over time. The following three notions are main elements of a conceptual representation of a temporal-causal network model:

  • Connection weight ωX,Y. Each connection from a state X to a state Y has a connection weight value ωX,Y representing the strength of the connection, between −1 and 1.

  • Combination function cY(..). For each state a combination function cY(..) to aggregate the causal impacts of other states on state Y.

  • Speed factor ηY. For each state Y a speed factor ηY to represent how fast a state is changing upon causal impact.

The conceptual and numerical representation of the model introduced here will be presented in this section. The model was designed by integrating a number of theories some of which were discussed in Sect. 2, and also elements from Damasio (1994; 1999; 2018)’s view on emotions and feelings, and Iacoboni (2009) on mirror neurons and social contagion.

The developed model shows the difficulties that persons with bvFTD can have regarding recognition of emotions. Not only the recognition of emotions of others is included, but also the experience of own emotional feelings which includes mirror links from observed emotions. Figure 1 gives an overview of the conceptual representation of the model. The following notations are used for the state names:

Fig. 1.
figure 1

Overview of the conceptual representation of the model

Full size image
figure a

For each state a label LPn refers to the corresponding numerical representation of the update equation of the state, as described below. An overview of the states, their connections and weights can be found in Table 1. States or weights with subscript h or s correspond to the emotional feelings happy or sad. An example is ssh meaning the sensor state for the own emotional response for happy (sensing the own body state, for example, the own smile). States indicated by a B correspond to the observation of emotional expression(s) of another person B. For example, srsB,h means the sensory representation state of B having a happy face. Finally, subscript e is used to indicate if someone is showing any emotion. Therefore, wsB,e means the world state of person B showing an emotion, for example, an emotional face. Overall, the upper part (the first three causal pathways) are used for recognizing the emotional state of someone else (person B).

Table 1. Overview of the connections, their weights, and their explanations; see also Fig. 1
Full size table

The lower part (the other two causal pathways) are used to model feeling the own emotions using body loops and as-if body loops as described by Damasio (1994; 1999; 2018). The model presented here incorporates parts of the model described in (Treur 2016b, Ch. 9). The part that is included from this model are the bottom two cycles of states with the body loops affecting the body state x of a person, representing own emotional feeling according to the theory of Damasio (1994; 1999; 2018). In this model body state x can be either s (sad) or h (happy) corresponding to the emotion. This emotion can also be expressed by another person B. Therefore, the communication of, for example, body state h (happy) to B expresses that the person self knows that B feels h (happy). The connections from srsB,h and srsB,s to psh and pss, respectively, provide mirroring functionality to the preparation states, following Iacoboni (2009). These connections make the person feel what the other person expresses.

Most connection weights have a positive value between 0 and 1 according to the strength of the effect they have on consecutive states. However, suppressing effects are modeled by using a negative weight. A few of those negative weights occur in the model. The connection weights with a negative value are ω3,h,s, ω3,s,h, ω7,h,s, ω7,s,h, ω3,h, and ω3,s.

A conceptual representation of the temporal-causal network model can be transformed in a systematic manner into a numerical representation of the model (Treur, 2016b, Ch 2):

  • at each time point t each state X connected to state Y has an impact on Y defined as impactX,Y(t) = ωX,Y X(t) where ωX,Y is the weight of the connection from X to Y

  • Based on the combination function cY(…) the aggregated impact of multiple states Xi on Y at t is: aggimpactY(t) = cY(impactX1,Y(t), …, impactXk,Y(t))

    $$ = {\mathbf{c}}_{Y} ({\varvec{\upomega}}_{{X_1\text{,}Y}} X_{1} (t)\text{,} \, \ldots \text{,}\,{\varvec{\upomega}}_{{X_k\text{,}Y}} X_{k} (t)) $$

    where Xi are the states with connections to state Y

  • Using the speed factor ηY the effect of aggimpactY(t) on Y is exerted over time gradually: Y(t + Δt) = Y(t) + ηY [aggimpactY(t) − Y(t)] Δt or

    $$ {\text{d}}Y (t )/dt = {\varvec{\upeta}}_{Y} \left[ {{\mathbf{aggimpactY}}({\mathbf{t}}) - Y (t )} \right] $$

  • Thus, the following difference and differential equation for Y are obtained:

    $$ Y (t + \Delta t )= Y (t )+ {\varvec{\upeta}}_{Y} [{\mathbf{c}}_{Y} ({\varvec{\upomega}}_{X_1,Y} X_{1} (t ), \, \ldots ,{\varvec{\upomega}}_{X_k,Y} X_{k} (t )) - Y(t)]\,\Delta t $$
    $$ {\text{d}}Y(t)/dt = {\varvec{\upeta}}_{Y} \,[{\mathbf{c}}_{Y} ({\varvec{\upomega}}_{X_1,Y} X_{1} (t), \, \ldots ,\,{\varvec{\upomega}}_{X_k,Y} X_{k} (t)) - Y\left( t \right)] $$

The states related to LP1, LP2, LP3, LP6, LP7, LP11, LP12, LP16, LP19, LP20, LP23, LP24, and LP25 make use of the identity combination function c(V) = id(V) = V. Those for LP8, LP9, LP13, LP14, LP17, LP18, LP21, and LP22 make use of the scaled sum combination function, which is represented numerically by:

$$ {\text{c}}(V_{1} , \, \ldots , \, V_{k} ) = {\mathbf{ssum}}_{{\varvec{\uplambda}}} (V_{1} , \, \ldots , \, V_{k} ) = (V_{1} + \, \ldots \, + \, V_{k} ) \, /{\varvec{\uplambda}} $$

where λ is the scaling factor. Finally, states related to LP10 and LP15 make use of a logistic function to get a binary all-or-nothing effect of these communications.

$$ {\text{c}}(V_{1} , \, \ldots , \, V_{k} ) \, = {\mathbf{alogistic}}_{{\upsigma,\uptau}} (V_{1} , \, \ldots ,V_{k} ) = \, [(1/(1 + {\text{e}}^{{{-}\upsigma(V_1 + \, \ldots \, + V_k -\uptau)}} )) \, {-} \, 1/(1 + {\text{e}}^{{\upsigma \uptau }} )] \, (1 + {\text{e}}^{{{-}\upsigma \uptau }} ) $$

4 Simulation Experiments

To explore the behaviour of the designed temporal-causal network model, two scenarios were simulated in Matlab. The first scenario describes the case of a normal person who would recognize emotions of others. In this case, it is expected that when person B shows an emotion, the person will correctly communicate this emotion at the communication states escomm,B,h or escomm,B,s. Also, the own feeling of that specific observed emotion will be activated through mirror neurons. The second scenario describes the specific case in which a person has difficulties recognizing the right emotions due to bvFTD. It is expected that when person B shows the emotion sad, this emotion will be wrongly interpreted as happy as the case explained in Sect. 2. Therefore, the communication states will yield activations that differ from the ones in the first scenario, although through the mirroring system still contagion takes place through which the sadness is felt.

The weights for the connection strengths ωk are for most connections set to 1; the exceptions are shown in Table 2 lower part. For ω7,h,s and ω7,s,h a value of −0.2 has been chosen, since the preparation states for communication that either it is a sad emotion that person B is showing or a happy emotion normally will not have a high activation level at the same time. In this way, negative weights will cause suppression between the states if one of them is activated. Similarly, for weights ω3,h,s and ω3,s,h a value of −0.05 has been chosen, to express that the belief states for either believing person B shows a happy emotion or a sad emotion usually will not have high activations at the same time. Note that the values 0.7 and 0.05 for ω2,h and ω2,s, respectively, indicate that when no specific emotion is recognized, usually an emotional face is more believed to indicate happiness than sadness.

Table 2. Settings for the scaling factors used and connection weights deviating from 1
Full size table

The simulations have been performed with speed factor η = 0.5 for all states, Δt = 0.5, and the scaling factors as displayed in Table 2 upper part. Since LP10 and LP15 make use of a logistic function, they have a threshold and steepness. Both states use a logistic function with steepness 200 and threshold 0.5. On the figures, time can be seen on the horizontal axis of the figures and the activation levels of the states are on the vertical axis.

The graphs in Figs. 2 and 3 display the results of the simulations that have been performed, for Scenario 1 and Scenario 2, respectively. The upper and lower graphs show a part of the results, to get a better view at them. The graph in Fig. 2 highlights a few of the states which show the results of the simulation. It can be seen that the states for a person showing emotional (wsB,e) and for a person showing a sad face (wsB,s) are highly activated at the start (blue and orange striped lines). Naturally, the sensor states and sensory representation states are becoming active as well (srsB,e and srsB,s) which can be seen by the yellow and black striped lines. Those states for the representation of the happy face (srsB,h) stay low, visible by the pink striped line. Furthermore, it can be seen that the believe state for recognizing a happy face (bsB,h) shows some activation (purple line). This is caused by the fact that the state for recognizing an emotional face is high, but when it gets clear to the person that the emotion is about a sad emotion the feeling that it might be a happy emotion is quickly reduced and it can be seen that the communication state for a happy emotion (escomm,B,s) stays low (dark blue line). In the end, the person communicates that a sad face has been observed (escomm,B,s, red line). Also, the mirror neuron system for the own sad feeling becomes active, showing that emotion contagion takes place for the observed sadness. This can be seen by the activation of esh which is in the emotion contagion cycle. When performing the simulation with the activation of a happy face instead of a sad face at the start, similar results are shown (with the activation of communication a happy face instead of a sad face). Therefore, this simulation shows what is expected of how someone without any condition that affects these processes would interpret an emotion.

Fig. 2.
figure 2

Simulation results for Scenario 1: normal functioning (Color figure online)

Full size image
Fig. 3.
figure 3

Simulation results for Scenario 2: the case with bvFTD (Color figure online)

Full size image

For the second scenario, the settings of four weights have been changed. The weights for ω1,h,h, ω1,s,s, ω4,h, and ω4,s have been set to a connection weight of 0.05. This has been done because in the second scenario the case of a person with bvFTD is simulated, which means that those links are damaged and therefore have a low weight. Figure 3 displays the result of the second scenario. In the graph, it can be seen that the external states for showing an emotional face (wsB,e) and showing a sad face (wsB,s) are high from the start, and are kept high, to simulate their presence.

These are the black and pink lines on top of the graph. However, due to the damaged links, the communication state for saying that a person shows a sad face (escomm,B,s, dark blue line) is not activated in the end, while still by emotion contagion the own sad feeling develops (ess, orange line). This can be seen as the dark blue dotted line at the bottom of the graph stays low throughout the entire simulation, which implies no communication of an observed sad feeling while the orange line indicates the own sad feeling to be active. As can be seen, in contrast the communication state for saying that a person shows a happy face (escomm,B,h) does get activated while the person never showed a happy face (red line), and no contagion of happiness took place. This can be explained from the fact that the person does recognize that there is an emotion visible (activation of srsB,e, yellow line). However, the interpretation of the specific kind of emotion is disrupted. Therefore, the simulation shows the specific case that has been observed in patients: how damaged links can cause someone with bvFTD to misinterpret emotions.

5 Verification of the Network Model by Mathematical Analysis

Dedicated methods have been developed for temporal-causal network models to verify whether an implemented model shows behaviour as expected; see (Treur 2016a, 2016b, Ch. 12). In this section equilibria of the designed model are addressed. By Mathematical Analysis their values are found and by comparing them to simulated values the model is verified. Stationary points and equilibria are defined as follows.

A state Y in a temporal-causal network model has a stationary point at t if dY(t)/dt = 0. A temporal-causal network model is in an equilibrium state at t if all states have a stationary point at t. In that case the above equations dY(t)/dt = 0 for all states Y are called the equilibrium equations. These are general notions, for temporal network models the following simple criterion was obtained in terms of the basic elements defining the network, in particular, the states Y, connection weights ωX,Y and the combination functions cY(..); see (Treur 2016a; 2016b, Ch. 12).

Criterion for Stationary Points and Equilibria in a Temporal-Causal Network Model

A state Y in an adaptive temporal-causal network model with nonzero speed factor has a stationary point at t if and only if

$$ {\mathbf{c}}_{Y} (\upomega_{X_1,Y} (t )X_{1} (t), \, \ldots ,\upomega_{X_k,Y} (t )X_{k} (t)) = Y(t) $$

where X1, …, Xk are the states with outgoing connections to Y.

A temporal-causal network model is in an equilibrium state at t if and only if for all states with nonzero speed factor the above criterion holds at t.

Equilibrium equations for an identity function id(.) or scaled sum combination function ssumλ(..) are

$$ \begin{aligned} & {\mathbf{id}}({\varvec{\upomega}}_{X,Y} X(t)) \, = {\varvec{\upomega}}_{X,Y} X(t) = Y(t) \\ & {\mathbf{c}}_{Y} ({\varvec{\upomega}}_{X_1,Y} X_{1} (t), \, \ldots ,{\varvec{\upomega}}_{X_k,Y} X_{k} (t)) = \, ({\varvec{\upomega}}_{X_1,Y} X_{1} (t) + \, \ldots \, + {\varvec{\upomega}}_{X_k,Y} X_{k} (t))/\uplambda_{Y} = Y(t) \\ \end{aligned} $$

So, they are linear equations in the state values involved with connection weights and scaling factors as coefficients:

$$ \begin{aligned} & {\varvec{\upomega}}_{X,Y} X(t) = Y(t) \\ & {\varvec{\upomega}}_{X_1,Y} (t)X_{1} (t) + \, \ldots \, + {\varvec{\upomega}}_{X_k,Y} (t)X_{k} (t) =\uplambda_{Y} Y(t) \\ \end{aligned} $$

In the presented model the scaling factors have been set as the sum of the positive weights of the incoming connections; therefore all coefficients are built from connection weights. Using this, the following equilibrium equations for the states were obtained for the presented network model here; to simplify the notation the reference to t has been left out, and underlining is used to indicate that this concerns equilibrium state values, not state names. Here the connection weights are named as shown in Table 1, and A1 to A3 are constants.

$$ \begin{aligned} & \underline{\text{srs}}_{B,h } = A_{1} \quad \quad \quad \underline{\text{srs}}_{B,s } = A_{2} \quad \quad \quad \underline{\text{srs}}_{B,e } = A_{3} \\ & (\upomega_{1,X,X } +\upomega_{1,Y,X } +\upomega_{2,X} +\upomega_{4,X} )\,\underline{\text{bs}}_{B,X} =\upomega_{1,X,X} \,\underline{\text{srs}}_{B,X} +\upomega_{1,Y,X} \,\underline{\text{srs}}_{B,Y} +\upomega_{2,X} \,\underline{\text{srs}}_{B,e} +\upomega_{4,X} \\ & \underline{\text{srs}}_{X} +\upomega_{3,Y,X} \,\underline{\text{bs}}_{B,Y} \\ & (\upomega_{5,X } +\upomega_{6,X} )\,\underline{\text{ps}}_{comm,B,X} =\upomega_{5,X} \underline{{\,{\text{bs}}}}_{B,X} +\upomega_{6,X} \,\underline{\text{cs}}_{selfother,B,e} +\upomega_{7,Y,X} \,\underline{\text{ps}}_{comm,B,Y} \\ & (\upomega_{8,X } +\upomega_{9,X} )\,\underline{\text{es}}_{comm,B,X} =\upomega_{8,X} \,\underline{\text{ps}}_{comm,B,X} +\upomega_{9,X} \,\underline{\text{cs}}_{selfother,B,e} \\ & \underline{\text{cs}}_{selfother,B,e } =\upomega_{10,e} \,\underline{\text{srs}}_{B,e} \\ & (\upomega_{11,X } +\upomega_{12,X} )\,\underline{\text{ps}}_{X} =\upomega_{11,X} \,\underline{\text{srs}}_{B,X} +\upomega_{12,X} \,\underline{\text{srs}}_{X} \\ & (\upomega_{13,X } +\upomega_{14,X} )\,\underline{\text{srs}}_{X} =\upomega_{13,X} \,\underline{\text{ss}}_{X} +\upomega_{14,X} \,\underline{\text{ps}}_{X} \\ & \underline{\text{es}}_{X} =\upomega_{15,X} \,\underline{\text{ps}}_{X} \\ & \underline{\text{ss}}_{X} =\upomega_{16,X} \,\underline{\text{es}}_{X} \\ \end{aligned} $$

Note that in the above equations in the equilibrium state values, variable names X and Y are used that have multiple instances for h (happy) and s (sad). If these equilibrium state values are instantiated and renamed as shown in Table 3, 19 linear equations in X1 to X19 are obtained with coefficients based on the connection weights and the constants A1 to A3.

Table 3. State names used in the equilibrium equations
Full size table

These 19 linear equations can be solved symbolically, for example using the WIMS Linear Solver (see WIMS 2018), thereby obtaining complex algebraic expressions for the equilibrium values, linear in the constants A1 to A3 with as coefficients rational (broken) functions in terms of the connection weights. For verification all connection weights have been set as the simulation shown and Table 2. For these connection weight values, the following solution was found in terms of A1 to A3:

figure b

For the above connection weight values and values A1 = 1, A2 = 0, and A3 = 1, the solution was found shown in the third and sixth row of Table 4.

Table 4. Results of the mathematical analysis
Full size table

A logistic function with steepness 200 and threshold 0.625 applied to the communication execution states X10 and X11 (multiplied by the scaling factor 1.25 to undo the scaling) provides X10 = 1, and X11 = 2.613 10−21. Similarly, for other values of A1 to A3, the equilibrium values have been found. For example, for A1 = 0, A2 = 0, A3 = 1, it was found X6 = 0.3176815847395451, X7 = 0.02201027146001467, X8 = 0.4476266702238826, X9 = 0.1788345672424908, X10 = 0.5581013361791062, X11 = 0.3430676537939928 (a logistic function with steepness 200 and threshold 0.625 applied to the communication execution states X10 and X11 multiplied by the scaling factor 1.25 to undo the scaling provides X10 = 1, and X11 = 0), and for A1 = 0, A2 = 1, A3 = 1, X6 = 0.2956713132795305, X7 = 0.9904622157006603, X8 = 0.3128971297716712, X9 = 0.9445252228817893, X10 = 0.4503177038173369, X11 = 0.9556201783054313 (a logistic function with steepness 200 and threshold 0.625 applied to the communication execution states X10 and X11 multiplied by the scaling factor 1.25 to undo the scaling provides X10 = 1, and X11 = 0). All these values have been checked with the values of the simulation scenarios and were found very accurate (deviations less than 0.001). This provides evidence that the implemented model does what is expected.

6 Discussion

This paper introduces a temporal-causal network model that describes the interpretation of emotions showed by others. The model can also show cases of when the interpretation of emotions is incorrect, such as can be the case of persons with bvFTD; this is based on the assumption that it is at least observed that there is an emotional face, although the specific type of emotion is not recognized correctly. Several simulations have been performed to test the model in both these behaviours. In the presented scenario for a person with bvFTD it was shown how an observed sad face led to contagion of sadness by the mirror system in a correct way, but at the same time the emotional face was nevertheless not recognized as sad, but instead as happy. A mathematical analysis was done confirming the simulation outcomes; this gave evidence that the model as implemented does what it is meant to do.

The model can be applied as the basis for human-like virtual agents, for example, to obtain a virtual patient model to study the way in which recognition of emotions can deviate for certain types of persons. Also, to study how to potentially enhance the recognition of emotions when damaged. In further research real data can be used to test the model in more detail. Furthermore, more scenarios or cases could be simulated to analyse more and different outcomes of the model. An example is the addition of specific therapies to enhance the damaged links and therefore see if this can have a positive effect on interpreting emotions for people that have difficulties with recognizing as a result of bvFTD. In future extensions of the model also more emotions than sad and happy can be addressed.