Keywords

1 Introduction

Human emotions’ study has been of great interest to Psychology and Sociology, which is why researchers disagree on the number of basic emotions, but there is consensus to include among them, joy, anger, fear, sadness, surprise and disgust. The results of emotional analysis contribute in a great deal to the treatment of mental illness to try to identify mechanisms of cure or slow the progression of the disease [1]. We focused our study on the hypothesis that finding patients with early Alzheimer have few disconnections between the emotions that are reflected in their face and feeling that can be expressed in a sentence. The proposal of this particular work is based on the idea that words alone have a certain orientation of feeling, especially adjectives and adverbs, so that one way to determine that orientation of the input text is based on its probability of occurrence with positive and negative terms. Section three details each phase of the methodology proposed for the analysis and recognition of human emotions in people with Alzheimer, as well as the techniques that we use for the detection of emotions in both text and video and as the final phase of the methodology performed for text mining using Naive Bayes algorithms and Bayesian Multinomial Network. To collect the conversations, the methodology proposed in [2] was used and for the transcripts of videos the data files of the UK were used as format (http://www.data-archive.ac.uk/) create Bayesian algorithms used in other invasive diagnostic investigations [3] and [4] are trained with manual labeling done by experts. The labeling belongs to the semantic polarity of each of the sentences said by the patients in video. To verify validity labels were used applying Kappa index. This process shows that there are connectors that do not contribute to the classification, and therefore it was necessary to repeat the experiments until finding the combination - pattern - of adjectives, connectors and articles suitable to achieve a classification with the smallest possible error. Hence a single word and phrase may have a positive, negative, or neutral percent-age charge of polarity depending on the context in which it is found. The results of Bayesian algorithms are compared with the analysis of emotions obtained with the software DetectionEmotion (HER) developed in our work team and validated in [5] and [1], this allows to determine whether or not there is a dissociation between what the patient says -text- and his facial expression, in order to conclude a possible advance of the disease-alert. The next section shows the experimentation phase. The sample for our study is in an age range between patients ranging from 60 to 90 years. This age range is important in a disease such as Alzheimer’s dementia because it is progressive and usually has more involvement in the range chosen. In this investigation, it was necessary to formalize the record of videos of patients with Alzheimer of the Adult Hospital and Foundation Perpetuo Socorro (Quito), and the Center for the Elderly Adult (Catamayo), by signing a document between the parties in which the relatives authorized the procedure respecting the patient’s identity. Here the multimodal Bayesian network showed a better result with an F1Score of 0.8. Finally some conclusions and recommendations were obtained. The ones that are used as basis for further research.

2 State of the Art

According to [6] and [1] there is a direct relationship between text polarity and the emotion of the person, for example if the person recorded in the video mainly positive emotions, then the polarity of the text should be positive as well, if given this case would be considered a normal person, but in case there is no coincidence between the polarity of the text and the emotions identified in the facial expression of the person to be analyzed, this one presents an alert. An advantage of this semantic disconnection hypothesis is taken in order to develop our proposal and present an alternative for patients’ caregivers, which may indicate a possible advance of Alzheimer when there is no relationship between the variables. This proposal fits within the group’s general line of work to apply artificial intelligence techniques to advance the early diagnosis of mild cognitive impairment, proposed by [7] in which it tries to identify by means of neuropsychological tests the cognitive decline, which allows them to obtain new characteristics of quantitative description of the advance of the decline. In the research of [4] a decision model based on a Bayesian network is proposed to support the diagnosis of dementia and cognitive decline. The proposed Bayesian network was modeled using a combination of expert knowledge oriented to get data. The structure of the network was built on the basis of current diagnostic criteria contributed by experts in this field. The decision model was evaluated using quantitative methods and a sensitivity analysis. [3] proposed the Multifold Bayesian Kernelization algorithm which is a synthesis analysis of multimodal biomarkers, which builts a nucleus for each biomarker that maximizes the local affinity of the block, and also evaluates the contribution of each biomarker based on a Bayesian frame achieving significant improvements in all diagnostic groups as compared to the methods used in the technique. In [8]. It uses a Bayesian model to automatically identify distinct latent factors of overlapping atrophy patterns from structural magnetic resonance imaging in patients with late-onset Alzheimer’s disease (AD). The results show that different patterns of atrophy influence the decline of different cognitive domains [8]. The lexical-semantic-conceptual deficit (LSCD) in the oral definitions of the semantic categories of the basic objects is an important early indicator in the evaluation of the cognitive state of the patients. Bayesian networks have been applied for the diagnosis of mild and moderate Alzheimer by analyzing the oral production of semantic characteristics. The performance of BN classification is remarkable compared to other methods of mechanical learning, achieving 91% accuracy and 94% accuracy in patients with mild and moderate AD. Aside from this, the BN model facilitates the explanation of the reasoning process and the validation of the conclusions and allows the study of the rare declarative semantic memory deficiencies [9]. In this study it is intend to use the multimodal bayesian network to classify the conversations of the patients into positive, negative or neutral. It is required to make an analysis phrase by phrase. This result will be compared to the emotion that prevails in the patient’s face, all of this in order to identify the semantic disconnection. The details of the classification and comparison are described in the following section.

3 Methodology

Conversations in which people with Alzheimer participated were recorded on video and transcribed as reference to the orientations of [2] for the analysis of conversations in special populations and with that a corpus was constructed. The expert performs a manual labeling of the polarity of each phrase of the conversation. To proved the manual labeling, the Kappa index is run to obtain the correctness of the label (Fig. 1).

Fig. 1.
figure 1

Sentiment analysis process

Full size image
Fig. 2.
figure 2

Architecture system classification

Full size image

Figure 2 Shows the set of steps for the classification by means of the application of the probabilistic algorithms of Naïve Bayes and Multinomial Bayesian Networks: 1. Probabilistic weights of Multinomial Bayesian Network links are automatically learned from the linguistic corpus, NB estimates the conditional probability of a word or phrases given to a type as the relative frequency of the term (t) in files Belonging to type (c). 2. Variables considered as text or input factors are deterministic. 3. To find conditional probabilities a priori is calculated directly as the proportion of cases in the corpus. 4. The Multinomial Bayesian network takes into account the number of times a word appears on the files of class c. This model captures the frequency information of words in the file fragments, so the file is an ordered sequence of phrase events, extracted from the same vocabulary. Then, to find the variables of interest, Eq. 1 is calculated by simplifying Bayesian Multinomial Networks using the following phrases of a file.

Table 1. Example of a multinomial Bayesian network of training data set
Full size table

As seen in Table 1. There are instances that are previously classified (Positive, + Negative and Neutral) these files are used as training to determine the class of a new file.

Equation 1:

Formula for positive phrases file (Frequency of the word in file pos +1)/(No. input words + Positive words).

Formula for positive phrases file (Frequency of the word in the file neg + 1)/(No. input words + Negative words).

Formula for positive phrases file (Frequency of the word in the file neu + 1)/(No. input words + Neutral words) Vocabulary = life, happy, sad, house, wall Number of input words is taken into account Features List = 28 Where: CPOS: Positive Characteristics CNEG: Negative Characteristics CNEU: Neutral Characteristics.

The Eq. 1 that is used to perform the calculation and find the probabilities of the training phrases using the algorithm of the Multinomial Bayesian Networks in the programming language Python we can see the following form of the generated file:

To model a Multinomial Bayesian Network is described below: 1. It starts from the transcripts of the conversations obtaining thus the corpus of training. 2. The assignment of each class of phrases of the conversations is done subjectively for their previous classification. 3. A phrase is obtained for each probability of each polarity, using the algorithm of Multinomial Bayesian Network trained with the conversations. 4. Identify the polarity that has the highest percentage, to assign the most representative class (Table 2).

Table 2. File generated with the training phrases and their probabilities using the algorithm of the Multinomial Bayesian Network.
Full size table

3.1 Structure of Multinomial Bayesian Networks

In the structure of Multinomial Bayesian Networks the following can be found:

1. Standardized files are used, from which the phrases are extracted that will be sent to the model who will be in charge of assigning a polarity to each phrase of the corpus.

2. After obtaining the normalized files, the frequency of characteristics of the phrases is obtained.

3. Natural language processing techniques are used for classifying texts consisting mainly of finding patterns and characteristics of the language that allow assigning a class to a document. For the case of the classification of short texts, the task of preprocessing is also performed, in order to standardize the texts, to obtain documents with words or characteristics that can be understood by the system, which in Summary is the conversion of the document into a structure of boolean variables (True: present and False: absent), when the phrase is not in the positive file it is false (absent) and if it is in the file of negative sentences it will be true (present), based on the dictionary of words or slogans created from the training corpus.

4. The objective variable of this research is to find that early or healthy dementia values that can be taken in patients. Keep in mind that our model only takes into account the classes included in the corpus; However, it could easily be extended to the diagnosis of other problems that is cause by the cognitive decline.

5. The variable that refers to the three types of polarities (positive, negative and neutral) are those that evaluate the result of the highest probability percentage of the phrases. Implementation Output of the Multinomial Bayesian Network Algorithm.

The system generates outputs throughout its execution, which are useful in different processes. These outputs are flat txt files that are listed below:

  1. 1.

    Dictionary of features.

  2. 2.

    Classified documents.

  3. 3.

    Evaluation of results and performance.

The purpose of the dictionary is to serve as a resource for future implementations, it means to avoid create a new dictionary each time it is run.

In order to determine the class of a new phrase of a file the following is done:

  1. 1.

    Enter a new phrase

  2. 2.

    Perform data cleaning (data preprocessing)

  3. 3.

    Converting Documents to a Feature Vector

  4. 4.

    Modifying vocabulary with the new phrases

  5. 5.

    Checking the number of input words

  6. 6.

    Applying with the framework nltk and scikit-learn the algorithm as a new phrase, the probability for the new words are multiplied by an estimate value of 0.5 since this phrase had no previous labeling, nor was it in training.

The sample was taken of F1 as training and F2 as we do not know which polarity has performed, calculations were done with the positive, negative and neutral words and the phrase is classified according the majority probability obtained (Fig. 3).

Fig. 3.
figure 3

Structure of Multinomial Bayesian Networks

Full size image

4 Evaluation of Results and Performance

The dictionary of characteristics has the purpose of serving as a resource for future implementations, that is to say, to avoid the system creating a dictionary each time it is executed.

In order to determine the class of a new phrase of a file the following is done:

  1. 1.

    There is input a new phrase.

  2. 2.

    Perform the data cleansing (preprocessing of data).

  3. 3.

    Conversion of documents to characteristics Vector.

  4. 4.

    Modifying vocabulary with new phrases.

  5. 5.

    Applying the algorithm with the nltk and scikit-learn framework and as the phrase is new, the probability for the new words is multiplied by the value of estimation of 0.5 as this phrase didn’t have previous label, nor was it in the training.

Comparing the Bayesian Network model with the publication of [9], the main advantage of the model is that it has the ability to extend new categories to predict. In our model the same way has the ability to predict new phrases, otherwise with the resulting methodology new patterns can be applied for the Cognitive impairment. With our Bayesian model, and the [9] model, it is possible to expand a structure that can be used in clinical tests to detect Alzheimer in its early stages.

5 Experimentation

This section describes the tests and results obtained with the implemented system. It was necessary to have a corpus on which the experiments were performed, with 100 conversations of patients with early Alzheimer’s disease. The phases of the experiment were:

Training Phase: The training structure contained 3 files, having positive, negative and neutral sentences, previously classified by an expert. Later, we applied the comparison of the labeling by the expert and HER.

Table 3. Example of the Naive Bayes classifier and Multinomial NB of the conversations.
Full size table

In Table 3 the results obtained from the Bayesian classifier and Bayesian Multinomial Network are shown in each of the conversations as can be seen in conversation 2, 3, 5 in two classifiers point to the same class. Which in this case is Neutral, but in conversation 1 and 4, point to a different class in both classifiers, which is expected with the analysis given that the expert was asked to analyze each class (positive, negative, neutral) of the conversations of the dataset and thus verify if what has obtained Naive of Bayes and Bayesian Multinomial Network is correct. In Table 4 proved with the analysis in each conversation and the classification of both classifiers by the expert and the classification obtained by the Bayesian Multinomial Network have the same similarity in the conversation 2, 3 and 5. In Tables 3 and 4 the alert generating semantic disconnection rows are shown for the caregiver. We proceeded to apply the methodology and to obtain the accuracy and recall.

Table 4. Results of analysis with the expert Human and HER.
Full size table
Table 5. Sensitivity and specificity analysis using the results of polarity in text and emotions in video by the software Detection Emotion (HER).
Full size table

Table 5 shows the results of the use of the software Detection Emotion (HER) and the use of the feeling analysis in the tests. It can be observed that the results of precision and recall are good because the F1-Score index is close to 1.

6 Conclusions

The interest of this type of research allows to contribute valuable information of the advance of Alzheimer’s disease to the expert. Our proposal allows us to identify the semantic disconnections between the emotions in the patient’s face and what he says, while the same is recorded on video. The HER software was used to obtain the facial emotions, while Bayesian, Naive Bayes and Bayesian Multinomial Network algorithms were used to obtain the classification of the conversation. This last one obtained improvements in relation to the first reaching an F1Score of 0.82 in one of the applied tests. This value is close to other Alzheimer’s classifiers in which Bayesian networks were also used as in [9]. Bayesian networks are probabilistic and act closely to the human behavior therefore they allow to work with probabilistic errors in order to detect the progress of the disease, in fact the results are comparable between HER, Multinomial Naive Bayes and Expert as shown in Tables 3, 4 and 5. Our work is undergoing experimentation, therefore, our next step is to compare our results with the application of Bayesian methodologies applied in cerebral images for Alzheimer’s detection [3], to identify more clearly the difference between invasive and non-invasive methods for the detection of the disease.