1 Introduction

Analyzing human skin health using artificial intelligence (AI) is neither a new idea nor a new technology. For decades, researchers have been developing automatic techniques to analyze human skin images in order to identify diseases, to quantify and to track their evolution over time. General-purpose image processing and, in particular, pattern recognition algorithms were used for this purpose. After the processing phase, extracted features are then passed through classifiers in order to assign each skin lesion to the most likely disease. Based on this analysis, this type of processing may somehow be considered as AI-based. However, it is controversial in the scientific community because some may argue that the processing models are not obviously inspired from human perception. Nowadays, technologies dedicated to skin analysis are getting much and more advanced. Consequently, AI-based methods make more sense due to the following reasons: (1) easy access to high-performance computers (2) recent advances in machine-learning and deep-learning, (3) increase of medical image databases in terms of quantity and quality. Considering dermatology applications, it is obvious that we reached a stage where dermatologists are currently using AI as a clinical assistant system. This statement doesn’t exclude the fact that, one day, AI might compete with humans in this field. Within this context, many research groups are collaborating with companies to develop interesting and promising applications such as the analysis of skin cancer. This is achieved by training the AI models over thousands or even hundred of thousands of skin images. Such systems are able to classify lesion images, either as benign or malignant (e.g., melanoma). Moreover, they can be used to predict or prevent the occurrence of some skin diseases. By pushing further the functionalities, AI-based systems can also consider multiple classes in order to distinguish between numerous skin lesions. Recently, smartphone applications have been designed for users so that they can remotely analyze their skin. Some of the functionalities aim to prevent the onset of skin diseases, and others provide useful skincare recommendations. Technically speaking, these applications require an interactivity between the user and the AI model. In some other cases, it can involve dermatologists when critical decisions are to be made. In this chapter, it specifically focuses on facial skin diseases using AI models. The case study in this chapter highlights an AI-based system (still under development) which is capable of classifying numerous facial skin diseases such as acne, Actinic Keratosis, and Blepharitis. This chapter is organized as follows: in Sect. 7.2, it focuses on the state of the art of facial skin diseases. In Sect. 7.3, a case study is presented and some specific AI approaches which are embedded in a software we have developed in Sect. 7.4 are discussed. Finally, a conclusion is provided in Sect. 7.5.

2 State of the Art

The use of automated methods to detect facial skin diseases has started in the past few years and was mostly focused on the detection of acne. Many techniques have been proposed and researches have been carried out to increase the accuracy of identification. Khongsuwan et al. used UV fluorescence images to detect and count acne. UV images are cropped, resized, converted to a gray image and enhanced and finally, H-maxima transform is used to count acne. Despite its efficiency, this technique may hurt the skin if exposed to UV light for a long duration [1]. Chang et al. detected automatically face skin defects and classified them into spots, acne, and normal skin using support-vector-machine-based classifier. The classification is preceded by an automatic selection of the region of interest and then extraction of the skin defects. They used images in front and profile views. The experimental results show the proposed approach is adequate and procures good accuracy [2]. Chantharaphaichit et al. proposed an automatic method to detect acne. Their approach consists first on marking the round-shaped areas in the face skin using blob detection method. Then they extract the specific features from the selected regions and finally send them to a Bayesian classifier to decide whether it is acne or not. The accuracy of this technique is affected by the place, form, and lighting conditions of acne in the image [3]. Kittigul worked on the detection and quantification of acne using Speeded Up Robust Features. She extracted 6 main characteristics to recognize acne. The classification is achieved by K-Nearest Neighbors algorithm [4]. Xiao lei Shen et al. detected facial acne vulgaris using CNN networks. Their method is based on two parts. Firstly, they detect skin to select the region of interest using a binary-classifier deciding skin or non-skin. Secondly, acne vulgaris is classified into one of seven classes including a healthy skin class using a seven-classifier [5]. Amini et al. developed a mobile application to detect and classify acne. The face image is captured by the camera of the mobile and then undergoes a series of image processing to extract the region of interest then identify and classify acne into two types: papules and pastules. The application could recognize and classify acne with high accuracy [6]. Binol et al proposed an aided diagnostic system to identify rosacea using deep learning. They used Inception and ResNet networks to extract features from facial images and then identify rosacea lesions [7]. Yang et al. developed a model to diagnose benign pigmented facial skin lesions using two CNN networks. They could identify six pigmentary facial skin pathologies. They extract lesions from images to train the networks. This model achieved good performance [8].

3 Study Case

We developed an automated system based on deep learning approach to help dermatologists diagnosing many facial skin diseases with high accuracy. Our system could recognize 8 face skin disorders, healthy skin, and no skin. For this purpose, we proposed a fine-tuned version of VGG-16 adapted to facial skin diseases recognition that we called Facial Skin Diseases Network (FSDNet). We created a dataset constituted of 20000 images gathered from online sources representing the ten classes classified by our network. The dataset is divided into training set to learn our network and fit the parameters and validation set to evaluate the efficiency and robustness of the system. The images are first labeled and then resized to fit our model. Five divide modes are used 90:10, 80:20, 70:30, 60:40, and 50:50 for training versus validation sets. Finally, to assess the effectiveness of the model, we give the system face skin images not included in the dataset to classify them.

3.1 Considered Skin Diseases

In our application, we considered eight most common facial skin diseases affecting all ages, genders, and races. These diseases are Acne, Actinic Keratosis, Angioedema, Blepharitis, Rosacea, Vitiligo, Melasma, and Eczema.

3.1.1 Acne

Acne is one of the most common facial skin diseases that affect especially teenagers. It is due to many factors such as hormonal changes, sebum production, bacteria, and inflammation [9]. It can be mild, moderate, or severe and is commonly located on the face (see Fig. 7.1), neck, shoulders, chest, and upper back; it is characterized by blackheads, whiteheads, pustules, papules, cysts, and nodules. This disease can scar or darken the skin if left untreated.

Fig. 7.1
figure 1

Acne disease [23]. a Acne in female, b Acne in male

Full size image

3.1.2 Actinic Keratosis

It is also known as Solar keratosis. Actinic keratosis or solar keratosis is a precancer characterized by scaly patches on the skin (see Fig. 7.2a). It is caused by recurrent exposure to ultraviolet radiation from the sun or tanning beds [10]. It can be found on the face, lips, ears, the backs of your hands, forearms, shoulders, scalp, or neck. It appears as small, scaly, thick spots on the skin of different colors.

3.1.3 Blepharitis

Blepharitis is an inflammation affecting the eyelids caused by bacteria, dandruff, rosacea, or even allergy to substances used in contact with lids [11]. This disorder is mostly characterized by redness, sticking, and crusting of the eyelids, itch, blurring sight (see Fig. 7.2b).

Fig. 7.2
figure 2

a Actinic Keratosis, b Blepharitis [23]

Full size image

3.1.4 Angioedema

Angioedema is a swell of short life in the layer just under the skin. It occurs in many parts of the body but mostly around the eyes and lips (see Fig. 7.3). The swell can cause ache and warmth in the affected area [12]. Angioedema can result from an allergic reaction to food, animal, plant, or drug and also can be hereditary.

Fig. 7.3
figure 3

Angioedema disease [23]. a Angioedema around the lips, b Angioedema around the eyes

Full size image

3.1.5 Vitiligo

Vitiligo is a long-term skin disease characterized by white spots (depigmentation) on the skin, of variable size, appearance, and location, which tend to enlarge (see Fig. 7.4). It is more observable in dark-skinned persons [13]. Vitiligo is caused by the loss of melanin due to an autoimmune disorder or heredity. Persons suffering from vitiligo are more exposed to sunburns, loss of hearing, stress, and sight problems.

Fig. 7.4
figure 4

Vitiligo disease [23]. a Vitiligo in male, b Vitiligo in female

Full size image

3.1.6 Rosacea

Rosacea is a skin disorder affecting mostly Caucasians persons of middle age in the face, mainly the nose, cheeks, forehead, and chin [14]. It occurs due to inflammatory, environmental, or hereditary factors. It has many symptoms such as red papules, red enlarged nose, red face bumps, swelling, eyelids redness, and dryness of facial skin (see Fig. 7.5a).

3.1.7 Melasma

Melasma is a skin pigmentation disease causing brown patches mainly on the face (see Fig. 7.5b). It affects women more than men and even persons with brown skin [15]. Sun exposure, hormone changes, pregnancy, and thyroid diseases may cause melasma. It doesn’t always necessitate treatment and also may come back even if treated.

3.1.8 Eczema

Eczema is a skin condition characterized by inflamed red and itchy skin. There are many types of eczema and it could affect all ages [16]. It is provoked foods, environmental conditions, soaps, detergents, sweating, and stress. Eczema is always accompanied by itch. Affected areas may be red, irritating, greasy, or oily (see Fig. 7.5c). Hair loss can occur in the area with the rash.

Fig. 7.5
figure 5

a Rosacea, b Melasma, c Eczema [23]

Full size image

3.2 Machine-Learning/Deep-Learning Approaches

Machine learning is a branch of artificial intelligence. It is described as the brain that permits systems to learn automatically, analyze data, and thus provide an adequate decision. Machine learning approaches are grouped depending on their learning type: supervised, unsupervised, and semi-supervised learning. It exists many machine learning techniques: regression algorithms including linear regression, and logistic regression, Decision Tree algorithm, Support Vector Machine, and Artificial Neural Networks forming the basis of deep learning.

Deep Learning is a sub-area of machine learning defined as neural network architectures with multiple layers and parameters. It is useful for recognition, information processing, and classification purpose and its use has broadened in industries such as cars, aerospace, electronics, and also in defense and medical fields. To obtain high system performance, deep learning requires large data to train the model and significant computing power to speed up data processing. There are many types of deep learning models:

  1. 1.

    Autoencoders which are unsupervised networks mainly used for dimensionality minimization.

  2. 2.

    Convolutional Neural Networks (CNN) used mainly in image recognition purposes.

  3. 3.

    Recurrent Neural networks designed for sequence identification such as speech, text, or signal.

  4. 4.

    Recursive Neural Networks used in natural language processing.

In our application, we are concerned by CNN models. CNN are the most prevalent type of deep learning models due to their architecture [17]. They are very efficient for image classification for many reasons. They use 2D convolution layers which makes them suitable to process 2D data such as images. CNN extract features automatically from images and learn them during the training process. This automatic extraction makes the model very accurate especially for computer vision purposes such as object classification and image classification. Even more, CNN can reduce the number of parameters in the network without losing any information which is very important in image recognition tasks. CNN architecture includes an input layer, convolution layers responsible for extracting the features, pooling layers reducing the number of parameters to learn while retaining information, and fully connected layers flattening outputs coming from preceding layers and predicting the class of the image (see Fig. 7.6) [18]. It exists in many CNN architectures differing by their structure, number of layers, and their design. The most important ones are LeNet [19], AlexNet [20], Googlenet (also called Inception) [21], and VGGNet [22] that we used in our application mainly VGG-16 model. VGG-16 is a deep network consisting of 16 weight layers: 13 convolutional layers and 3 fully connected layers followed by a Softmax classifier. The convolutional layers are divided into five groups each followed by a Maxpooling layer. We proposed a fine-tuned version of the VGG-16 model adapted to face skin disorders classification. We modify the structure of the fully connected layers (see Fig. 7.7).

Fig. 7.6
figure 6

General architecture of CNN

Full size image
Fig. 7.7
figure 7

FSDNet architecture

Full size image

3.3 Preliminary Results

In this method, 20000 images were used and partitioned randomly into training and validation purposes. Five divide modes were considered to study the efficiency of the network. FSDNet gave promising results in facial skin diseases recognition with accuracies of up to 97%. The accuracy and loss in each case are shown in Table 7.1 The accuracy varies between 94.9% and 97% and the loss between 0.2 and 0.09 for split cases 50:50 and 90:10, respectively, and a batch size of 16 and epoch size of 10.

Table 7.1 Accuracy of facial skin diseases network
Full size table

To study how well the diseases are identified, we calculated the confusion matrices presented in Fig. 7.8. These matrices show that few images were wrongly classified. Compared to related works in number and variety of used diseases, our model reveals uniqueness in the performance and implementation. It reached a precision comparable and in some cases higher than that obtained previously. Noting that the majority of works in the state of the art deals with acne and its subtypes.

Fig. 7.8
figure 8

Confusion matrix in five data split cases for training versus validation sets. a 90:10, b 80:20, c 70:30, d 60:40, e 50:50

Full size image

The model performance was tested with images from outside our dataset. They were all well classified accuracies up to 100% (see Fig. 7.9).

Fig. 7.9
figure 9

Test of images out of the database. a Acne classified with accuracy of 99.7%, b Melasma classified with accuracy of 98.1%, c Angioedema classified with accuracy of 100%, d Eczema classified with accuracy of 99.9%, e Vitiligo classified with accuracy of 100%, f normal skin classified with accuracy of 100%

Full size image

4 Developed Software

When the training and testing phase is finished, the trained deep learning model is ready to be deployed. Therefore, our biometrics group developed a software to become an assistant to the doctor in his work. This software can assist the doctor in many tasks including, organizing patients profiles, remote communication and consultations with the patients, notification system for easily notifying patients of new changes, classifying skin diseases, issuing treatments and tracking patient condition and treatment progress. The prototype software includes two parts: patient side and doctor side.

4.1 Patient Actions

After creating and submitting the registering form and creating an account, the patient can proceed to start having consultations and treatments from the assigned doctor while using many tools that will help him and the doctor build a patient profile with all the needed data. In the automatic diagnosis tab, the patient has the ability to take images that automatically crops the patient’s face, adjust image conditions (brightness and contrast) and save it to the database to be sent to the doctor for evaluation. In addition, the patient can use the trained Deep learning model to get a real-time prediction on the video feed with automatic face detection to identify which class the facial skin disease belongs to (see Fig. 7.10). This AI can be useful as an assistant to the doctor in identifying the skin disease type.

Fig. 7.10
figure 10

Real-time facial skin disease detection

Full size image

The patient can review the taken images in the E-Consultation tab, delete the selected images, upload new images from a local directory, and use the trained AI model to get class predictions on these locally loaded images. The doctor’s diagnosis area contains the final doctor’s decision about patient condition evaluation (see Fig. 7.11).

Fig. 7.11
figure 11

E-Consultation Tab

Full size image

A metric will be defined and calculated from several factors in the future to show the progress of the patient’s condition. This tab will include several visualizations of the metric starting from the start date of consultation and will include also visualization of treatments. These visualizations will help to track if the progress is becoming worse to do the necessary changes specified by the doctor or when the treatment becomes better then that means the treatment is going well. The chat box feature lets the patients chat with the assigned doctor for discussing the patient’s condition or any other matter related to this subject.

4.2 Doctor Actions

Patient data tab contains the images that the patient has uploaded using the real-time face capture feed or images uploaded locally. The doctor can use the trained deep learning model as an assistant to his classifying of the facial skin disease.

The doctor can also use the history table to keep track of the patient’s progress by adding new records containing the doctor notes taken in the evaluation date and entering the metric used to visualize the patient progress. Finally, the doctor can input his final evaluation for the patient to be shown in the doctor’s diagnosis area.

The face analysis tab provides the doctor with many useful features to evaluate the facial skin disease. This tab contains the following features:

  • Automatic selection of face regions

  • Manual selection of face regions

This feature allows the doctor to divide the face into the following parts:

  • Forehead

  • Left Eye

  • Right Eye

  • Left Cheek

  • Right Cheek

  • Mouth

  • Nose

After applying a transparent mask to visually divide the face to the mentioned parts, the doctor can select each part to zoom in to have a better view of the selected part for a better evaluation. This feature also allows us to gather data to train future deep learning models to classify individual parts of the face. In Fig. 7.12, we can see examples of automatic region selection.

Fig. 7.12
figure 12

Automatic region selection tool. a Forehead region, b Right cheek region

Full size image

Manual selection of face Regions tool gives the doctor the ability to draw a polygon over the desired area to crop it and visualize it in a new window for better condition evaluation. It can be used as one of the factors defining the metric. Saving the cropped area can also help gather a dataset to train a model for individual face parts classification. Figure 7.13 shows an example of the manual region selection tool.

Fig. 7.13
figure 13

Manual face region selection tool

Full size image

Allowing the patient to use the software freely without any doctor restrictions can be dangerous for the patients because they might stop consulting the doctor and use it for doing self-evaluations. Therefore, the doctor has access to the a permissions tab, which he can change depending on the access doctors think patients can have.

When the patient submits the information form, the doctor can visualize this information and use them to evaluate the skin disease. It will provide help with ways of treatment. These forms can be useful in gathering datasets that can be used in training a machine learning models to study patterns to detect the cause of some skin diseases or to be able to suggest treatment with the approval of the doctor based on input data of the model. A notification system will be created to allow the doctor to receive a notification about being assigned a new patient and any new changes done by the patient.

In this part, we reviewed the different functionalities and tools in the desktop application developed to help the doctor in the diagnosis process using the trained deep learning model, managing patients’ profile quickly, and making remote consultations easier and efficient for the patient. There are plenty of additions that could be done to improve the doctor–patient experience, such as expanding to new popular platforms(web application, mobile application).

5 Conclusion

In this chapter, an overview of artificial intelligence as an advanced technology in the field of dermatology is provided through a case study. The focus is on facial skin diseases using deep-learning based identification of skin diseases. One can point out that the whole facial features are considered as input for the AI-system rather than injecting separate lesions. Preliminary results highlight an accuracy of more than 94%. This is, in fact, a promising result when considering the technology as an “assistance-system”. At this stage, the role of dermatologists is either to confirm, or correct AI-system estimation. Such actions highly contribute to the increase of accuracy and, consequently, improve the performances. If the system is used by more than one dermatologist, more data is collected and the system will naturally be updated more often. Consequently, AI-systems will be smarter, in the sense that this will narrow down the differences between human-evaluation and machine-evaluation. In the future, it is technically not impossible for AI to provide better diagnosis than humans do.