Keywords

1 Introduction

In today’s modern era, skincare has become a common concern for both women and men, as the desire for beauty is universal. Among all aspects of skincare, the condition of facial skin holds paramount importance. Our facial skin is exposed daily to environmental factors such as smoke, dust, and ultraviolet rays, along with the effects of life pressures, work stress, nutritional deficiencies, and hormonal changes. These factors can weaken the facial skin and lead to issues like acne, dark spots, and freckles.

To maintain the long-term beauty and health of the skin, it is essential to engage in daily skincare routines that nourish the skin from within and protect it from external agents. Consistent skin care practices can effectively prevent early skin problems and minimize aging and damage. However, not everyone has sufficient time or knowledge about proper facial skincare procedures. The demands of busy daily lives often make it challenging to visit beauty salons or allocate hours for skincare routines. Furthermore, the ongoing challenge of the Covid-19 pandemic has further complicated matters, making it difficult for individuals to access beauty salons or seek professional advice while adhering to social distancing measures.

Artificial Intelligence (AI) has emerged as a practical solution in various domains [4, 5], attracting considerable attention due to its potential to process vast amounts of data at an unprecedented speed. AI technology enables machines to perceive the world like humans, learn autonomously, and solve real-life problems effectively. Computer Vision, a subset of AI, has witnessed rapid expansion and has found application in diverse real-world scenarios [6]. In light of these considerations, we propose a solution that leverages FastAI [1, 2], a powerful deep-learning framework, for detecting facial skin conditions and determining skin types. Additionally, we utilize TFLearn [3], a transparent and modular deep learning library built on top of TensorFlow, to enhance the accuracy and effectiveness of our approach.

By employing AI technologies for facial skin detection and classification, we aim to address the challenges mentioned earlier and provide individuals with a comprehensive skincare system that can assess their skin conditions and recommend suitable skincare routines. This solution offers a convenient alternative for those who have limited time, lack knowledge of skincare, or face difficulties accessing traditional skincare services due to the constraints of daily life or external factors such as the Covid-19 pandemic.

The remainder of this paper is organized as follows: Sect. 2 introduces some approaches and projects related to our work; Sect. 3 presents the proposed solution, Sect. 4 presents the experimental results, and lastly, Sect. 5 gives a conclusion of what has been currently achieved and future work.

2 Related Work

In the modern era of information and communication systems, people have become reliant on online applications [7], which have become indispensable tools for addressing various life challenges [8].

FastAI [1] is a deep learning library that offers practitioners high-level components capable of delivering state-of-the-art results in standard deep learning domains. Additionally, it provides researchers with flexible low-level components that can be combined to create novel approaches.

Numerous studies have emphasized the significance of skincare, given the difficulty people face in determining their skin type and effectively caring for their skin to maintain its health. This system incorporates CNN and DNN to implement facial recognition functionalities and chatbots.

Several existing applications cater to skincare users and offer various features, as outlined in Table 1. For instance, TroveSkin [9] enables users to identify their skin triggers by monitoring skincare products, lifestyle choices, habits, and more. VietSkin [10] serves as a platform connecting patients with dermatologists for remote dermatology consultations. Skincare Routine [11] assists in creating personalized skincare routines, organizing beauty and makeup products, understanding their proper usage order, and tracking usage. FeelinMySkin Skincare Routine [12] provides users with a comprehensive skincare app that facilitates routine creation for consistent skincare, tracks product usage, and expiration dates, monitors changes in the skin, and offers skincare tips and information about ingredient purposes.

Table 1. Compare FaSkare with the app on the market.
Full size table

Facial skin detection and classification using deep learning has garnered significant attention in the field of computer vision, resulting in numerous research endeavors. Noteworthy contributions in this domain include the works of Deng et al. [13], which provide a comprehensive overview of recent advancements in deep learning-based face recognition techniques, covering face detection, alignment, feature extraction, and classification. Mellouk et al. [14] present the current state-of-the-art in Facial Expression Recognition (FER) utilizing deep learning, discussing crucial insights and challenges across datasets and preprocessing, model architecture, training and optimization, and evaluation metrics. Prasad [16] proposes a deep learning-based approach for face representation under various conditions, encompassing occlusions, misalignment, different head poses, changing illuminations, and flawed facial feature localization. This paper investigates two popular deep learning models, Lightened CNN and VGG-Face, for extracting face representations. Additionally, Jin et al. [15] introduce Deep facial diagnosis, a technique that employs transfer learning from face recognition to facilitate facial diagnosis. This approach involves fine-tuning a pre-trained CNN on a smaller labeled dataset of facial images tailored to the specific diagnosis task. Deep facial diagnosis demonstrates promising results in diverse medical applications, including skin cancer detection, predicting cardiovascular disease risk, and identifying genetic disorders. The utilization of deep learning techniques in medical diagnosis holds the potential to enhance accuracy and efficiency, ultimately leading to improved patient outcomes.

Despite the abundance of beauty care applications in the current market, there is a lack of applications capable of swiftly detecting facial skin conditions or identifying acne types while seamlessly integrating additional utilities for users, such as personalized cosmetic suggestions, direct consultation with doctors, and scheduling within the system.

To address this gap, we have developed FaSkare, a comprehensive system that not only detects acne on the skin but also integrates a multitude of utilities to deliver an exceptional user experience.

3 Proposed Solution

A high-level overview of the process that we propose for building a facial skin condition detection system using deep learning is as follows:

  1. 1.

    Data Collection: Gather a diverse dataset of facial images that cover different skin conditions, such as acne, wrinkles, dark spots, rosacea, etc. The images are labeled with the corresponding skin condition for training purposes.

  2. 2.

    Data Preprocessing: Clean and preprocess the collected dataset. This step involves resizing the images, normalizing pixel values, and applying any necessary image augmentation techniques to increase the robustness of the model.

  3. 3.

    Model Selection: FastAI model and Convolutional Neural Networks (CNNs) are used for facial skin detection and image classification tasks (detailed in the experiment section).

  4. 4.

    Model Training: Split the dataset into training and validation sets. Use the training set to train the selected model on the labeled facial images. During training, the model will learn to extract relevant features from the images and classify them into different skin conditions. Optimize the model using an appropriate loss function and update the model’s parameters using backpropagation and gradient descent.

  5. 5.

    Model Evaluation: Evaluate the trained model’s performance on the validation set to assess its accuracy, precision, recall, and F1 score. Fine-tune the model and hyperparameters, if necessary, to improve its performance.

  6. 6.

    Deployment: the model can be deployed as part of a skincare system. This system can take input images of faces, process them using the trained model, and provide predictions and recommendations based on the detected skin conditions. It will be deployed as a web-based application - FaSkare.

  7. 7.

    Ongoing Improvement: Continue to collect new data and periodically retrain the model to improve its accuracy and keep it up to date with emerging skin conditions and trends. Incorporate user feedback and iterate on the system based on user experiences and requirements.

The process of facial skin condition detection and classification using a CNN model and Fastai can be carried out using five steps, as depicted in Fig. 1. The experimental section of this paper provides detailed information about each of these steps. In addition to facial skin condition recognition and classification, our approach involves the development of an intelligent chatbot using the TFLearn model, which offers automated skin care advice.

Fig. 1.
figure 1

FastAI model workflow utilizing our proposed solution.

Full size image

The overall architecture of our proposed solution, along with its components and related elements, is illustrated in Fig. 2.

Fig. 2.
figure 2

Architecture of the FaSkare system using Deep Learning

Full size image

This architecture consists of two subsystems: the customer subsystem and the management subsystem. Within the customer subsystem, there are implemented models for facial recognition, skin classification, and chatbot AI. Additionally, the customer subsystem includes components/features that support users, such as recommendations for skincare services and medical products, consultation services, scheduling appointments, and online payment. The management subsystem, on the other hand, comprises components that manage the post-service interactions related to skincare consultation for users who have utilized the online services. It also handles the management of interactions among relevant parties within the system. Table 2 describes the roles and responsibilities of each component in the architecture of the FaSKare system shown in Fig. 1.

Table 2. Component and Connector Element & Responsibility Description
Full size table

The system FaSkare has been developed based on the business need illustrated in Fig. 3 below:

Fig. 3.
figure 3

Context diagram of a FaSkare system

Full size image

Face Skin Detect: This component receives images from customers and predicts their facial skin condition and type.

Manager: Also known as the system administrator, this role involves managing the services, viewing statistics and reports, overseeing center employees, and handling user notifications regarding their schedules.

Doctor: The doctor directly examines customers’ skin and provides face-to-face or online consultations, including real-time chat and calls.

Employee: Employees can manage customer bookings, including accepting or denying appointments, scheduling bookings, maintaining customer records, engaging in chat conversations to address inquiries, and providing advice.

Customers: Customers have various capabilities, such as booking appointments and making online payments, engaging with a chatbot for automated assistance and answers to questions, communicating with doctors for advice and additional inquiries, using a camera or selecting a picture from their gallery to detect their facial skin condition, managing their medical records, and connecting with other individuals within the system’s community.

AI Chatbot: This intelligent chatbot is responsible for answering customer questions and performing automated tasks on their behalf.

4 Experiment Results

FaSkare is deployed according to the business requirements and system architecture depicted in Fig. 4. The system is deployed within a website environment, where it interacts with the server through APIs to access and modify data in the MySQL database. Additionally, the system utilizes APIs to provide results back to customers.

Fig. 4.
figure 4

Deployment Architecture of FaSkare

Full size image

The implementation of facial skin condition recognition and classification, based on the FastAI model, is carried out through the following five steps in the experimental phase.

Step 1: Download data or prepare data set and label for them. Figure 5 shows the example of preparing and labeling dataset and data training for each skin.

We have built datasets for 20 different facial skin types shown in Table 3.

Table 3. The dataset for training and testing of 20 facial skin types
Full size table

STEP 2: Structure Data has restructured the datasets before training them.

figure a

FastAI has an extremely flexible system called the data block API. With this API you can fully customize every stage in the creation of your DataLoaders. Here is what we need to create DataLoaders for the dataset that we just prepared.

Fig. 5.
figure 5

Example of prepare and label data set and data train for each skin type

Full size image

We provide a tuple where we specify what types we want for the independent and dependent variables: blocks = (ImageBlock, CategoryBlock).

  • The independent variable is the thing we are using to make predictions (image) - is often referred to as x.

  • The dependent variable is our target (label - type of facial skin condition) for each image. - is often referred to as y.

The get_image_files function takes a path and returns a list of all of the images in that path. We simply want to split our training and validation sets randomly.

  • Validation: 20%

  • Training: 80%

  • parent_label is a function provided by FastAI that simply gets the name of the folder a file is in.

  • To group them in a big array (usually called a tensor) that is going to go through our model, they all need to be of the same size (224-pixel square). We use 224 pixels because this is the standard size for historical reasons. If you increase the size, you’ll often get a model with better results (since it will be able to focus on more details), but at the price of speed and memory consumption; the opposite is true if you decrease the size.

  • Data augmentation refers to creating random variations of our input data, such that they appear different, but do not actually change the meaning of the data. Examples of common data augmentation techniques for images are rotation, flipping, perspective warping, brightness changes, and contrast changes.

Step 3: Create DataLoaders.

dls = skin_condition.dataloaders(‘./skin_condition’)

  • DataLoaders: A class provided by FastAI that allows the storage of multiple DataLoader objects. Typically, it includes a training DataLoader and a validation DataLoader.

  • For these DataLoaders, the underlying items are file paths. It is necessary to instruct FastAI on how to obtain a list of these file paths.

Step 4: Create Learner:

Learn = cnn_learner(dls, resnet34, metrics = error_rate)

FastAI is utilized to create a convolutional neural network (CNN) by specifying the desired architecture, training data, and metric selection. Here are the key points:

  • CNNs are the cutting-edge approach for developing computer vision models, drawing inspiration from the human visual system’s functioning.

  • Specifically, we will employ the ResNet34 architecture, which is a state-of-the-art model widely used in solving various computer vision tasks. The “34” in ResNet34 denotes the number of layers in this variant (other options include 18, 50, 101, and 152).

  • Models with more layers require longer training time and are susceptible to overfitting. However, with larger datasets, they can yield higher accuracy.

  • A metric is a function that assesses the model’s predictions using the validation set and is displayed at the end of each training epoch.

  • The error_rate metric, provided by FastAI, determines the percentage of misclassified images in the validation set.

  • The accuracy metric can be calculated as 1.0 minus the error_rate, providing the proportion of correctly classified images.

Step 5: Train model:

  • Fine-tuning: Fine-tuning is a transfer learning technique where a pre-trained model’s parameters are utilized, and these parameters are updated by training for additional epochs using a different dataset.

  • When employing the fine-tuning method, FastAI automates several techniques for you. There are a few adjustable parameters, and the process consists of two steps:

    • The first step involves fitting only the necessary parts of the model for the new random head to function with your dataset. This is accomplished during a single epoch.

    • The second step involves fitting the entire model using the predefined number of epochs specified when calling the method. During this step, the weights of the later layers, particularly the head, are updated at a faster rate compared to the earlier layers. Typically, the earlier layers do not require significant modifications from the pre-trained weights, as we will explore further (Fig. 6 and Table 4).

Fig. 6.
figure 6

Train model

Full size image

Result of the Experiment:

Table 4. The experiment results.
Full size table

Discussion:

The experimental results demonstrate the practicality and effectiveness of FaSkare. The average time per detection attempt is approximately 0.2 s on a 200 x 200px frame, with slight variations for larger frames. The overall accuracy of FaSkare is reported to be 75%, with over 85% accuracy on facial skin with a filled background and average size. However, the accuracy drops below 65% when the facial skin occupies more than 60% of the image.

Fig.7.
figure 7

Example screenshot of FaSkare for detection of skin type

Full size image

We acknowledge the limitations encountered during the research, such as the need for a larger and more diverse dataset for certain facial skin conditions. Additionally, this study highlights the potential for future improvements, such as incorporating more advanced image augmentation techniques and exploring other deep learning models for enhanced accuracy. Figure 7 demonstrates an instance of skin type detection.

5 Conclusion

The study presents FaSkare as a comprehensive system that addresses the lack of applications capable of swiftly detecting facial skin conditions while seamlessly integrating additional utilities for users. By utilizing the FastAI model and Convolutional Neural Networks (CNNs), FaSkare demonstrates its effectiveness in detecting and classifying various skin conditions, including acne, wrinkles, dark spots, and rosacea.

The result highlights the importance of data collection and preprocessing in building an accurate and robust facial skin condition detection system. By gathering a diverse dataset of labeled facial images and applying necessary preprocessing techniques, FaSkare ensures the reliability of its model.

The limitation of the study is the need for a larger and more diverse dataset for certain skin conditions that will improve in the future. We also explore advanced image augmentation techniques and consider alternative deep learning models to further enhance the accuracy and performance of FaSkare.

Overall, the study result demonstrates the effectiveness of FaSkare in providing an exceptional user experience by offering personalized cosmetic suggestions, direct consultation with doctors, and scheduling within the system. FaSkare fills the existing gap in the market for comprehensive beauty care applications and showcases the potential of AI-based solutions in the beauty industry.