Keywords

1 Introduction

The use of smartphones in the world has become common in people’s lives. These contain several features, from the simplest to the most complex, such as making calls and typing documents resembling notebooks. It is considered that “Currently there is a great interest of an organization in guaranteeing the quality of its products and services in order to remain in the consumer market” [1].

In Brazil, to carry out an analysis of the quality of products, the standards defined in technical standard 5426 must be followed to obtain, test, and/or compare to verify the list of possible defects according to their severity and identification of what is called AQL (Acceptable Quality Level) [2].

The AQL parameters also assess the superficial condition of smartphones during manufacturing. These are performed visually by employee in the productive environment, through a visual inspection process [3, 4] but this visual inspection can impact the worker’s visual stress because of a long periodicity of this activity [5].

Looking at the study done by Mora [6], it shows a sample of the main symptoms related to visual discomfort. This approach was defined to distinguish the different symptoms associated with this problem, namely the visual fatigue, blurred vision, visual irritability, headaches, stress, and difficulty concentrating. It is interesting to note that this portrays activities in the office, but it can lead to a parallel reference on what it would be like to work with vision in a factory environment.

For Mora [6], his study showed that for people who work in offices that 50% of employees wear glasses and 60% have ophthalmological problems, such as myopia and astigmatism, thus generating greater visual fatigue. The prevalence of the visual fatigue is expressed in percentage, relatively to the total number of the participants. In this case, 30% of respondents feel visual fatigue 17% have visual fatigue at the beginning of the day, 33% in the middle of the day and 50% at the end of the day. These results indicate that after several hours of continued work with few pauses, it can lead to long-term visual problems.

Among the types of symptoms that can be generated with the use of prolonged human vision, there are terms called fatigue and visual stress and for Pimenta et al. [7], fatigue is as one of the main causes of human error. Many times, its symptoms are ignored, as well as its importance for a good mental and physical condition, elementary for human performance and health. Fatigue is however a very subjective concept and difficult to define from a scientific point of view. It may be a combination of symptoms that include loss in performance.

During technological advances, methods of intelligent computing systems based on images like human vision, capable of being processed through computers, have emerged, according to Szeliski, 2010 (as cited Araújo [8]). These become alternatives for dealing with impacts related to visual stress in long periods and fatigue.

The technological fronts that can contribute to reduce impacts of visual stress and fatigue are computer vision & Artificial Intelligence (AI) through computer vision software and AI algorithms in machines. It has been defined as the science and engineering of making intelligent machines and as human-like intelligence, either in machines or software. AI technologies are used widely today to optimize processes, to make products easier to use, and to automate tasks. Within AI research, we look for new methods of solving problems, based on action-research approach [9]. This search generates challenges in terms of knowledge representation and reasoning Introduction 35 methods, planning, learning, natural language processing, motion, manipulation, perception, social and evolutionary intelligence, feelings, and creativity. These techniques can be applied and used in areas such as medicine, psychology, weather, finance, transportation, gaming, aviation, and in the law, where they have been applied. Many of these advances have been targeted at vehicle drivers [10].

This paper is organizing the following demonstrating the context of the importance of Smartphone manufacturing in society associated with good quality of human visual inspection and impacts, some technologies digitalization that can reduce these impacts, the application of the computer vision in the manufacture and conclusions.

2 Context

Today there is a use of approximately 355 million smartphones produced in the first quarter in the world, data from 2021, and massively manufactured by around 7 large companies, including: Samsung, Huawei, Apple, Xiaomi, Oppo, Vivo, Motorola. These data reflect the reality of a demanding market [11] in Fig. 1. This paper was analysed in a company from the Industrial Pole of Manaus, Brasil, in smartphone manufacturing.

Fig. 1.
figure 1

Smartphone production in the world.

Full size image

For Lobato [1], the market requires companies to produce quality products at a lower cost. To achieve results in this direction, it is necessary to maintain a dedication to constant improvements.

Looking at the course of history, for Pinheiro et al. [12], “quality went through four stages of development: inspection, process supervision, control and strategic quality management”.

“The first, the quality inspection phase, in which the final products were examined based on visual inspection, separating the products with defects that should be destroyed or return to the production process for correction” [13].

To understand a little more, about how superficial defect analysis has been done, the next section presents a brief description of the product quality analysis carried out in this work.

2.1 Product Quality Analysis Performed by People in the Production Environment

To analyse the quality of products, standards established in technical standard 5426 must be followed to obtain measurements, tests, and/or comparisons to verify the list of possible defects according to their severity and identification of what is called AQL [2].

Within the AQL parameters it is necessary to find the non-Conformity in the day-to-day and this is expressed in terms of “percent defective” or in terms of “defects per hundred units”.

  • Defective percentage:

    $$=\frac{Defective\,Units}{Inspectiined\,unit}\,x \,100$$
    (1)

  • Defects per hundred Units

    $$= \frac{Defective\,Units}{Inspectiined\,unit}\, x\, 100$$
    (2)

** any product unit may have one or more defects.

In addition to the representation of non-compliance in terms of defective parts, it is necessary to classify defects according to their severity, such as:

  • Critical defect;

  • Major defect;

  • Tolerable defect.

Therefore, the superficial inspection work itself, being a work entirely carried out by the human being, even if well directed by the AQL, is subject to errors at the time of its application. Thinking about it, and looking at existing technologies, there are currently computer vision techniques associated with Artificial Intelligence that can minimize these impacts, as can realised through the information summarized below.

2.2 Digitalization with Computer Vision and Artificial Intelligence Techniques

According to the Data Science Academy team, published in 2018 [14], computer vision is the process of modelling and replicating human vision using software and hardware that studies how to reconstruct, interrupt, and understand a 3d scene from its 2d images in terms of its properties of the structure present in the scene.

Therefore, the superficial inspection work itself, which guarantees quality, being carried out by the human being, and even if well directed by the AQL, is subject to errors at the time of its application. Considering this, and looking at existing technologies, there are currently techniques of computer vision system (SVC). With the technology advancing, methods of intelligent computer systems based on images, like human vision have emerged, capable of being processed through computers, or computer vision. In the context of the presented case study, i.e. detection of visual superficial defects on smartphone devices, there are multiple works that demonstrate various machine learning based approaches to detect smartphone surface defects, through computer vision [15]. The most common superficial defects are shown in Fig. 2 (Fig. 3).

Fig. 2.
figure 2

Common superficial defects as ‘areas of interest’.

Full size image
Fig. 3.
figure 3

Automatic detection of ‘areas of interest’ with smartphone surface defects.

Full size image

Computer vision can be used to digitize social media platforms to find relevant images that cannot be discovered through traditional searches. The technology is complex and, like all the tasks mentioned above, it requires more than just image recognition, but also semantic analysis of large data sets. Data Science published in 2018 [14].

Computer vision systems (SVC) started in the 70s with approaches in Artificial Intelligence (AI) according to Lopes [16]. It is a technology that has been gaining strength and giving directions to image processing, using cameras, optical sensors, scanners, among others [16].

It can be said that AI aims to perform and simulate human functions such as the ability to learn, adapt and make decisions in different situations according to knowledge according Oliveira [17].

In the field of AI, one can cite the branch named machine learning (ML), or also known as machine learning, which intends to perform human functions for learning with provisioning and decision-making [18].

Machine learning is a subset of AI applications capable of learning on their own. In fact, it reprograms itself, as it digests more data, to perform the specific task for which it was designed with increasing precision [19].

In this article ML is used to provide surface analysis on smartphones through computer vision and Artificial Intelligence techniques, but there are other 6 powerful, and frequently mentioned use cases for machine learning in manufacturing, which are: 1- predictive maintenance, 2 - predictive quality and throughput, 3 - digital twins, 4 - generative design/intelligent manufacturing, 5 - energy consumption prediction and 6 - cognitive supply chain management (six powerful use cases for machine learning in manufacturing) [20,21,22].

These approaches have technologically followed the main objectives underlying the Industry 4.0 (I4.0), which is aimed at digital manufacturing, also frequently called as smart factory, further associated to intelligent networks, mobility, flexibility of industrial operations and its interoperability” and collaboration [23,24,25].

In a general approach context, Artificial Intelligence is the area of computer science that emphasizes the creation of intelligent machines that work and react like humans. Machine Learning is a branch of Artificial Intelligence that allows computers to learn to perform new tasks without being explicitly pre-programmed for the corresponding end. Thus, it enables to study algorithms and perform statistical analysis by using computers to accurately perform a specific task without the use of explicit instructions.

In this work a mathematical model was built on sample data, known as “training data”, to make predictions or decisions without any direct programming to perform the task. These types of algorithms learn on their own and grow when new data is provided for the machine to learn.

Machine learning can be divided into three main branches, as we can be seen in Fig. 4, through which specific sub-branches are shown [10].

The sub-branches are:

  • Supervised Learning.

  • Unsupervised Learning.

  • Reinforcement Learning.

Fig. 4.
figure 4

Machine learning in its three main sub-branches [10].

Full size image

2.2.1 Supervised Learning

In supervised learning, the objective is stated with labelled/classified training data. There is a value, an assigned class, that allows you to have an idea of what you are looking to learn. That training data consists of the input vector X and output vector Y with labels. A vector Y label is the explanation of its respective input vector X data. Together, they form a training example that can describe a relationship that makes sense.

For example, as a child, parents teach their children about the names (labels) of objects by pointing at them and pronouncing their names, in a supervised manner.

2.2.2 Unsupervised Learning

In unsupervised learning, there is no supervisor indicating/labelling a vector with a defined objective and therefore there is no data for training. There is no clarity about what you want to learn. This is a fact that data scientists and machine learning professionals often find, where a large amount of data is available, but without labels. Still, it is possible to learn from the search for significant or hidden structures or behaviours in the data [10].

2.2.3 Reinforcement Learning

According to Deep Learning Book [26] the Reinforcement Learning, is the training of machine learning models to make a sequence of decisions. The agent learns to achieve a goal in an uncertain and potentially complex environment. In reinforcement learning, the artificial intelligence system faces a situation. The computer uses trial and error to find a solution to the problem. For the machine to do what the programmer wants, artificial intelligence receives rewards or penalties for the actions it performs. Your goal is to maximize the total reward. We can see below the Fig. 5.

Fig. 5.
figure 5

Reinforcement learning process.

Full size image

All branches lead to machine learning and show approaches that can be applied to consider what type of data is input and which answer is sought in the output of the modelled algorithm. Thus, it is necessary to analyse what type of data is generated in the smartphones’ surface defects inspection through a vision system (system input data) and which metrics are sought in the output.

3 Smartphone Production Quality Control in the Company

The general processes of the smartphone company are related according to the macro layout expressed in Fig. 6, and an explanation of each function, in Table 1. In the image, we can see where the superficial inspection process is, this at post 5, and in addition, there is a sample, at post 7, for review to review in a lower percentage if the smartphones have a good surface.

By analysis of BPMN As-Is, and in-loco check, it was possible to identify improvement opportunities, as listed below:

  • The data collected from some KCs is not always reliable and are not always recorded or are recorded incorrectly.

  • Manufacturing operators manually writes data on paper.

  • Quality operator needs to go to each machine to collect data and takes too long to manually write all data collected on a spreadsheet.

Fig. 6.
figure 6

Macro smartphone assembly process.

Full size image
Table 1. Macro details of mounting smartphones (own creation)
Full size table

During the monitoring of the process, in the period of 6 months of superficial inspection on smartphones (inspection of smartphones’ surface quality, whether there is any visible defect in appearance, such as scratches etc.), carried out by human operators (inspectors), it was observed that there were defective pieces that passes through the operators as if they were “good” products and, similarly good products, i.e. without any visible surface defects were labelled to be defective by the human operators. The portions of good quality product, and superficially defective product, out of all the smartphones manufactured in the company, and sampled at Station 7, are highlighted in Table 1. This index is based on the behavioural average of monthly production of the company. Looking at the graph in Fig. 7, which represents the inspection performed by the human operators, it is evident that there were failures that occurred in the judgment of the inspection station, the station 5. This could have been caused by several factors. Considering the data collected, the biggest challenge is to reduce the 1.67% of % false negatives, i.e., the undetected ‘defective’ phones. This could mean that the system may pass some superficially defective phones to consumers.

Fig. 7.
figure 7

Percentage of surface inspection – station 7 (own creation).

Full size image

However, in this company there is another filter that reviews the smartphones again before packaging. Therefore, the main objective of the company was to guarantee that it would not pass superficial defects in production, without the need for a second revision before packaging the products. Hence, it installed an automatic surface inspection machine to recognize superficial defects by the capturing images, through computer vision assisted by Artificial Intelligence using Reinforcement learning and Neural Networks (NN) algorithms. In this case, the model is trained with an image bank of surface defects. However, the company had not yet made an efficiency comparison between both alternatives (with and without the second inspection phase), to be able to clearly conclude whether the result of the completely automated station would be meeting its needs. Therefore, a case study was considered for comparing the efficiencies of both alternatives, the AI station and the human based solution, which will be briefly discussed below.

4 Results Discussion and Analysis

During a period of 20 days of smartphone production in the factory, a total of 1824 samples were collected for the present study. A machine with an automatic surface inspection was used to recognize superficial defects through the captured images, based on computer vision assisted by Artificial Intelligence using Reinforcement learning and Neural Networks (NN) algorithms. There were produced 3 batches, to compare the efficiency of the operator and the AI station. The objective was to understand the effectiveness of using artificial intelligence in the day-to-day processes in the production facility, to help employees in the detection of surface defects in the smartphones. These passed at the AI station in a similar way as those evaluated by the human operators, without one knowing what the other has identified. Again, since the objective of the inspection was to detect a defect, the samples with defect were labelled as ‘positives’ and the others are ‘negatives’.

After the period of collection and evaluation by both, the operator and the AI station, the results have been compiled, as summarized in Table 2.

Table 2. Smartphone’s inspection results – confusion matrix data for both inspection methods
Full size table

To evaluate and compare the two inspection methods, the values from the confusion matrix for each method from Table 2 were then applied to calculate the AI system evaluation metrics as in previous examples. The values of these metrics are presented in Table 3.

Table 3. Results of the analysis with the AI parameters with data from the operator and the AI station
Full size table

Based on the results of Table 3, it is necessary to understand what the client asked for as a priority in Business and data understanding, which was to avoid passing surface defects and this is represented by what we call recall, that is, the greater the recall the greater the capacity to identify defects. Considering what has been measured, the AI station has a greater capacity to detect defects as compared to the human operators’ powered station. This helps to achieve the objective of this company. Although the operator had greater accuracy, this does not represent the result that the client expects. This may have occurred due to visual stress of the worker, due to a long periodicity of this activity or the changing of operator in the 3 shifts, or some other issues which are not the scope of this study.

However, the AI powered station helped clearly to achieve the goal, under the given circumstances. In other words, after the comparison of the results in terms of evaluation parameters from both methods, it may be declared helpful to have the AI-powered station to help employees/inspectors in the detection of surface defects in daily routine of the production process. Besides providing better quality and efficiency, the AI powered station may help reducing the cost of quality control as well.

5 Conclusions

This work aimed at presenting the basic concepts in machine learning, showing its historical background and how the ML algorithms can be evaluated and applied in an industrial context, more precisely in smartphones surface quality inspection process. The main purpose was to enable a digital transformation in the factory, along with an improvement on production productivity, and in quality, along with costs reduction.

The case study considered for the application of data digitization analysis on superficial inspections carried out in smartphone manufacturing system, which did enable to realize about the support that digitization can bring to the daily lives of the company and its workers.

The main objective was achieved, through the application of machine learning in a world industrial scenario, through the application of data digitization analysis on smartphones’ surface visual inspection process in the considered company, which did enable to improve the process efficiency, and further in terms or production management, and quality control, while reducing costs. Therefore, the use of these kind of technology and approaches enable the manufacturing company to be more efficient and remain innovative and competitive. The main achievements are thus mainly related to, better quality control, higher production volumes, reduction of human stress due to repetitive labour (with occupational risks and avoiding human-induced errors), and gaining more accuracy through artificial intelligence powered machines with human support, thus achieving better general results.

Contrary to the third industrial revolution, which was a digital revolution, where computers started replacing humans for many ‘intelligent’ tasks, the current fourth revolution considers the ‘humans’ in the centre, despite some of the ‘intelligence-intensive’, ‘smart’ and ‘automatic decision-making’ tasks that have been being passed to the machines, as emphasized in [21, 22], regarding the two-part issues, related to the I4.0, and to the underlying importance of collaboration [25].