Abstract
Metal additive manufacturing (AM) has the advantages of novel part morphology, new material property, and short production cycle, attracting significant attention in automobile, aerospace, and other fields. Nevertheless, the repeatability and stability of part quality are difficult to guarantee in the current metal AM process, hindering the development of metal AM technology. Vital process variables such as molten pool characteristics, spatter characteristics, surface morphology, and radiation temperature are directly related to the quality of parts. Monitoring the dynamic changes vital variables is critical. Image processing techniques such as image transformation, recognition, segmentation, and enhancement are introduced to obtain manufacturing process information through processing and analyzing rich image features. The monitoring system’s processing results are usually collected in real time to help solve the repeatability and stability problems of the metal AM process. Presently, in situ and real-time monitoring methods are the most popular, and unfortunately, the literature lacks a comprehensive report. Thus, this paper thoroughly describes in situ and real-time process monitoring on the basis of image processing for metal AM. This paper reviews the in situ and real-time monitoring on the basis of traditional image processing to analyze monitoring objects, process classifications, and image processing carriers. Then, the advantages and disadvantages of in situ and real-time monitoring of metal AM based on artificial intelligence are discussed and compared. Finally, image processing algorithm generalization, quality, small samples, and image labeling problems are analyzed and discussed. A technical route for metal AM real-time feedback control is proposed by combining image processing with other technologies.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Metal additive manufacturing (AM) technology has developed into a fascinating digital manufacturing technology, attracting tremendous attention across the globe [1,2,3,4]. Metal AM technology has several outstanding advantages: (1) It can manufacture complex parts because there may not be the limitation of traditional processing; (2) it does not take up space and is portable so that it can be used in disaster areas and battlefields; (3) it produces new material properties, such as the introduction of dislocation networks while significantly enhancing the strength and ductility of the parts [5]; and (4) it requires short production cycle. The metal AM can be divided into several main types according to different manufacturing technologies. Traditional metal AM technologies include laser powder bed fusion (L-PBF), electron beam melting (EBM), wire arc AM (WAAM), and laser-directed energy deposition (L-DED). Metal AM can also be divided into powder bed fusion (PBF) and deposition energy deposition (DED) according to the molten types of materials. SLS, L-PBF, and EBM are the import components of PBF. SLS and L-PBF use a laser as an energy source to melt selected areas of powder, layer by layer, through computer-aided design data. In L-PBF, the material is completely melted rather than sintered as in SLS [6]. EBM, similar in principle to SLS and L-PBF, uses an electron beam as a heat source to create parts by selectively melting each layer of powder [7].
WAAM is a process that melts metal wire by using an electric arc [8]. L-DED is a process that blows powder into the molten pool to make parts [9]. WAAM and L-DED are import components of DED. There are also some other metal AM technologies. Laser cladding (LC) is a variant of DED. It uses the same principles as the DED but adds additional metal layers to existing parts [10]. Plasma arc additive manufacturing (PAM) is a kind of DED that uses plasma arc as a heat source but wire or powder as raw material [11]. Table 1 summarizes the characteristics of different metal AM processes by listing critical detrimental properties and critical process variables.
The significant detrimental properties associated with metal AM include pores, cracks, spheroidization, and residual stress. These defects seriously affect the internal structure and mechanical properties of parts, and ultimately affect the quality of parts [12,13,14,15]. During the manufacturing process, vital process variables such as molten pool characteristics, spatter characteristics, surface morphology, and radiation temperature are directly related to the above defects. Monitoring the dynamic behaviors of the above variables is essential to ensure parts’ quality. By adding monitoring equipment beside the printer, the process of metal AM can be monitored. There are many monitoring methods, including laser ultrasonic monitoring and optical imaging monitoring [16,17,18]. Koester et al. proved that metal AM processes could be detected using a passive acoustic monitoring [19]. Donadello et al. proposed a system to measure deposition height during the L-DED process, extended to complex 3D geometries and different materials [20]. Li et al. proposed a coherent gradient sensing method to monitor stress and strain by first measuring the real-time full-field deformation of a thick substrate during a disc-shaped specimen deposition [21]. Then, they analyzed the deformation distribution characteristics at different positions. Min et al. developed a system on the basis of a miniature electron impact ion source time-of-flight mass spectrometer. Their proposed system could monitor the changes in the atmosphere above the molten pool area in real time [22].
As the visual basis of human perception of the world, an image is a means of obtaining, expressing, and transmitting information. Image processing transforms two-dimensional data for image information retrieval [23]. Image processing technologies include transformation, recognition, segmentation, and enhancement. Image transformation captures information in different spaces using techniques such as Fourier transform [24]. Image recognition can identify different objects by analyzing and understanding images [25]. Image segmentation divides an image into distinct, uniform regions but variable adjacent regions [26]. Image enhancement filters specific noise to make the edge part clearer and highlights the needed features [27]. Table 2 summarizes the possible applications of image processing techniques in monitoring metal AM processes. As shown in Table 2, image processing technology has a great application prospect in metal AM process monitoring. Reviews from the literature have mentioned image processing applications for metal AM, but no comprehensive report focuses on its processing methods. Everton et al. conducted an overview of in situ process monitoring and metrology of metal AM, mentioning the applications of in situ image process monitoring [28]. Digital image correlation technology carries out correlation operation on two digital images before and after the part deformation to obtain the deformation information of the region of interest. Cunha et al. reviewed digital image correlation technology for the in situ metal AM process monitoring [29]. It mainly included defect characterization, residual stress evaluation, geometric distortion, and numerical simulation verification based on field measurement, monitoring, and part feature description.
Real-time acquisition of monitoring results is used to judge the quality of parts in real time, which is helpful to solve the problems of repetition and stability of the metal AM process. Presently, in situ and real-time monitoring methods are essential for the manufacturing process of products. Thus, this paper reviews the progress of in situ and real-time monitoring of metal AM on the basis of image processing. The difficulties and solutions of image processing techniques applied to in situ and real-time monitoring of metal AM are analyzed and discussed. A detailed and feasible technical route for metal AM real-time feedback control is proposed by combining image processing with other technologies. The remainder of this paper is organized as follows: Sect. 2 introduces the application of traditional and artificial intelligence (AI) image processing in in situ and real-time monitoring of metal AM processes. Section 3 discusses the challenges encountered and possible image processing and real-time feedback control solutions. Finally, Sect. 4 concludes the paper.
2 Methods
Eighty-three percent of the information humans acquire comes from vision. The information obtained in monitoring metal AM processes based on image processing is more abundant than monitoring technologies like laser ultrasound. The dynamic characteristics of the molten pool and spatter, such as the evolution of size, quantity, and temperature, can be obtained from monitored images; the topography of each layer of the parts can also be acquired at higher image resolutions as well as cracks on the surface of the parts. With the development of metal AM-related technologies, images will be able to obtain more information. To solve the repeatability and stability problem of the process, it must be possible to monitor the manufacturing process in real time. In situ and real-time monitoring is the mainstream monitoring method. At present, some work has been done for in situ and real-time monitoring of metal AM processes by using image processing techniques. According to the image processing methods, existing works from the literature are mainly divided into traditional image processing and AI image processing. Table 3 summarizes the current representative work.
2.1 In situ and real-time monitoring of metal AM process based on traditional image processing
In situ and real-time process monitoring based on traditional image processing has been well studied in the literature. The traditional image processing algorithms mainly include image transformation, threshold segmentation, edge extraction, and projection transformation.
These algorithms can be used to monitor the size of the molten pool. Hofman et al. adopted filter and threshold processing on the basis of a CMOS camera to obtain the width of the molten pool in real time during the LC process [30]. The width of the molten pool was kept within a specific range via feedback-adjusted laser power, improving the quality of the parts. OpenCV is the huge open-source library for the computer vision, machine learning, and image processing. And it can carry out rich image processing operations. For L-DED processes, Moralejo et al. used the image processing algorithm in OpenCV and optimized the processing time to 10 ms to detect the molten pool’ size in real time [31]. A unique feedforward controller was designed for real-time control and in situ adjustment of molten pool morphology. Similarly, Song et al. proposed a phase congruency with a log-Gabor filter to extract molten pool edge [32]. Compared with the traditional threshold methods, the methods proposed by Song et al. method could effectively eliminate the interference caused by plasma, molten pool disturbance, and other factors. It could be used for real-time monitoring of molten pool geometry. In 2020, Yang et al. proposed a coaxial CCD imaging system with a dual-wavelength narrow-band filter to obtain the molten pool image [33]. Then, the minimum boundary rectangle method was used to extract the molten pool width; the molten pool’s width was controlled by PID feedback. This method could effectively improve the quality of samples.
The above methods all used CPU for image data processing. The field-programmable gate array (FPGA) chip is used in in situ and real-time process monitoring to reduce image transmission time and improve image processing speed. Seltzer et al. designed corresponding optical equipment and image processing devices to control molten pool’ geometry in real time [34]. FPGA chip was adopted to realize the image processing algorithm for estimating molten pool width and build height in the image processing. Some work has also been done during L-PBF process. Craeghs et al. proposed a real-time visual monitoring system for molten pool shape during the L-PBF process [35]. They adopted FPGA chip technology to realize fast and real-time processing of molten pool images. Clijsters et al. proposed a monitoring system comprising software and hardware for L-PBF to monitor the molten pool in real time at high speed [36]. Figure 1a shows the experimental optical platform of the real-time monitoring system. The FPGA chip was adopted to achieve better data analysis speed and molten pool image processing technology. Figure 1b shows the monitoring curves of the molten pool area at different positions during the manufacturing process. In the above work, different algorithms and hardware processing systems were used to monitor the molten pool morphology in real time, but the monitoring accuracy may be limited in complex environments due to the robustness of the algorithm.
The above work almost exclusively monitored a single object. Zhang et al. proposed a monitoring system for multiple objects and an off-axis visual sensing system on the basis of high-speed cameras [37]. The characteristics of the molten pool intensity, plume area, orientation, spatter number, spatter area, and spatter velocity were obtained through the Kalman filter and image segmentation algorithm, providing a good foundation for real-time monitoring and feedback control of the L-PBF manufacturing process. It can be seen from the above work that L-DED and L-PBF are the main manufacturing processes used to obtain the geometrical morphology of molten pool. This may be due to the mature development of L-DED and L-PBF technologies, making it more convenient to realize in situ real-time monitoring.
Traditional image processing algorithms can also monitor the geometries of parts and feedstock. Asselin et al. implemented trinocular optical CCD-based detector, threshold segmentation, and projection transformation to realize in situ and real-time image processing to obtain the clad height, clad width, and solidification rate [38]. Unfortunately, rapid heating and cladding may cause cracks and other defects in parts. Biegler et al. first achieved in situ monitoring of the printing part deformation rather than the substrate during the L-DED process by using digital image correlation with the optical filter [39]. Figure 2a shows the in situ monitoring experimental equipment of deformation. Figure 2b shows the monitoring curve of workpiece wall deformation during the manufacturing process. Yao et al. proposed a new multifractal methodology of image contour by monitoring part surface profile. The quality of parts could be controlled during the L-PBF process [40]. During the WAAM process, the wire feed position dramatically influences the final quality of the parts. Zhan et al. proposed a vision-based detection of wire feed position deviation in the WAAM process [8]. The edge of the wire was obtained using the adaptive threshold and Hough transform, and the deflection was measured by Radon transform. The above work shows that the traditional image processing method can quickly obtain parts and feedstock geometry. The above work only monitored the surface geometry information of solids during the manufacturing process, and did not involve temperature information and internal defect information, which was not enough to realize monitoring of the part quality during the manufacturing process.
Radiance temperature characteristic can also be used for in situ and real-time monitoring of metal AM. Barua et al. focused on vision-based defect detection during the L-DED process [41]. The heat conduction will change when defects occur during the printing process so that the defect can be detected. There is a linear mapping relationship between the RGB value of the image and the workpiece temperature. The temperature distribution of the workpiece can be obtained by data mapping of printing process images taken by a camera. Figure 3a shows the in situ monitoring experimental equipment. Because of the uneven deposition, it could be found that the radiation temperature measurement of the deposit changed significantly. Figure 3b shows the temperature distribution near the defect. Khanzadeh et al. proposed a dual control chart system that used T2 and Q control charts to detect the extracted low dimensional features and residuals, respectively. Principal component analysis (PCA) reduces the dimension of data by mapping multiple variables to fewer variables with almost the same amount of information. The authors adopted multilinear PCA to extract the low dimensional features and residuals of the process thermal image before monitoring the abnormality of the printing process [42]. Figure 4a shows the experimental equipment for in situ monitoring of the manufacturing process. Figure 4b shows the judgment result of adopting dual control chart. The core content of the above work was that the temperature was abnormal indirectly indicating that defects might occur. In addition, the monitoring of metal additive manufacturing based on radiation characteristics might be susceptible to environmental factors leading to problems in collecting data or judging defects; thus, the controllability of the manufacturing process and the processing environment is crucial.
According to the highlighted works from the literature, in terms of monitoring objects, using traditional image processing for monitoring molten pool geometry is the focus of scholars’ investigation, followed by the monitoring of parts geometry and temperature anomalies. The traditional image processing algorithms can realize fast feature extraction from images captured in simple manufacturing environment, but may be not feasible for low-quality images captured in complex manufacturing environment. Thus, it is necessary to introduce an AI algorithm to monitor the manufacturing process.
2.2 In situ and real-time monitoring of AM process based on AI image processing
AI has been widely used in robots, control systems, and simulation systems. In recent years, it has also been gradually applied to metal AM. AI, based on image processing, accounts for a large proportion. Its introduction makes it possible to process the image of the metal AM process in real time and adjust the process parameters. Studies from the literature on in situ and real-time process monitoring of metal AM based on AI image processing from four aspects: processing objects, algorithm type, accuracy, and processing time. Table 4 shows the comparative analysis of these aspects.
Image processing based on AI can be used to monitor the geometries of molten pool and spatters. Some work has been done during L-DED process. Supervised learning uses a group of known samples to adjust the parameters of AI algorithm to achieve the required performance. Khanzadeh et al. introduced supervised learning into porosity prediction. They proposed an AI framework to establish the relationship between molten morphological characteristics and microstructure abnormalities by real-time monitoring of molten pool abnormalities [43]. Unlike the traditional method, their proposed model detects the porosity after manufacturing. A new data processing method was proposed to reduce the dimension of the thermal image of the molten pool. Then, the porosity could be predicted in real time with high accuracy by using the reduced dimensional characteristics of the molten pool boundary. Five supervised learning methods to classify standard molten pool and pores were analyzed. Figure 5a shows real-time monitoring experimental equipment. Figure 5b shows the results of K-nearest neighbor (KNN) which had the highest rate of accurate classification under the first and second principal components.
Zhang et al. built an in situ porosity monitoring system during the L-DED process [44]. As shown in Fig. 6a, the system used a high-speed digital camera to collect the dynamic image of the molten pool in situ. A set of image processing tools was developed to mark the molten pool data by cross-sectioning. X-ray tomography methods were used to obtain the porosity of the sample. Convolutional neural network (CNN) can extract the features of interested objects from images through a large number of convolution operations. A compact CNN architecture was designed to establish the relationship between molten pool image and sample porosity by learning the characteristics of molten pool image. Figure 6b shows the prediction results of local volume porosity of the two samples by using CNN. The above experimental results showed that the method of predicting sample porosity through real-time monitoring of molten pool morphology features based on AI algorithm has a promising potential. However, in order to achieve an accurate prediction of sample porosity, a large number of experiments are needed to obtain enough sample data.
In 2021, Mi et al. proposed an in situ monitoring system for the L-DED process with a high-speed camera, image processing system, and laser light source [45]. As shown in Fig. 7a, the system could quickly collect process images, and the information that could be extracted includes the area of the molten pool and the number of spatters. The deep CNN (DCNN) was introduced to monitor the L-DED process; by adopting the lightweight network structure and improving the loss function, the molten pool and spatters could be segmented with high precision and high speed to achieve the purpose of real-time processing of the process images. Lightweight network refers to the modification of network structure to reduce network parameters and keep the data processing effect unchanged. And loss function refers to the calculation of the difference between the predicted value and actual value, which is used for the iterative modification of network parameters. Figure 7b shows the extraction results of the molten pool and spatters. The position positioning of spatters was realized for the first time, and the change of spatter position was analyzed. However, due to the wide spatial distribution of spatters during L-DED process, the geometric characteristics of spatters captured by high-speed cameras with different focal lengths will change in some degree, which may affect final performance of AI algorithm.
For L-PBF processes, Yang et al. focused on the real-time monitoring and classification of molten pool in the L-PBF process [46]. Using CNN, the size of molten pool was classified into four categories: no molten pool observed, smaller molten pool, normal molten pool, and larger molten pool with high speed and high precision. Figure 8a shows the monitoring equipment, and Fig. 8b shows the real-time classification of molten pool size on the energy density (expressed by the ratio of power and scanning speed). The monitoring platform of their work was complex, but the processing results were more accurate.
To understand the dynamics of the L-PBF process, Fang et al. [47], Tan et al. [48], and Yang et al. [49] established in situ and real-time monitoring systems for the L-PBF process as shown in Figs. 9a, 10a, and 11a. A high-speed camera was used to obtain the image of the L-PBF process, and an AI algorithm processed the image to extract the information of molten pool and spatters. Fang et al. focused on capturing the characteristics of the molten pool in real time [47]. The U-net based on a CNN was used to extract the molten pool. The neural network introduced a lightweight network to reduce the calculation time and improved the loss function to improve the accuracy of the molten pool. As shown in Fig. 9b, the molten pool could be extracted accurately. Tan et al. focused on capturing spatter features in real time [48]. The captured L-PBF process image was segmented into blocks. The obtained block image was segmented by a two-layer network composed of a convolutional neural network and a threshold neural network (TNN) to obtain the spatter morphology information. The final spatters segmented image was obtained by combining the block image. This method had the advantages of simple labeling of training data and high extraction accuracy. Figure 10b shows the spatter extraction accuracy. Fang et al. [47] and Tan et al. [48] provided powerful help for predicting the formation of defects in the L-PBF process.
Yang et al. also focused on real-time capture of spatter features [49]. Different from the neural network methods of Fang et al. [47] and Tan et al. [48], Yang et al. used the maximum-entropy double-threshold image processing algorithm based on genetic algorithm (MEDTIA-GA) to extract spatter morphological information, effectively avoiding noise sensitivity. There were three types of error conditions: noise sensitivity, spatter conglutination, and spatter omission. As shown in Fig. 11b, their proposed method could extract spatter information at high speed compared with three traditional image segmentation algorithms. Finally, the relationship between spatters area, quantity, and laser energy density was analyzed in detail. The phenomenon that a metallic jet connected with the molten pool could be generated under high energy density was obtained. In the work of Fang et al. [47], Tan et al. [48], and Yang et al. [49], there was a smear phenomenon in the spatter morphology, which may need to be eliminated by high-performance high-speed cameras and image processing algorithms.
During WAAM processes, Wang et al. built the molten pool monitoring system, as shown in Fig. 12a [50]. The improved prediction network (PredNet) predicted the future molten pool image. The obtained prediction image was input into the proposed regression network (SERes) to analyze the weld reinforcement in advance. Verified by experiments, the prediction network could predict the molten pool image after 140 ms for a long time. Figure 12b shows the actual molten pool image and predicted molten pool image under different time steps. It was observed that the prediction results were good, and the method could analyze the quality and accuracy of welding shape, which was of great significance for AM process real-time feedback control.
In 2021, Lu et al. designed monitoring equipment for the molten pool to predict deposition reinforcement in AM, as shown in Fig. 13a [51]. The preprocessed molten pool image was input into the residual-based network designed, and the weld reinforcement was obtained quickly. The residual network structure can effectively reduce the gradient vanishing problem when the network is deepened. A multisystem suitable measurement method was proposed to unify the molten pool image, weld position, and corresponding weld reinforcement. The molten pool vision sensor was used to collect the molten pool image in real time, and the positioning system and 3D system were used to detect the corresponding reinforcement of each molten pool. Figure 13b compares distinct layers’ predicted results and errors. Nevertheless, their proposed network could predict the weld reinforcement in real time only when the deposited layer is well-formed.
Zhang et al. built an in situ online monitoring system to monitor the PAM process, as shown in Fig. 14a [11]. The fully convolutional network (FCN) was introduced into PAM to extract the molten pool and plasma arc. Figure 14b shows that the molten pool and plasma arc morphologies could be accurately removed. The changes in the area of the molten pool and the plasma arc were quantitatively studied under different current intensities and scanning speeds. Finally, the surface roughness of the sample was measured, and the relationship between the surface roughness and the area of the molten pool and plasma arc was analyzed. The paper provided a guide for process prediction and real-time feedback control.
The above works focused on the feature extraction of molten pool and spatter images based on AI algorithm. Fast manufacturing process will make the high-accuracy capture of process images difficult, thus affect the performance of AI algorithm to process data. It is worth mentioning that high-speed cameras can capture more detailed information, but its monitoring platform built is a relatively expensive and the high-speed cameras require a strict shooting environment. AI algorithms can also be used for in situ and real-time monitoring of part surface profiles. Caggiano et al. introduced the bi-stream DCNN into the in situ monitoring process of L-PBF for quick identification of the defects in the surface image of each layer with high precision [52]. Figure 15a shows the scanning strategy for the in situ monitoring experiment. Figure 15b is a representative feature map of the last convolutional layer with different classification accuracies under different laser energy densities. With the gradual convergence of the training process, image classification accuracy was constantly improved, and the defects became more apparent. The in situ monitoring method could adaptively control the L-PBF process to ensure the quality of parts.
Scime et al. proposed a new convolutional neural network (dynamic segmentation CNN, DSCNN) for semantic segmentation of each image layer [53]. Figure 16 shows the segmentation result on the printer Arcam Q10. Different parts could be effectively segmented and extracted. The DSCNN could be seamlessly transferred among the three AM technologies. Compared with traditional image processing methods, AI can obtain more image features. Thus, more details of the sample surface can be monitored, but much time is needed to debug the AI algorithm.
Scime et al. mainly studied the geometry of manufacturing process objects, such as molten pool and spatters. By contrast, other works from the literature presented studies from a radiation temperature perspective. From the scheme proposed by Scime et al., hot spots occur during metal AM because of thermal conductivity issues, leading to the formation of part defects. Colosimo et al. proposed a spatially weighted PCA (ST-PCA) to describe the spatiotemporal correlation of images acquired with high-speed cameras whose grayscale values are expressed as temperature [54]. An alarm rule based on K-means clustering was proposed to detect hot spot problems. Compared with other defect detection methods, the method proposed by Colosimo et al. could quickly detect temporal and spatial defects in the manufacturing process. Li et al. proposed a DCNN model for feature extraction and process classification to directly analyze the complete information of process images [55]. The model was used to identify process parameters such as laser power or scanning speed according to the real-time thermal images collected during the manufacturing process. Figure 17a shows the monitoring equipment of thermal images, and Fig. 17b shows the thermal images of the manufacturing process monitored in real time. Colosimo et al. and Li et al. did not use expensive high-speed cameras, and the combination of temperature images was obtained. AI algorithm can extract more detailed features, improve the robustness of the whole monitoring system, and reduce the cost of the monitoring platform.
Compared with the traditional image processing algorithm, the AI-based algorithm can better deal with the complex manufacturing process, have the strong anti-interference ability, and process data faster. In terms of development time, image processing technology based on AI has only been introduced into the monitoring of metal AM in recent years. A neural network is the most popular used AI technology. In terms of accuracy and processing time, the AI algorithms used by Mi et al. [45], Fang et al. [47], and Zhang et al. [11] have the best performance on feature extraction such as molten pool and spatters. Yang et al. [49] and Tan et al. [48] achieved in situ and real-time monitoring of metal AM on the basis of AI image processing at the millisecond level for the first time. Although the AI technology can process objects quickly and accurately, there remains less literature introducing AI-based image processing technology into in situ and real-time monitoring of metal AM. For neural networks to obtain better monitoring performance, many images for neural network training from a large number of experiments should be obtained. It is also vital to ensure the image quality, but this requires a high cost.
However, the above so-called real-time monitoring is only rapid off-line processing after image acquisition, rather than real-time data acquisition, real-time transmission to the image processor, and real-time processing of results, which must be addressed in the implementation of in situ and real-time monitoring for metal AM.
3 Future perspectives on image processing
Currently, traditional image processing technologies and image processing technologies based on AI have been introduced into the in situ and real-time monitoring of the metal AM process. Image processing technology can obtain rich characteristic information of vital process variables such as molten pool and spatters in metal AM. But there remain some problems with image processing technology, including monitoring image quality, small samples, image labeling, algorithm generalization, and the specific implementation of real-time feedback control based on image processing and other technologies. These problems hinder the monitoring of image processing technology in metal AM processes. There is a need to focus on this study topic in the future.
3.1 Image quality
For the molten pool and spatter images obtained by in situ and real-time monitoring, the morphologies of molten pool and spatters are indirectly or directly disturbed because of the interference of laser brightness, camera position, and other factors, which may have a destructive impact on the final processing results. Image preprocessing is necessary for cleaning data, but it is currently limited to traditional methods, which do not significantly improve image quality. The super-resolution reconstruction algorithm based on AI can better fit the nonlinear mapping between low- and high-resolution images. The network structures of pre-sampling, post-sampling, stepwise up-sampling, and alternating up–down sampling are used to obtain the high- and low-resolution information of the image. The low-resolution image is gradually expanded to the high-resolution image. Dong et al. proposed a single image super-resolution algorithm based on deep learning, which inputted low- and high-resolution images [56]. Their algorithm depicted high-quality repair performance and can realize online real-time processing of a single image, making it a possible approach to improve the quality of monitoring images during AM process.
3.2 Small samples
Existing deep learning-based image processing requires a large number of samples to train a network model. Small samples will lead to insufficient information diversity, which in turn leads to an overfitting problem of the trained model. The overfitting problem refers to that the image processing effect is good on the training sets, but poor on the test sets. Collecting images of the metal AM process often requires many experiments and is economically expensive, so it is difficult to collect a large number of training samples. Existing methods for expanding datasets mainly use traditional means such as rotational transformations, but this method is ineffective. Effective AI methods mainly include one-shot learning and few-shot learning. The datasets of similar objects are input into the neural network for model pretraining. The pretraining model processes the actual samples according to the features acquired by similar object samples. In other technology areas, AI has been introduced for small-sample problems to develop high-quality images. Kitamura et al. used CNN integration and combined multi-views for ankle fracture detection [57]. The authors achieved the detection performance of large-sample datasets on small-sample datasets in the medical field. For image processing, Gao et al. proposed a small-sample defect-recognition method on the basis of hierarchical feature fusion and verified it on the Northeastern University dataset [58]. Solving the problem of small samples in in situ and real-time monitoring of the metal AM process may be necessary for the future.
3.3 Image labeling
For metal AM processes image segmentation based on deep learning, the production of label images requires manual labeling. The label images here are to assign specific gray values to different areas of molten pool, spatters, and other objects. The result of labeling is directly related to the final segmentation and extraction of objects such as molten pool and spatters. High-quality label images require a long time for manual and careful labeling, and the labeling time may be close to the network model debugging time, which is very long. Semi-supervised deep learning is an effective method to solve the difficulty of image labeling. Labeled and unlabeled data constitute the training dataset, and the unlabeled dataset generally requires category balancing. The category distribution should also be similar to the category distribution of the labeled dataset. Unlabeled data can pretrain neural networks, whereas labeled data can debug neural networks. Semi-supervised deep learning has been applied to medical image segmentation. Li et al. proposed a new semi-supervised skin damage segmentation method for skin damage [59]. Regularization’s performance on pixel-level prediction was enhanced by introducing a transformation consistency scheme in the self-ensembling model. Luo et al. proposed a semi-supervised medical image segmentation framework based on dual-task consistency [60]. They achieved the most advanced results on the left atrial dataset in MR scans, and the pancreas dataset in CT scans. Given the low efficiency of manual labeling of datasets, using AI to label images automatically may be a significant method for obtaining image datasets for metal AM monitoring in the future.
3.4 Generalization of image processing algorithm
Presently, most methods used for in situ real-time process monitoring of metal AM can only be used in a single machine. Existing algorithms are sensitive to device, power, material, environment, and image quality. When the laser power is too high, metal smoke may appear and affect the process image quality [45]. In the process of high-speed camera shooting, it is necessary to illuminate the molten pool to obtain a clear molten pool image. Different intensities of illumination will cause differences in the acquired process image [11, 45, 47,48,49]. The literature has yet to provide an efficient algorithm to handle so many scenes, even with the same printing device such as L-DED. When the same algorithm is transplanted to another machine, it still needs much debugging, even changing the structure of the network model. Possible solutions include inputting more types of datasets to learn more features. The process images captured in different complex environments are used to train neural networks. The loss function will affect the iteration of network parameters. According to the diversity of targets and the complexity of optical background in the image of metal additive manufacturing process, a more suitable loss function will be found to optimize the direction of parameter adjustment. More features can be learned by increasing the number of network layers, that is, deeper features of process images in various manufacturing scenes can be learned by deepening the neural network layers. Improving the generalization of monitoring methods for metal AM may be one of the main tasks in the future.
3.5 Real-time feedback control based on image processing and other technologies
The in situ and real-time monitoring of the AM process lays a solid foundation for realizing closed-loop real-time adjustment and control of laser power, scanning speed, and other process parameters. This effectively improves the stability and repeatability of the manufacturing process and ensures the quality of parts. Real-time feedback control involves rapid coordinated control in multiple fields such as mechanical structures and control systems. At the same time, data must be processed quickly. Image processing technology can obtain the characteristics of part morphology, molten pool, and spatters. Still, to ensure the final part’s quality, image processing technology must be combined with other technologies such as pulse laser shock, which the literature is yet to provide. A primary future task for real-time feedback control may be multifaceted in situ and real-time monitoring, in situ quality evaluation, and ultimately in situ feedback.
Figure 18 shows a technical route for real-time feedback control. First, AI is introduced into the online real-time monitoring of metal AM. Then, the in situ quality evaluation is carried out by deep learning to analyze multisource monitoring results. Finally, according to the evaluation results, the pulsed laser impact control is carried out. This technical route includes two vital scientific questions: (1) the generation and evolution mechanism of macro–micro defects and (2) the influence mechanism of online feedback control on macro–micro-morphology.
The metal AM process can be obtained through a high-speed camera, photodiodes, 3D vision, and laser ultrasound. This image information will be used to train AI for online real-time monitoring. The multisource dataset is subjected to data registration, feature extraction, and data coupling when performing an in situ quality evaluation. The processing results of multisource data are used for deep learning in in situ quality evaluation. The results after deep learning processing are used to control the parameters of pulsed laser shock. The stomatal suppression, stress regulation, and precision control are carried out for the molten pool and the solidification zone by controlling the process parameters of laser energy, impact frequency, scanning speed, and spot spacing. The final quality of metal AM parts can be characterized by surface accuracy, mechanical properties, and high-temperature properties. The real-time feedback control technical route described establishes a complete forming process, monitoring information, and part quality correlation relationship. The problem of repeatability and stability of the metal AM process will be effectively solved by collecting data, analyzing data, and using pulse laser shock control according to the above technical route.
4 Conclusion
This paper reviews the in situ and real-time monitoring of metal AM based on traditional image processing technology and AI image processing technology. The monitoring object, process classification, and image processing carrier are analyzed and compared for the traditional image processing technology. The traditional image processing technology has strict limitations on the environment. Thus, other algorithms may be needed for monitoring in complex environments. AI technology can process data with high speed and high precision. AI technology can also process images under challenging conditions, allowing real-time and accurate metal AM monitoring data processing. For image processing based on AI technology, in situ and real-time monitoring of metal AM is analyzed from three aspects: processing objects, accuracy, and processing time. Their advantages are compared in detail in terms of processing time and accuracy. In the future perspectives, image quality monitoring, small samples, image annotation, algorithm promotion, and real-time feedback control are analyzed in detail. The future development of real-time feedback control is discussed, and the feasible technical route is given by combining image processing technology with other technologies.
Availability of data and material
Not applicable.
Code availability
Not applicable.
References
Lu B, Li D, Tian X (2015) Development trends in additive manufacturing and 3D printing. Engineering-prc 1:85–89
Herzog D, Seyda V, Wycisk E, Emmelmann C (2016) Additive manufacturing of metals. Acta Mater 117:371–392
Wang Y, Zheng P, Peng T, Yang H, Zou J (2020) Smart additive manufacturing: current artificial intelligence-enabled methods and future perspectives. Sci China Technol Sc 63:1600–1611
Gardan J (2016) Additive manufacturing technologies: state of the art and trends. Int J Prod Res 54:3118–3132
Liu L, Ding Q, Zhong Y, Zou J, Wu J, Chiu Y, Li J, Zhang Z, Yu Q, Shen Z (2018) Dislocation network in additive manufactured steel breaks strength–ductility trade-off. Mater Today 21:354–361
Kruth JP, Mercelis P, Van Vaerenbergh J, Froyen L, Rombouts M (2005) Binding mechanisms in selective laser sintering and selective laser melting. Rapid Prototyping J 11:26–36
Rafi HK, Karthik NV, Gong H, Starr TL, Stucker BE (2013) Microstructures and mechanical properties of Ti6Al4V parts fabricated by selective laser melting and electron beam melting. J Mater Eng Perform 22:3872–3883
Zhan Q, Liang Y, Ding J, Williams S (2017) A wire deflection detection method based on image processing in wire + arc additive manufacturing. Int J Adv Manuf Tech 89:755–763
Lia F, Park JZ, Keist JS, Joshi S, Martukanitz RP (2018) Thermal and microstructural analysis of laser-based directed energy deposition for Ti-6Al-4V and Inconel 625 deposits. Mater Sci Eng A 717:1–10
Tapia G, Elwany A (2014) A review on process monitoring and control in metal-based additive manufacturing. J Manuf Sci E-T Asme 136:060801
Zhang Y, Mi J, Li H, Shen S, Yang Y, Song C, Zhou X (2022) In-situ monitoring plasma arc additive manufacturing process with fully convolutional network. Int J Adv Manuf Tech 120:2247–2257
Angelo LD, Stefano PD, Guardiani E (2020) Search for the optimal build direction in additive manufacturing technologies: a review. J Manuf Mater Process 4:71
Tino R, Leary M, Yeo A, Kyriakou E, Kron T, Brandt M (2020) Additive manufacturing in radiation oncology: A review of clinical practice, emerging trends and research opportunities. Int J Extrem Manuf 2:012003
Varela J, Merino J, Pickett C, Abu-Issa A, Arrieta E, Murr L, Wicker R, Ahlfors M, Godfrey D, Medina F (2020) Performance characterization of laser powder bed fusion fabricated inconel 718 treated with experimental hot isostatic processing cycles. J Manuf Mater Process 4:73
Leung CLA, Marussi S, Towrie M, Atwood RC, Withers PJ, Lee PD (2019) The effect of powder oxidation on defect formation in laser additive manufacturing. Acta Mater 166:294–305
Cerniglia D, Montinaro N (2018) Defect detection in additively manufactured components: laser ultrasound and laser thermography comparison. Procedia Struct Integr 8:154–162
Hossain MS, Taheri H (2020) In situ process monitoring for additive manufacturing through acoustic techniques. J Mater Eng Perform 29:6249–6262
Leung CLA, Marussi S, Atwood RC, Towrie M, Withers PJ, Lee PD (2018) In situ X-ray imaging of defect and molten pool dynamics in laser additive manufacturing. Nat Commun 9:1–9
Koester LW, Taheri H, Bigelow TA, Bond LJ, Faierson EJ (2018) In-situ acoustic signature monitoring in additive manufacturing processes. In AIP Conference Proceedings 1949:020006
Donadello S, Motta M, Demir AG, Previtali B (2019) Monitoring of laser metal deposition height by means of coaxial laser triangulation. Opt Lasers Eng 112:136–144
Li J, Duan Q, Hou J, Xie H, Liu S (2020) In-situ monitoring of substrate deformation in directed energy deposition process using the coherent gradient sensing method. Addit Manuf 36:101547
Min Y, Shen S, Li H, Sheng L, Mi J, Zhou J, Mai H, Chen J (2022) Online monitoring of an additive manufacturing environment using a time-of-flight mass spectrometer. Measurement 189:110473
Huang TS, Schreiber WF, Tretiak OJ (1971) Image processing. In Proceedings of the IEEE 59:1586–1609
Sarid O, Huss E (2011) Image formation and image transformation. Art Psychother 38:252–255
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 770–778.
Cheng HD, Jiang XH, Sun Y, Wang J (2001) Color image segmentation: advances and prospects. Pattern Recogn 34:2259–2281
Wang DCC, Vagnucci AH, Li CC (1983) Digital image enhancement: a survey. Comput Vision, Graph Image Process 24:363–381
Everton SK, Hirsch M, Stavroulakis PI, Leach RK, Clare AT (2016) Review of in-situ process monitoring and in-situ metrology for metal additive manufacturing. Mater Des 95:431–445
Cunha FG, Santos TG, Xavier J (2021) In situ monitoring of additive manufacturing using digital image correlation: a review. Mater 14:1511
Hofman JT, Pathiraj B, Van Dijk V, De Lange DF, Meijer J (2012) A camera based feedback control strategy for the laser cladding process. J Mater Process Technol 212:2455–2462
Moralejo S, Penaranda X, Nieto S, Barrios A, Arrizubieta I, Tabernero I, Figueras J (2017) A feedforward controller for tuning laser cladding melt pool geometry in real time. Int J Adv Manuf Tech 89:821–831
Song L, Wang F, Li S, Han X (2017) Phase congruency melt pool edge extraction for laser additive manufacturing. J Mater Process Technol 250:261–269
Yang Q, Yuan Z, Zhi X, Yan Z, Tian H, Chen X (2020) Real-time width control of molten pool in laser engineered net shaping based on dual-color image. Opt Laser Technol 123:105925
Seltzer D, Schiano J L, Nassar A R, Reutzel E W (2016) Illumination and image processing for real-time control of directed energy deposition additive manufacturing. In Proceedings of the 26th Annual International Solid Freeform Fabrication Symposium – An Additive Manufacturing Conference 1479–1486.
Craeghs T, Clijsters S, Yasa E, Kruth J P (2011) Online quality control of selective laser melting. In Proceedings of the 20th Solid Freeform Fabrication (SFF) Symposium 212–226.
Clijsters S, Craeghs T, Buls S, Kempen K, Kruth JP (2014) In situ quality control of the selective laser melting process using a high-speed, real-time melt pool monitoring system. Int J Adv Manuf Tech 75:1089–1101
Zhang Y, Fuh JYH, Ye D, Hong GS (2019) In-situ monitoring of laser-based PBF via off-axis vision and image processing approaches. Addit Manuf 25:263–274
Asselin M, Toyserkani E, Iravani-Tabrizipour M, Khajepour A (2005) Development of trinocular CCD-based optical detector for real-time monitoring of laser cladding. In IEEE International Conference Mechatronics and Automation 3:1190–1196
Biegler M, Graf B, Rethmeier M (2018) In-situ distortions in LMD additive manufacturing walls can be measured with digital image correlation and predicted using numerical simulations. Addit Manuf 20:101–110
Yao B, Imani F, Sakpal AS, Reutzel EW, Yang H (2018) Multifractal analysis of image profiles for the characterization and detection of defects in additive manufacturing. J Manuf Sci E-T Asme 140:031014
Barua S, Liou F, Newkirk J, Sparks T (2014) Vision-based defect detection in laser metal deposition process. Rapid Prototyp J 20:77–86
Khanzadeh M, Tian W, Yadollahi A, Doude HR, Tschopp MA, Bian L (2018) Dual process monitoring of metal-based additive manufacturing using tensor decomposition of thermal image streams. Addit Manuf 23:443–456
Khanzadeh M, Chowdhury S, Marufuzzaman M, Tschopp MA, Bian L (2018) Porosity prediction: Supervised-learning of thermal history for direct laser deposition. J Manuf Syst 47:69–82
Zhang B, Liu S, Shin YC (2019) In-process monitoring of porosity during laser additive manufacturing process. Addit Manuf 28:497–505
Mi J, Zhang Y, Li H, Shen S, Yang Y, Song C, Zhou X, Duan Y, Lu J, Mai H (2021) In-situ monitoring laser based directed energy deposition process with deep convolutional neural network. J Intell Manuf. https://doi.org/10.1007/s10845-021-01820-0
Yang Z, Lu Y, Yeung H, Krishnamurty S (2019) Investigation of deep learning for real-time melt pool classification in additive manufacturing. In 2019 IEEE 15th International Conference on Automation Science and Engineering (case) 640–647.
Fang Q, Tan Z, Li H, Song C, Zhou X, Yang Y, Shen S (2021) In-situ capture of melt pool signature in selective laser melting using U-Net-based convolutional neural network. J Manuf Process 648:347–355
Tan Z, Fang Q, Li H, Liu S, Zhu W, Yang D (2020) Neural network based image segmentation for spatter extraction during laser-based powder bed fusion processing. Opt Laser Technol 130:106347
Yang D, Li H, Liu S, Song C, Yang Y, Shen S, Lu J, Liu Z, Zhu Y (2020) In situ capture of spatter signature of SLM process using maximum entropy double threshold image processing method based on genetic algorithm. Opt Laser Technol 131:106371
Wang Y, Zhang C, Lu J, Bai L, Zhao Z, Han J (2020) Weld reinforcement analysis based on long-term prediction of molten pool image in additive manufacturing. IEEE Access 8:69908–69918
Lu J, He H, Shi Y, Bai L, Zhao Z, Han J (2021) Quantitative prediction for weld reinforcement in arc welding additive manufacturing based on molten pool image and deep residual network. Addit Manuf 41:101980
Caggiano A, Zhang J, Alfieri V, Caiazzo F, Gao R, Teti R (2019) Machine learning-based image processing for on-line defect recognition in additive manufacturing. Cirp Ann-Manuf Techn 68:451–454
Scime L, Siddel D, Baird S, Paquit V (2020) Layer-wise anomaly detection and classification for powder bed additive manufacturing processes: a machine-agnostic algorithm for real-time pixel-wise semantic segmentation. Addit Manuf 36:101453
Colosimo BM, Grasso M (2018) Spatially weighted PCA for monitoring video image data with application to additive manufacturing. J Qual Technol 50:391–417
Li X, Siahpour S, Lee J, Wang Y, Shi J (2020) Deep learning-based intelligent process monitoring of directed energy deposition in additive manufacturing with thermal images. Procedia Manuf 48:643–649
Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE T Pattern Anal 38:295–307
Kitamura G, Chung CY, Moore BE (2019) Ankle fracture detection utilizing a convolutional neural network ensemble implemented with a small sample, de novo training, and multiview incorporation. J Digit Imaging 32:672–677
Gao Y, Gao L, Li X (2019) A hierarchical feature fusion-based method for defect recognition with a small sample. In IEEE International Conference on Industrial Engineering and Engineering Management 1048–1052
Li X, Yu L, Chen H, Fu C W, Heng P A (2018) Semi-supervised skin lesion segmentation via transformation consistent self-ensembling model. arXiv preprint arXiv:1412.7062
Luo X, Chen J, Song T, Wang G (2020) Semi-supervised medical image segmentation through dual-task consistency. arXiv preprint arXiv:2009.04448
Funding
This work was supported by the Key Research and Development Program of Sichuan Province, China (2020YFSY0054).
Author information
Authors and Affiliations
Contributions
Yikai Zhang: conceptualization, methodology, formal analysis, data curation, visualization. Shengnan Shen: writing—review and editing, supervision. Hui Li: writing—review and editing. Yaowu Hu: writing—review and editing.
Corresponding authors
Ethics declarations
Ethics approval
The authors confirm that the submitted work is original and has not been published elsewhere.
Consent to participate
All authors have agreed to participate in this research.
Consent for publication
All authors have informed and agreed to publish this research.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, Y., Shen, S., Li, H. et al. Review of in situ and real-time monitoring of metal additive manufacturing based on image processing. Int J Adv Manuf Technol 123, 1–20 (2022). https://doi.org/10.1007/s00170-022-10178-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00170-022-10178-3