1 Introduction

Metal additive manufacturing (AM) technology has developed into a fascinating digital manufacturing technology, attracting tremendous attention across the globe [1,2,3,4]. Metal AM technology has several outstanding advantages: (1) It can manufacture complex parts because there may not be the limitation of traditional processing; (2) it does not take up space and is portable so that it can be used in disaster areas and battlefields; (3) it produces new material properties, such as the introduction of dislocation networks while significantly enhancing the strength and ductility of the parts [5]; and (4) it requires short production cycle. The metal AM can be divided into several main types according to different manufacturing technologies. Traditional metal AM technologies include laser powder bed fusion (L-PBF), electron beam melting (EBM), wire arc AM (WAAM), and laser-directed energy deposition (L-DED). Metal AM can also be divided into powder bed fusion (PBF) and deposition energy deposition (DED) according to the molten types of materials. SLS, L-PBF, and EBM are the import components of PBF. SLS and L-PBF use a laser as an energy source to melt selected areas of powder, layer by layer, through computer-aided design data. In L-PBF, the material is completely melted rather than sintered as in SLS [6]. EBM, similar in principle to SLS and L-PBF, uses an electron beam as a heat source to create parts by selectively melting each layer of powder [7].

WAAM is a process that melts metal wire by using an electric arc [8]. L-DED is a process that blows powder into the molten pool to make parts [9]. WAAM and L-DED are import components of DED. There are also some other metal AM technologies. Laser cladding (LC) is a variant of DED. It uses the same principles as the DED but adds additional metal layers to existing parts [10]. Plasma arc additive manufacturing (PAM) is a kind of DED that uses plasma arc as a heat source but wire or powder as raw material [11]. Table 1 summarizes the characteristics of different metal AM processes by listing critical detrimental properties and critical process variables.

Table 1 Characteristics of major metal AM techniques
Full size table

The significant detrimental properties associated with metal AM include pores, cracks, spheroidization, and residual stress. These defects seriously affect the internal structure and mechanical properties of parts, and ultimately affect the quality of parts [12,13,14,15]. During the manufacturing process, vital process variables such as molten pool characteristics, spatter characteristics, surface morphology, and radiation temperature are directly related to the above defects. Monitoring the dynamic behaviors of the above variables is essential to ensure parts’ quality. By adding monitoring equipment beside the printer, the process of metal AM can be monitored. There are many monitoring methods, including laser ultrasonic monitoring and optical imaging monitoring [16,17,18]. Koester et al. proved that metal AM processes could be detected using a passive acoustic monitoring [19]. Donadello et al. proposed a system to measure deposition height during the L-DED process, extended to complex 3D geometries and different materials [20]. Li et al. proposed a coherent gradient sensing method to monitor stress and strain by first measuring the real-time full-field deformation of a thick substrate during a disc-shaped specimen deposition [21]. Then, they analyzed the deformation distribution characteristics at different positions. Min et al. developed a system on the basis of a miniature electron impact ion source time-of-flight mass spectrometer. Their proposed system could monitor the changes in the atmosphere above the molten pool area in real time [22].

As the visual basis of human perception of the world, an image is a means of obtaining, expressing, and transmitting information. Image processing transforms two-dimensional data for image information retrieval [23]. Image processing technologies include transformation, recognition, segmentation, and enhancement. Image transformation captures information in different spaces using techniques such as Fourier transform [24]. Image recognition can identify different objects by analyzing and understanding images [25]. Image segmentation divides an image into distinct, uniform regions but variable adjacent regions [26]. Image enhancement filters specific noise to make the edge part clearer and highlights the needed features [27]. Table 2 summarizes the possible applications of image processing techniques in monitoring metal AM processes. As shown in Table 2, image processing technology has a great application prospect in metal AM process monitoring. Reviews from the literature have mentioned image processing applications for metal AM, but no comprehensive report focuses on its processing methods. Everton et al. conducted an overview of in situ process monitoring and metrology of metal AM, mentioning the applications of in situ image process monitoring [28]. Digital image correlation technology carries out correlation operation on two digital images before and after the part deformation to obtain the deformation information of the region of interest. Cunha et al. reviewed digital image correlation technology for the in situ metal AM process monitoring [29]. It mainly included defect characterization, residual stress evaluation, geometric distortion, and numerical simulation verification based on field measurement, monitoring, and part feature description.

Table 2 Possible image processing applications to monitor metal AM process
Full size table

Real-time acquisition of monitoring results is used to judge the quality of parts in real time, which is helpful to solve the problems of repetition and stability of the metal AM process. Presently, in situ and real-time monitoring methods are essential for the manufacturing process of products. Thus, this paper reviews the progress of in situ and real-time monitoring of metal AM on the basis of image processing. The difficulties and solutions of image processing techniques applied to in situ and real-time monitoring of metal AM are analyzed and discussed. A detailed and feasible technical route for metal AM real-time feedback control is proposed by combining image processing with other technologies. The remainder of this paper is organized as follows: Sect. 2 introduces the application of traditional and artificial intelligence (AI) image processing in in situ and real-time monitoring of metal AM processes. Section 3 discusses the challenges encountered and possible image processing and real-time feedback control solutions. Finally, Sect. 4 concludes the paper.

2 Methods

Eighty-three percent of the information humans acquire comes from vision. The information obtained in monitoring metal AM processes based on image processing is more abundant than monitoring technologies like laser ultrasound. The dynamic characteristics of the molten pool and spatter, such as the evolution of size, quantity, and temperature, can be obtained from monitored images; the topography of each layer of the parts can also be acquired at higher image resolutions as well as cracks on the surface of the parts. With the development of metal AM-related technologies, images will be able to obtain more information. To solve the repeatability and stability problem of the process, it must be possible to monitor the manufacturing process in real time. In situ and real-time monitoring is the mainstream monitoring method. At present, some work has been done for in situ and real-time monitoring of metal AM processes by using image processing techniques. According to the image processing methods, existing works from the literature are mainly divided into traditional image processing and AI image processing. Table 3 summarizes the current representative work.

Table 3 Typical in situ and real-time monitoring of metal AM based on image processing
Full size table

2.1 In situ and real-time monitoring of metal AM process based on traditional image processing

In situ and real-time process monitoring based on traditional image processing has been well studied in the literature. The traditional image processing algorithms mainly include image transformation, threshold segmentation, edge extraction, and projection transformation.

These algorithms can be used to monitor the size of the molten pool. Hofman et al. adopted filter and threshold processing on the basis of a CMOS camera to obtain the width of the molten pool in real time during the LC process [30]. The width of the molten pool was kept within a specific range via feedback-adjusted laser power, improving the quality of the parts. OpenCV is the huge open-source library for the computer vision, machine learning, and image processing. And it can carry out rich image processing operations. For L-DED processes, Moralejo et al. used the image processing algorithm in OpenCV and optimized the processing time to 10 ms to detect the molten pool’ size in real time [31]. A unique feedforward controller was designed for real-time control and in situ adjustment of molten pool morphology. Similarly, Song et al. proposed a phase congruency with a log-Gabor filter to extract molten pool edge [32]. Compared with the traditional threshold methods, the methods proposed by Song et al. method could effectively eliminate the interference caused by plasma, molten pool disturbance, and other factors. It could be used for real-time monitoring of molten pool geometry. In 2020, Yang et al. proposed a coaxial CCD imaging system with a dual-wavelength narrow-band filter to obtain the molten pool image [33]. Then, the minimum boundary rectangle method was used to extract the molten pool width; the molten pool’s width was controlled by PID feedback. This method could effectively improve the quality of samples.

The above methods all used CPU for image data processing. The field-programmable gate array (FPGA) chip is used in in situ and real-time process monitoring to reduce image transmission time and improve image processing speed. Seltzer et al. designed corresponding optical equipment and image processing devices to control molten pool’ geometry in real time [34]. FPGA chip was adopted to realize the image processing algorithm for estimating molten pool width and build height in the image processing. Some work has also been done during L-PBF process. Craeghs et al. proposed a real-time visual monitoring system for molten pool shape during the L-PBF process [35]. They adopted FPGA chip technology to realize fast and real-time processing of molten pool images. Clijsters et al. proposed a monitoring system comprising software and hardware for L-PBF to monitor the molten pool in real time at high speed [36]. Figure 1a shows the experimental optical platform of the real-time monitoring system. The FPGA chip was adopted to achieve better data analysis speed and molten pool image processing technology. Figure 1b shows the monitoring curves of the molten pool area at different positions during the manufacturing process. In the above work, different algorithms and hardware processing systems were used to monitor the molten pool morphology in real time, but the monitoring accuracy may be limited in complex environments due to the robustness of the algorithm.

The above work almost exclusively monitored a single object. Zhang et al. proposed a monitoring system for multiple objects and an off-axis visual sensing system on the basis of high-speed cameras [37]. The characteristics of the molten pool intensity, plume area, orientation, spatter number, spatter area, and spatter velocity were obtained through the Kalman filter and image segmentation algorithm, providing a good foundation for real-time monitoring and feedback control of the L-PBF manufacturing process. It can be seen from the above work that L-DED and L-PBF are the main manufacturing processes used to obtain the geometrical morphology of molten pool. This may be due to the mature development of L-DED and L-PBF technologies, making it more convenient to realize in situ real-time monitoring.

Traditional image processing algorithms can also monitor the geometries of parts and feedstock. Asselin et al. implemented trinocular optical CCD-based detector, threshold segmentation, and projection transformation to realize in situ and real-time image processing to obtain the clad height, clad width, and solidification rate [38]. Unfortunately, rapid heating and cladding may cause cracks and other defects in parts. Biegler et al. first achieved in situ monitoring of the printing part deformation rather than the substrate during the L-DED process by using digital image correlation with the optical filter [39]. Figure 2a shows the in situ monitoring experimental equipment of deformation. Figure 2b shows the monitoring curve of workpiece wall deformation during the manufacturing process. Yao et al. proposed a new multifractal methodology of image contour by monitoring part surface profile. The quality of parts could be controlled during the L-PBF process [40]. During the WAAM process, the wire feed position dramatically influences the final quality of the parts. Zhan et al. proposed a vision-based detection of wire feed position deviation in the WAAM process [8]. The edge of the wire was obtained using the adaptive threshold and Hough transform, and the deflection was measured by Radon transform. The above work shows that the traditional image processing method can quickly obtain parts and feedstock geometry. The above work only monitored the surface geometry information of solids during the manufacturing process, and did not involve temperature information and internal defect information, which was not enough to realize monitoring of the part quality during the manufacturing process.

Radiance temperature characteristic can also be used for in situ and real-time monitoring of metal AM. Barua et al. focused on vision-based defect detection during the L-DED process [41]. The heat conduction will change when defects occur during the printing process so that the defect can be detected. There is a linear mapping relationship between the RGB value of the image and the workpiece temperature. The temperature distribution of the workpiece can be obtained by data mapping of printing process images taken by a camera. Figure 3a shows the in situ monitoring experimental equipment. Because of the uneven deposition, it could be found that the radiation temperature measurement of the deposit changed significantly. Figure 3b shows the temperature distribution near the defect. Khanzadeh et al. proposed a dual control chart system that used T2 and Q control charts to detect the extracted low dimensional features and residuals, respectively. Principal component analysis (PCA) reduces the dimension of data by mapping multiple variables to fewer variables with almost the same amount of information. The authors adopted multilinear PCA to extract the low dimensional features and residuals of the process thermal image before monitoring the abnormality of the printing process [42]. Figure 4a shows the experimental equipment for in situ monitoring of the manufacturing process. Figure 4b shows the judgment result of adopting dual control chart. The core content of the above work was that the temperature was abnormal indirectly indicating that defects might occur. In addition, the monitoring of metal additive manufacturing based on radiation characteristics might be susceptible to environmental factors leading to problems in collecting data or judging defects; thus, the controllability of the manufacturing process and the processing environment is crucial.

According to the highlighted works from the literature, in terms of monitoring objects, using traditional image processing for monitoring molten pool geometry is the focus of scholars’ investigation, followed by the monitoring of parts geometry and temperature anomalies. The traditional image processing algorithms can realize fast feature extraction from images captured in simple manufacturing environment, but may be not feasible for low-quality images captured in complex manufacturing environment. Thus, it is necessary to introduce an AI algorithm to monitor the manufacturing process.

2.2 In situ and real-time monitoring of AM process based on AI image processing

AI has been widely used in robots, control systems, and simulation systems. In recent years, it has also been gradually applied to metal AM. AI, based on image processing, accounts for a large proportion. Its introduction makes it possible to process the image of the metal AM process in real time and adjust the process parameters. Studies from the literature on in situ and real-time process monitoring of metal AM based on AI image processing from four aspects: processing objects, algorithm type, accuracy, and processing time. Table 4 shows the comparative analysis of these aspects.

Table 4 Comprehensive comparison of different methods
Full size table

Image processing based on AI can be used to monitor the geometries of molten pool and spatters. Some work has been done during L-DED process. Supervised learning uses a group of known samples to adjust the parameters of AI algorithm to achieve the required performance. Khanzadeh et al. introduced supervised learning into porosity prediction. They proposed an AI framework to establish the relationship between molten morphological characteristics and microstructure abnormalities by real-time monitoring of molten pool abnormalities [43]. Unlike the traditional method, their proposed model detects the porosity after manufacturing. A new data processing method was proposed to reduce the dimension of the thermal image of the molten pool. Then, the porosity could be predicted in real time with high accuracy by using the reduced dimensional characteristics of the molten pool boundary. Five supervised learning methods to classify standard molten pool and pores were analyzed. Figure 5a shows real-time monitoring experimental equipment. Figure 5b shows the results of K-nearest neighbor (KNN) which had the highest rate of accurate classification under the first and second principal components.

Zhang et al. built an in situ porosity monitoring system during the L-DED process [44]. As shown in Fig. 6a, the system used a high-speed digital camera to collect the dynamic image of the molten pool in situ. A set of image processing tools was developed to mark the molten pool data by cross-sectioning. X-ray tomography methods were used to obtain the porosity of the sample. Convolutional neural network (CNN) can extract the features of interested objects from images through a large number of convolution operations. A compact CNN architecture was designed to establish the relationship between molten pool image and sample porosity by learning the characteristics of molten pool image. Figure 6b shows the prediction results of local volume porosity of the two samples by using CNN. The above experimental results showed that the method of predicting sample porosity through real-time monitoring of molten pool morphology features based on AI algorithm has a promising potential. However, in order to achieve an accurate prediction of sample porosity, a large number of experiments are needed to obtain enough sample data.

In 2021, Mi et al. proposed an in situ monitoring system for the L-DED process with a high-speed camera, image processing system, and laser light source [45]. As shown in Fig. 7a, the system could quickly collect process images, and the information that could be extracted includes the area of the molten pool and the number of spatters. The deep CNN (DCNN) was introduced to monitor the L-DED process; by adopting the lightweight network structure and improving the loss function, the molten pool and spatters could be segmented with high precision and high speed to achieve the purpose of real-time processing of the process images. Lightweight network refers to the modification of network structure to reduce network parameters and keep the data processing effect unchanged. And loss function refers to the calculation of the difference between the predicted value and actual value, which is used for the iterative modification of network parameters. Figure 7b shows the extraction results of the molten pool and spatters. The position positioning of spatters was realized for the first time, and the change of spatter position was analyzed. However, due to the wide spatial distribution of spatters during L-DED process, the geometric characteristics of spatters captured by high-speed cameras with different focal lengths will change in some degree, which may affect final performance of AI algorithm.

For L-PBF processes, Yang et al. focused on the real-time monitoring and classification of molten pool in the L-PBF process [46]. Using CNN, the size of molten pool was classified into four categories: no molten pool observed, smaller molten pool, normal molten pool, and larger molten pool with high speed and high precision. Figure 8a shows the monitoring equipment, and Fig. 8b shows the real-time classification of molten pool size on the energy density (expressed by the ratio of power and scanning speed). The monitoring platform of their work was complex, but the processing results were more accurate.

To understand the dynamics of the L-PBF process, Fang et al. [47], Tan et al. [48], and Yang et al. [49] established in situ and real-time monitoring systems for the L-PBF process as shown in Figs. 9a, 10a, and 11a. A high-speed camera was used to obtain the image of the L-PBF process, and an AI algorithm processed the image to extract the information of molten pool and spatters. Fang et al. focused on capturing the characteristics of the molten pool in real time [47]. The U-net based on a CNN was used to extract the molten pool. The neural network introduced a lightweight network to reduce the calculation time and improved the loss function to improve the accuracy of the molten pool. As shown in Fig. 9b, the molten pool could be extracted accurately. Tan et al. focused on capturing spatter features in real time [48]. The captured L-PBF process image was segmented into blocks. The obtained block image was segmented by a two-layer network composed of a convolutional neural network and a threshold neural network (TNN) to obtain the spatter morphology information. The final spatters segmented image was obtained by combining the block image. This method had the advantages of simple labeling of training data and high extraction accuracy. Figure 10b shows the spatter extraction accuracy. Fang et al. [47] and Tan et al. [48] provided powerful help for predicting the formation of defects in the L-PBF process.

Yang et al. also focused on real-time capture of spatter features [49]. Different from the neural network methods of Fang et al. [47] and Tan et al. [48], Yang et al. used the maximum-entropy double-threshold image processing algorithm based on genetic algorithm (MEDTIA-GA) to extract spatter morphological information, effectively avoiding noise sensitivity. There were three types of error conditions: noise sensitivity, spatter conglutination, and spatter omission. As shown in Fig. 11b, their proposed method could extract spatter information at high speed compared with three traditional image segmentation algorithms. Finally, the relationship between spatters area, quantity, and laser energy density was analyzed in detail. The phenomenon that a metallic jet connected with the molten pool could be generated under high energy density was obtained. In the work of Fang et al. [47], Tan et al. [48], and Yang et al. [49], there was a smear phenomenon in the spatter morphology, which may need to be eliminated by high-performance high-speed cameras and image processing algorithms.

During WAAM processes, Wang et al. built the molten pool monitoring system, as shown in Fig. 12a [50]. The improved prediction network (PredNet) predicted the future molten pool image. The obtained prediction image was input into the proposed regression network (SERes) to analyze the weld reinforcement in advance. Verified by experiments, the prediction network could predict the molten pool image after 140 ms for a long time. Figure 12b shows the actual molten pool image and predicted molten pool image under different time steps. It was observed that the prediction results were good, and the method could analyze the quality and accuracy of welding shape, which was of great significance for AM process real-time feedback control.

In 2021, Lu et al. designed monitoring equipment for the molten pool to predict deposition reinforcement in AM, as shown in Fig. 13a [51]. The preprocessed molten pool image was input into the residual-based network designed, and the weld reinforcement was obtained quickly. The residual network structure can effectively reduce the gradient vanishing problem when the network is deepened. A multisystem suitable measurement method was proposed to unify the molten pool image, weld position, and corresponding weld reinforcement. The molten pool vision sensor was used to collect the molten pool image in real time, and the positioning system and 3D system were used to detect the corresponding reinforcement of each molten pool. Figure 13b compares distinct layers’ predicted results and errors. Nevertheless, their proposed network could predict the weld reinforcement in real time only when the deposited layer is well-formed.

Zhang et al. built an in situ online monitoring system to monitor the PAM process, as shown in Fig. 14a [11]. The fully convolutional network (FCN) was introduced into PAM to extract the molten pool and plasma arc. Figure 14b shows that the molten pool and plasma arc morphologies could be accurately removed. The changes in the area of the molten pool and the plasma arc were quantitatively studied under different current intensities and scanning speeds. Finally, the surface roughness of the sample was measured, and the relationship between the surface roughness and the area of the molten pool and plasma arc was analyzed. The paper provided a guide for process prediction and real-time feedback control.

The above works focused on the feature extraction of molten pool and spatter images based on AI algorithm. Fast manufacturing process will make the high-accuracy capture of process images difficult, thus affect the performance of AI algorithm to process data. It is worth mentioning that high-speed cameras can capture more detailed information, but its monitoring platform built is a relatively expensive and the high-speed cameras require a strict shooting environment. AI algorithms can also be used for in situ and real-time monitoring of part surface profiles. Caggiano et al. introduced the bi-stream DCNN into the in situ monitoring process of L-PBF for quick identification of the defects in the surface image of each layer with high precision [52]. Figure 15a shows the scanning strategy for the in situ monitoring experiment. Figure 15b is a representative feature map of the last convolutional layer with different classification accuracies under different laser energy densities. With the gradual convergence of the training process, image classification accuracy was constantly improved, and the defects became more apparent. The in situ monitoring method could adaptively control the L-PBF process to ensure the quality of parts.

Scime et al. proposed a new convolutional neural network (dynamic segmentation CNN, DSCNN) for semantic segmentation of each image layer [53]. Figure 16 shows the segmentation result on the printer Arcam Q10. Different parts could be effectively segmented and extracted. The DSCNN could be seamlessly transferred among the three AM technologies. Compared with traditional image processing methods, AI can obtain more image features. Thus, more details of the sample surface can be monitored, but much time is needed to debug the AI algorithm.

Scime et al. mainly studied the geometry of manufacturing process objects, such as molten pool and spatters. By contrast, other works from the literature presented studies from a radiation temperature perspective. From the scheme proposed by Scime et al., hot spots occur during metal AM because of thermal conductivity issues, leading to the formation of part defects. Colosimo et al. proposed a spatially weighted PCA (ST-PCA) to describe the spatiotemporal correlation of images acquired with high-speed cameras whose grayscale values are expressed as temperature [54]. An alarm rule based on K-means clustering was proposed to detect hot spot problems. Compared with other defect detection methods, the method proposed by Colosimo et al. could quickly detect temporal and spatial defects in the manufacturing process. Li et al. proposed a DCNN model for feature extraction and process classification to directly analyze the complete information of process images [55]. The model was used to identify process parameters such as laser power or scanning speed according to the real-time thermal images collected during the manufacturing process. Figure 17a shows the monitoring equipment of thermal images, and Fig. 17b shows the thermal images of the manufacturing process monitored in real time. Colosimo et al. and Li et al. did not use expensive high-speed cameras, and the combination of temperature images was obtained. AI algorithm can extract more detailed features, improve the robustness of the whole monitoring system, and reduce the cost of the monitoring platform.

Compared with the traditional image processing algorithm, the AI-based algorithm can better deal with the complex manufacturing process, have the strong anti-interference ability, and process data faster. In terms of development time, image processing technology based on AI has only been introduced into the monitoring of metal AM in recent years. A neural network is the most popular used AI technology. In terms of accuracy and processing time, the AI algorithms used by Mi et al. [45], Fang et al. [47], and Zhang et al. [11] have the best performance on feature extraction such as molten pool and spatters. Yang et al. [49] and Tan et al. [48] achieved in situ and real-time monitoring of metal AM on the basis of AI image processing at the millisecond level for the first time. Although the AI technology can process objects quickly and accurately, there remains less literature introducing AI-based image processing technology into in situ and real-time monitoring of metal AM. For neural networks to obtain better monitoring performance, many images for neural network training from a large number of experiments should be obtained. It is also vital to ensure the image quality, but this requires a high cost.

However, the above so-called real-time monitoring is only rapid off-line processing after image acquisition, rather than real-time data acquisition, real-time transmission to the image processor, and real-time processing of results, which must be addressed in the implementation of in situ and real-time monitoring for metal AM.

3 Future perspectives on image processing

Currently, traditional image processing technologies and image processing technologies based on AI have been introduced into the in situ and real-time monitoring of the metal AM process. Image processing technology can obtain rich characteristic information of vital process variables such as molten pool and spatters in metal AM. But there remain some problems with image processing technology, including monitoring image quality, small samples, image labeling, algorithm generalization, and the specific implementation of real-time feedback control based on image processing and other technologies. These problems hinder the monitoring of image processing technology in metal AM processes. There is a need to focus on this study topic in the future.

3.1 Image quality

For the molten pool and spatter images obtained by in situ and real-time monitoring, the morphologies of molten pool and spatters are indirectly or directly disturbed because of the interference of laser brightness, camera position, and other factors, which may have a destructive impact on the final processing results. Image preprocessing is necessary for cleaning data, but it is currently limited to traditional methods, which do not significantly improve image quality. The super-resolution reconstruction algorithm based on AI can better fit the nonlinear mapping between low- and high-resolution images. The network structures of pre-sampling, post-sampling, stepwise up-sampling, and alternating up–down sampling are used to obtain the high- and low-resolution information of the image. The low-resolution image is gradually expanded to the high-resolution image. Dong et al. proposed a single image super-resolution algorithm based on deep learning, which inputted low- and high-resolution images [56]. Their algorithm depicted high-quality repair performance and can realize online real-time processing of a single image, making it a possible approach to improve the quality of monitoring images during AM process.

3.2 Small samples

Existing deep learning-based image processing requires a large number of samples to train a network model. Small samples will lead to insufficient information diversity, which in turn leads to an overfitting problem of the trained model. The overfitting problem refers to that the image processing effect is good on the training sets, but poor on the test sets. Collecting images of the metal AM process often requires many experiments and is economically expensive, so it is difficult to collect a large number of training samples. Existing methods for expanding datasets mainly use traditional means such as rotational transformations, but this method is ineffective. Effective AI methods mainly include one-shot learning and few-shot learning. The datasets of similar objects are input into the neural network for model pretraining. The pretraining model processes the actual samples according to the features acquired by similar object samples. In other technology areas, AI has been introduced for small-sample problems to develop high-quality images. Kitamura et al. used CNN integration and combined multi-views for ankle fracture detection [57]. The authors achieved the detection performance of large-sample datasets on small-sample datasets in the medical field. For image processing, Gao et al. proposed a small-sample defect-recognition method on the basis of hierarchical feature fusion and verified it on the Northeastern University dataset [58]. Solving the problem of small samples in in situ and real-time monitoring of the metal AM process may be necessary for the future.

3.3 Image labeling

For metal AM processes image segmentation based on deep learning, the production of label images requires manual labeling. The label images here are to assign specific gray values to different areas of molten pool, spatters, and other objects. The result of labeling is directly related to the final segmentation and extraction of objects such as molten pool and spatters. High-quality label images require a long time for manual and careful labeling, and the labeling time may be close to the network model debugging time, which is very long. Semi-supervised deep learning is an effective method to solve the difficulty of image labeling. Labeled and unlabeled data constitute the training dataset, and the unlabeled dataset generally requires category balancing. The category distribution should also be similar to the category distribution of the labeled dataset. Unlabeled data can pretrain neural networks, whereas labeled data can debug neural networks. Semi-supervised deep learning has been applied to medical image segmentation. Li et al. proposed a new semi-supervised skin damage segmentation method for skin damage [59]. Regularization’s performance on pixel-level prediction was enhanced by introducing a transformation consistency scheme in the self-ensembling model. Luo et al. proposed a semi-supervised medical image segmentation framework based on dual-task consistency [60]. They achieved the most advanced results on the left atrial dataset in MR scans, and the pancreas dataset in CT scans. Given the low efficiency of manual labeling of datasets, using AI to label images automatically may be a significant method for obtaining image datasets for metal AM monitoring in the future.

3.4 Generalization of image processing algorithm

Presently, most methods used for in situ real-time process monitoring of metal AM can only be used in a single machine. Existing algorithms are sensitive to device, power, material, environment, and image quality. When the laser power is too high, metal smoke may appear and affect the process image quality [45]. In the process of high-speed camera shooting, it is necessary to illuminate the molten pool to obtain a clear molten pool image. Different intensities of illumination will cause differences in the acquired process image [11, 45, 47,48,49]. The literature has yet to provide an efficient algorithm to handle so many scenes, even with the same printing device such as L-DED. When the same algorithm is transplanted to another machine, it still needs much debugging, even changing the structure of the network model. Possible solutions include inputting more types of datasets to learn more features. The process images captured in different complex environments are used to train neural networks. The loss function will affect the iteration of network parameters. According to the diversity of targets and the complexity of optical background in the image of metal additive manufacturing process, a more suitable loss function will be found to optimize the direction of parameter adjustment. More features can be learned by increasing the number of network layers, that is, deeper features of process images in various manufacturing scenes can be learned by deepening the neural network layers. Improving the generalization of monitoring methods for metal AM may be one of the main tasks in the future.

3.5 Real-time feedback control based on image processing and other technologies

The in situ and real-time monitoring of the AM process lays a solid foundation for realizing closed-loop real-time adjustment and control of laser power, scanning speed, and other process parameters. This effectively improves the stability and repeatability of the manufacturing process and ensures the quality of parts. Real-time feedback control involves rapid coordinated control in multiple fields such as mechanical structures and control systems. At the same time, data must be processed quickly. Image processing technology can obtain the characteristics of part morphology, molten pool, and spatters. Still, to ensure the final part’s quality, image processing technology must be combined with other technologies such as pulse laser shock, which the literature is yet to provide. A primary future task for real-time feedback control may be multifaceted in situ and real-time monitoring, in situ quality evaluation, and ultimately in situ feedback.

Figure 18 shows a technical route for real-time feedback control. First, AI is introduced into the online real-time monitoring of metal AM. Then, the in situ quality evaluation is carried out by deep learning to analyze multisource monitoring results. Finally, according to the evaluation results, the pulsed laser impact control is carried out. This technical route includes two vital scientific questions: (1) the generation and evolution mechanism of macro–micro defects and (2) the influence mechanism of online feedback control on macro–micro-morphology.

The metal AM process can be obtained through a high-speed camera, photodiodes, 3D vision, and laser ultrasound. This image information will be used to train AI for online real-time monitoring. The multisource dataset is subjected to data registration, feature extraction, and data coupling when performing an in situ quality evaluation. The processing results of multisource data are used for deep learning in in situ quality evaluation. The results after deep learning processing are used to control the parameters of pulsed laser shock. The stomatal suppression, stress regulation, and precision control are carried out for the molten pool and the solidification zone by controlling the process parameters of laser energy, impact frequency, scanning speed, and spot spacing. The final quality of metal AM parts can be characterized by surface accuracy, mechanical properties, and high-temperature properties. The real-time feedback control technical route described establishes a complete forming process, monitoring information, and part quality correlation relationship. The problem of repeatability and stability of the metal AM process will be effectively solved by collecting data, analyzing data, and using pulse laser shock control according to the above technical route.

4 Conclusion

This paper reviews the in situ and real-time monitoring of metal AM based on traditional image processing technology and AI image processing technology. The monitoring object, process classification, and image processing carrier are analyzed and compared for the traditional image processing technology. The traditional image processing technology has strict limitations on the environment. Thus, other algorithms may be needed for monitoring in complex environments. AI technology can process data with high speed and high precision. AI technology can also process images under challenging conditions, allowing real-time and accurate metal AM monitoring data processing. For image processing based on AI technology, in situ and real-time monitoring of metal AM is analyzed from three aspects: processing objects, accuracy, and processing time. Their advantages are compared in detail in terms of processing time and accuracy. In the future perspectives, image quality monitoring, small samples, image annotation, algorithm promotion, and real-time feedback control are analyzed in detail. The future development of real-time feedback control is discussed, and the feasible technical route is given by combining image processing technology with other technologies.

Fig. 1
figure 1

a Experimental equipment based on L-PBF process for real-time monitoring of molten pool; b monitoring curve of molten pool area [36]

Full size image
Fig. 2
figure 2

a Experimental equipment based on L-DED process for in situ monitoring of workpiece wall deformation; b monitoring curve of workpiece wall deformation [39]

Full size image
Fig. 3
figure 3

a Experimental equipment based on L-DED process for in situ monitoring of workpiece temperature; b temperature distribution near the defect [41]

Full size image
Fig. 4
figure 4

a Experimental equipment based on L-DED process for in situ monitoring of thermal image of molten pool; b judgment results of dual control chart [42]

Full size image
Fig. 5
figure 5

a Experimental equipment based on L-DED process for real-time monitoring of molten pool; b results of abnormal classification of the molten pool using KNN model which adopting Euclidean distance [43]

Full size image
Fig. 6
figure 6

a Experimental equipment based on L-DED process for in situ monitoring of porosity; b prediction results of local volume porosity of the two samples (Test#1: power, 250 W; speed, 1 mm/s; Test#4: power, 150 W; speed, 4 mm/s) [44]

Full size image
Fig. 7
figure 7

a Experimental system based on L-DED process for in situ monitoring of molten pool; b extraction result of molten pool and spatters at a laser power of 400 W and a scanning speed of 6 mm/s [45]

Full size image
Fig. 8
figure 8

a Experimental equipment based on L-PBF process for real-time monitoring of molten pool; b classification result of CNN on molten pool area map (red dot, larger molten pool; blue dot, smaller molten pool) [46]

Full size image
Fig. 9
figure 9

a Experimental system based on L-PBF process for in situ monitoring of molten pool; b extraction result of molten pool [47]

Full size image
Fig. 10
figure 10

a Experimental system based on L-PBF process for in situ monitoring of spatter; b extraction result of spatter at a scanning speed of 200 mm/s [48]

Full size image
Fig. 11
figure 11

a Experimental system based on L-PBF process for in situ monitoring of spatter; b extraction result of spatter at a scanning speed of 400 mm/s [48]

Full size image
Fig. 12
figure 12

a Experimental platform based on WAAM process for in situ monitoring of molten pool; b comparison of predicted results and actual results at different time steps [50]

Full size image
Fig. 13
figure 13

a Experimental equipment based on WAAM process for in situ monitoring of molten pool; b prediction results of weld reinforcement at 5th layer [51]

Full size image
Fig. 14
figure 14

a Experimental system based on PAM process for in situ monitoring of molten pool and plasma arc; b extraction result of molten pool and plasma arc [11]

Full size image
Fig. 15
figure 15

a Scanning strategy for in situ monitoring experiment of workpiece surface during L-PBF process; b evolution of the representative feature map of the last convolutional layer by using DCNN at different classification accuracies (Test A, standard energy; Test B, low energy; Test C, high energy; and Test D, very low energy) [52]

Full size image
Fig. 16
figure 16

Superimposed image of the segmented image and post-fusion image based on the EBM process [53]

Full size image
Fig. 17
figure 17

a Experimental equipment for in situ monitoring of thermal images of the L-DED process; b thermal images under different process parameters [54]

Full size image
Fig. 18
figure 18

Technical route for real-time feedback control

Full size image