1 Introduction

Nowadays, the universal law of life has been recognized by the study of phenotypic features of common modal animals, such as zebrafish [10, 18], mouse [26], and rat [32] etc. Traditional modal animals have made contributions to the understanding of the elements, such as cells, tissues functions, and the basic mechanisms of life. Therefore, the modal animal phenotypic feature analysis is an effective way to understand human physiology and pathology. However, the life ways of small-scale modal animals are different from human, which do not meet the needs of human major diseases research. So the study on multi-feature representation method of large-scale modal animals has become an important research area. In this regard, the subject of this study encompasses the detection of pig-body shape and temperature feature for breeding purposes.

The current pig-body shape segmentation systems capture pig-bodies in a controlled environment as part of the body shape segmentation process. It is possible to control the illumination, background, and quality of images. Under these conditions, the performance of pig-body shape segmentation algorithms is greatly enhanced. However, there is still a need for more robust and efficient pig-body shape segmentation algorithms to address challenges such as variations in illumination, pose, etc.

With the development of imaging technology, the pig-body segmentation method based on visible images (VI) is proposed, which has achieved high accuracy [15]. However, it cannot effectively segment pig-body in low light, especially the tail and legs, as shown in Fig. 1e,g. As the hidden targets can be found in infrared (IR) images based on dim environment, the pig-body segmentation through IR images is achieved according to bull-body segmentation [34]. However, it cannot detect ears effectively, as shown in Fig. 1b, d, f, h. Under different environments, the segmentation results of various parts pig-body, such as ears, legs, and tails, are different in VI and IR images, as shown in Fig. 1. Therefore, the IR and VI image fusion can provide comprehensive and useful information through integrating complementary information and removing redundancy in a multi-source image [2, 42]. Moreover, the temperature feature of pig-body can effectively extract based on more accurate pig-body shape segmentation results and infrared pig-body images.

Fig. 1
figure 1

The segmentation of pig-body shape based on visible and infrared images under different situations. a-d the detection results based on clearer environment, e-h the detection results based on the dimmer environment

Full size image

The structure of this paper is that the related works are introduced in section 2, materials and methods are presented in section 3, experimental results and discussion are provided in section 4, and the conclusions and future work are given in section 5.

2 Related works

2.1 Pig-body segmentation

Pig-body is important due to its unique shape and structure. So several methods have been devised to segment pig-body. To exactly recognize the size and features of pigs’ bodies, Shen and Luo [33] proposed a segmentation system based on visual image analysis. To automatically segment the three-dimensional structure of a pig-body, a 3D visual imaging based pig-body reconstruction system was proposed by Font-I-Furnols et al. [7]. However, the pig-body segmentation method based on visible images varies due to environmental changes, such as illumination, etc. Therefore, it is hard to obtain good results concerning a pig-body segmentation. Based on the advantages of the infrared imaging [34], some multi-source image fusion methods have been proposed for target segmentation [6, 27, 45]. The experimental results reveal that these methods, based on visible and infrared image fusion, achieve better segmentation.

2.2 Multi-source image fusion

The traditional multi-focus image fusion algorithms usually are divided into two aspects, based on spatial and transform domain. Spatial domain based fusion methods, such as principal component analysis (PCA) [14] and intensity hue saturation (IHS) [17], provide a fused image with lower contrast information. As the image features with various scales are sensitive to the human visual system [4], multi-scale transformations are effective for fusing the information of multi-focus images. For example discrete wavelet transform (DWT) [44], discrete cosine transform (DCT) [13, 37], contourlet pyramid (CT) [8], and shearlet transform (ST) [35]. However, there are two drawbacks. One is that these transform methods cannot fully represent the curves and edges of multisource images. The other is that ST and CT are sensitive to shift variance and easily cause Gibbs phenomenon in the fused image due to sampling operator. Nowadays, the learning-based fusion methods has proposed to improve the adaptability of algorithm, such as convolutional neural network (CNN) [21], generative adversarial network (GAN) [25]. However, they are time consuming for training and have high hardware requirements.

To effectively solve the above issues and improve the performance of pig-body segmentation rate, a novel multisource image fusion method is proposed for pig-body shape segmentation and temperature detection based on Gabor feature and IAG, named as NSCT-GF-IAG. Through this procedure, a pig-body multi-feature detection model is achieved to segment the shape feature by fusing VI and IR images using Gabor energy map and improved average gradient (IAG) in the NSCT domain. Then the fused images are segmented by automatic threshold method and morphological processing. Next, the temperature of pig-bodies is detected based on segmentation results. Finally, the experimental results show that the proposed fusion method has a better effect in the pig-body shape segmentation and provides a higher accuracy rate in a homemade database. The flowchart of the proposed method is shown in Fig. 2.

Fig. 2
figure 2

Flowchart of the proposed detection framework

Full size image

Generally, the advantages of the proposed method are summarized as follows:

  1. (1)

    In the low-frequency sub-bands, to construct a new significant measuring method, the local energy features of multisource pig-body images are extracted by even-symmetry Gabor filter with different parameters, respectively. Then, an effective fusion rule of low-frequency sub-bands is proposed based on the new significant measures for detecting pig-body.

  2. (2)

    In the high-frequency sub-bands, to better represent large-scale texture and edge information, the high-frequency energy features are extracted by improved average gradient based on pig-body images. Then, the maximum strategy is adopted for fusing high frequency sub-bands.

  3. (3)

    The proposed pig-body segmentation framework achieves a better segmentation rate than other prevailing methods, and improve the detection rate of pig-body shape and temperature.

3 Materials and methods

The goal of this work was to improve the segmentation rate of pig-body images based on situations with different illumination. For this purpose, an effective method was proposed based on Gabor filter with even-symmetry and improved average gradient in NSCT domain. The algorithm was divided into four major phases. First, IR and VI images were decomposed into a series of multi-scale and multi-directional sub-bands by NSCT. Next, the proposed fusion rules were adapted to fuse the low and high-frequency coefficients. Then, the fused images of pig-body were obtained by inverse NSCT based on different situations. Finally, binary images were presented by automatic threshold segmentation and morphological operation, and the temperature features were obtained based on segmentation results.

3.1 Non-subsampled contourlet transform

Due to the shift invariance of NSCT, the IR and VI images were decomposed into multiscale and multidirectional sub-bands using the non-subsampled pyramid filter bank (NSPFB) and the non-subsampled directional filter bank (NSDFB). First, the multiscale sub-bands were obtained by pyramid filter bank (PFB) without up-sampling and down-sampling. Thus, one low-frequency and one high-frequency sub-band were generated at each decomposition level. Non-subsampled pyramid (NSP) decomposition was achieved by iteratively decomposing the low-frequency sub-band. For the number of decomposition level (k), sub-bands (k + 1) were obtained by NSP, which were the same size as the original images. Then, the high-frequency sub-bands at each level were decomposed by NSDFB at l directions. There were 2l directional sub-bands generated, which contained richer directional information. The processing of decomposition with NSCT is shown in Fig. 3.

$$ {H}_n(Z)=\Big\{{\displaystyle \begin{array}{cc}{H}_1\left({z}^{2^{n-1}}\right){\prod}_{j=0}^{n-2}{H}_0\left({z}^{2^j}\right)& 1\le n<{2}^k\\ {}{\prod}_{j=0}^{n-1}{H}_0\left({z}^{2^j}\right)& n={2}^k\end{array}} $$
(1)
Fig. 3
figure 3

The multi-scale and multi-directional decompositions of NSCT

Full size image

Here, H0(z) and H1(z) represent a low-pass and a high-pass filters respectively, in first step, H0(z2) and H1(z2) are low and high pass filters respectively in the second step, k = 3.

3.2 The fusion rule of low-frequency sub-bands

To enhance the curve structure and fine-scale texture information [38], the low-frequency sub-bands were described by Gabor filter with even-symmetry. The magnitude feature with eight orientations (0°, 22.5°, 45°, 67.5°, 90°, 112.5°, 135°, and 157.5°) [39] were extracted through Gabor filter with even-symmetry, which is represented by M{i}. The formula of Gabor filter with even symmetrical and multi-orientation is shown as

$$ G\left\{1\right\}\left(x,y\right)=\frac{\gamma }{2\pi {\sigma}^2}\mathit{\exp}\left\{-\frac{1}{2}\left(\frac{x_{\theta_i}^2+{\gamma}^2{y}_{\theta_i}^2}{\sigma^2}\right)\right\}\exp \left(\hat{j}2\pi {x}_{\theta_i}\right) $$
(2)
$$ M\left\{i\right\}\left(x,y\right)=I\left(x,y\right)\bigotimes \mathit{\operatorname{Re}}\left(G\left\{i\right\}\left(x,y\right)\right) $$
(3)
$$ \left[\begin{array}{c}{x}_{\theta_i}\\ {}{y}_{\theta_i}\end{array}\right]=\left[\begin{array}{cc}{\mathit{\cos}}_{\theta_i}& {\mathit{\sin}}_{\theta_i}\\ {}-{\mathit{\sin}}_{\theta_i}& cos\theta \end{array}\right]\left[\begin{array}{c}x\\ {}y\end{array}\right] $$
(4)

Where, fi is the center frequency of the ith orientation, θi is the angle of the ith orientation, i = 8, σ and γ are the scale of Gabor filter and the length-width radio of envelope respectively. Due to the different curve structure of IR and VI images, varying parameters were adapted in multisource images. σIR = 3 and σVI = 6, I(x, y) represents IR or VI image, and Re(G{i}) is the real part of G{i}.

Then, the Gabor energy feature was obtained by choosing the maximum of all directions in the m × m local window and the formula is shown as follows:

$$ E\left(x,y\right)=\mathit{\operatorname{MAX}}\left(\left( abs\left(M\left\{i\right\}\left(x,y\right)\right)\right)\hat{\mkern6mu} 2\right) $$
(5)

Where, i represents the number of orientations, MAX(.) is the maximum value in the energy of all orientations, E(.) is the Gabor energy feature, m = 7.

Finally, to better fine-scale feature description of pig-body in low-frequency sub-bands, the maximum fusion rule was adopted based on local Gabor-based energy feature.

$$ L{F}_F=\Big\{{\displaystyle \begin{array}{cc}L{F}_{IR}& {E}_L^{IR}\left(x,y\right)\ge {E}_L^{VI}\left(x,y\right)\\ {}L{F}_{VI}& {E}_L^{IR}\left(x,y\right)<{E}_L^{VI}\left(x,y\right)\end{array}} $$
(6)

Where, \( {E}_L^{IR} \) and \( {E}_L^{VI} \) are Gabor energy map of IR and VI low-frequency sub-bands, respectively. LFF represents the low-frequency fused sub-bands.

3.3 The fusion rule of high-frequency sub-bands

Previously, the average gradient fusion rule represented better coarse-scale textures of high-frequency sub-bands [8]. However, the diagonal gradients were ignored in the fusion rule. In this work, an improved average gradient (IAG) was adopted to enhance the representation of coarse-scale details in high-frequency coefficients fusion. For this purpose, the average gradient of region R was computed first, and formulated as follows:

$$ G=\frac{1}{\left(M-1\right)\left(N-1\right)}\sum \limits_{i=1}^{M-1}\sum \limits_{J=1}^{N-1}\sqrt{\Big(\Delta {I}_x^2\left(x,y\right)+\Delta {I}_y^2\left(x,y\right)}+\Delta {I}_d^2\left(x,y\right)+\Delta {I}_b^2\left(x,y\right)\Big)/4 $$
(7)

Where, ∆Ix is derivative of I(x, y)in X direction, ∆Iy is derivative of I(x, y) in Y direction, ∆Id is derivative of I(x, y)in diagonal direction, ∆Ib is derivative of I(x, y) in back-diagonal direction. The size of region R is M × N.

Next, the fusion rule of high-frequency sub-bands was determined by choosing the maximum based on IAG.

$$ H{F}_F=\Big\{{\displaystyle \begin{array}{cc}H{F}_{IR},& {G}_H^{IR}\left(x,y\right)\ge {G}_H^{VI}\left(x,y\right)\\ {}H{F}_{VI},& {G}_H^{IR}\left(x,y\right)<{G}_H^{VI}\left(x,y\right)\end{array}} $$
(8)

Where, \( {G}_H^{IR} \) and \( {G}_H^{VI} \) are IAG of IR and VI high-frequency sub-bands, respectively. HFF is the high-frequency fused sub-bands.

3.4 Image reconstruction and binarization

To achieve pig-body image reconstruction, inverse non-subsampled contourlet transform (INSCT) was used to obtain the final fused image. Then, the binary pig-body images were obtained by automatic threshold segmentation, to remove the holes and noise, the morphological operation was adopted to optimize the binary pig-body image. Finally, highest temperature of pig-body was detected based on segmentation results.

4 Experimental results and discussion

The proposed detection framework was implemented using MATLAB R2014a on a standard desktop PC, which was equipped with an Intel(R) Core(TM) i5–6200 CPU @2.3GHz and 4GB RAM. To analyze the performance of the proposed pig-body multi-feature detection method, a homemade database was built by FLIR C2. In this section, the schematic diagram of mulit-source image pairs were shown in Fig. 4. The figure showed the difference illumination situation of visible pig-body images in each line. Moreover, in the following detailed experimental image results, a pair of multi-source images were selected under different illumination situations, and there were four pairs of images in total, which to test performance of proposed method under illumination variation situations.

Fig. 4
figure 4

The schematic diagram of homemade database (a) VI pig-body images (b) IR pig-body images

Full size image

4.1 Objective evaluation

In this study, seven objective metrics were adopted to evaluate the fusion and detection results, which were average gradient (AG) [1], information entropy (IE) [23], standard deviation (SD) [24], spatial frequency (SF) [16], average pixel intensity (API) [31], similarity of structure information measure (SSIM) [22] and Accuracy (Acc).

AG was used to measure the sharpness of the fused image and defined as:

$$ AG=\frac{1}{\left(M-1\right)\left(N-1\right)}\times \sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}\sqrt{\left({\Delta I}_x^2+{\Delta I}_y^2\right)/2} $$
(9)

Where M and N is the number of row and column in the fused image respectively.

IE was used to measure the abundance of information in the fused image and defined as:

$$ IE=-\sum \limits_{i=0}^L{P}_i{\mathit{\log}}_2{P}_i $$
(10)

Where \( {P}_i=\frac{N_i}{N} \), Ni is the number of pixels whose value is i, and N is the number of total pixels.

SD was used to measure the contrast of the fused image and defined as:

$$ SD=\sqrt{\frac{1}{\left(M-1\right)\left(N-1\right)}\sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}\left(I\left(i,j\right)-\overline{I\left(i,j\right)}\right)} $$
(11)

Where, \( \overline{I\left(i,j\right)} \) is the average of the fused image.

SF was used to measure the overall structural information in the fused image and defined as:

$$ SF=\sqrt{(RF)^2}+{(CF)}^2 $$
(12)

Where, \( RF=\sqrt{\frac{1}{M\times N}\sum \limits_{i=1}^M\sum \limits_{j=2}^N{\left(I\left(i,j\right)-I\left(i,j-1\right)\right)}^2, CF=\sqrt{\frac{1}{M\times N}\sum \limits_{i=2}^M\sum \limits_{j=1}^N{\left(I\left(i,j\right)-I\left(i-1,j\right)\right)}^2}} \)

API was used to measure the average brightness of the fused image and defined as:

$$ API=\frac{1}{M\times N}\sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}I\left(i,j\right) $$
(13)

Where I(i, j) is the intensity of pixel at (i, j), M and N are the number of row and column in the fused image respectively.

SSIM was used to measure the similarity of image structure information and defined as:

$$ {SSIM}_{X,F}=\sum \limits_{x,f}\frac{2{\mu}_x{\mu}_f+{C}_1}{\mu_x^2+{\mu}_f^2+{C}_1}.\frac{2{\sigma}_x{\sigma}_f+{C}_2}{\sigma_x^2+{\sigma}_f^2+{C}_2}.\frac{\sigma_{xf}+{C}_3}{\sigma_x{\sigma}_f+{C}_3} $$
(14)
$$ SSIM={SS\mathrm{IM}}_{A,F}+{SS\mathrm{IM}}_{B,F} $$
(15)

Where A and B are multi-source images, μx and σx are mean and variance of multisource image, respectively. μf and σf are mean and variance of fused image, respectively. σxf is covariance between multisource image and fused image.

Acc was used to measure the detection rate of the fused image and defined as:

$$ Acc=\frac{TP+ TN}{TP+ TN+ FP+ FN}\times 100\% $$
(16)

Where TP and TN are true positive and true negative respectively, FP and FN are false positive and false negative respectively.

4.2 Fusion performance of the proposed model

To verify the performance of the proposed fusion method, four pairs of multisource pig-body were selected based on different situations and shown in Fig. 5a-b. The corresponding fused images based on the proposed method are shown in Fig. 5c. It can be clearly noticed that the fused pig-body images were approximately similar to the VI images by fusing the features of VI and IR images. Moreover, the visual effects of VI, IR, and the fused images were also verified by computing AG, API, IE, SD and SF. The results are shown in Table 1. Furthermore, the comparative results of IE and SD are shown in Fig. 5d-e, which depict that higher information entropy and standard deviation were obtained by the proposed fusion model. They also reflect that the proposed fusion model had a great improvement in information and contrast of fused images.

Fig. 5
figure 5

Source images of pig-body and comparable results (a) VI pig-body image (b) IR pig-body image (c) Fused image obtained by the proposed fusion method (d) comparison of information entropy (e) comparison of standard deviation

Full size image
Table 1 Performance of proposed fusion model in different situations
Full size table

4.3 Fusion results of the proposed detection model

The performance of the proposed multi-source image fusion method was evaluated by different experimentation with pig-body images, which was further utilized to extract pig-body regions. To verify the performance of the fusion method, four VI and IR pig-body image pairs in different illumination were considered and shown in Fig. 6a-b. Moreover, the proposed fusion method was compared with six existing methods, namely, discrete wavelet transform (DWT) [3, 30], discrete cosine transform (DCT) [19, 29], contourlet transform (CT) [28, 41], shearlet transform (ST) [9, 20], NSCT [5, 43], NSCT-PCNN [36, 40], NSST [11, 12], CNN [21], GAN [25]. The main difference between proposed method and other considered methods was that the fusion methods of low-frequency and high-frequency subbands were different, which were shown in Table 2. The final fused images of different fusion methods based on different situations are shown in Fig. 6c-l. The figure shows that the proposed method had some advantages in the visual effect, especially in dimmer situations, it clearly represents the pig-body shape feature.

Fig. 6
figure 6

The fusion results of different methods based on different situations (a) VI pig-body image, (b) IR pig-body image, (c) DWT, (d) DCT, (e) ST, (f) CT, (g) NSCT, (h) NSCT-PCNN, (i) NSST, (j) CNN, (k) GAN, (l) Proposed

Full size image
Table 2 The difference fusion methods based on low-frequency and high-frequency subbands
Full size table

Furthermore, it was also superior to the other nine image fusion methods in objective evaluations, as shown in Table 3. To intuitively compare the performance of different multisource fusion methods, the comparative results are represented in scatter diagrams as shown in Fig. 7. This also reflects that due to the better description of Gabor filter with even-symmetry and IAG, the proposed method provided a significant improvement in visual effect, as well as, the objective evaluation based on different situations, especially dimmer situations.

Table 3 Objective Evaluation results of various fusion methods in different situations
Full size table
Fig. 7
figure 7

The comparison of different methods with objective evaluations based on different situations (a-b) brighter environment (c-d) dimmer environment

Full size image

To testify the performance of the proposed fusion model in homemade database, the comparative results were represented in line charts as shown in Fig. 8. The x-axis of Fig. 8 represented the number of multi-source image pairs of breeding pigs. The figure shows that the fused image quality is best among 10 fusion methods in homemade database with different situations. Moreover, the running times of various fusion methods based on transform domain are shown in Table 4. Compared with NSCT-PCNN, NSCT and NSST, the proposed fusion method consumed less running time. Because of the train processing of CNN and GAN take a long time, respectively, they are not advantageous in running time.

Fig. 8
figure 8

The comparison of different methods with objective evaluations in homemade database

Full size image
Table 4 Computation times of various fusion methods
Full size table

4.4 Pig-body segmentation results

To verify the performance of the proposed pig-body segmentation model, 24 pig-body image pairs were processed with the proposed fusion method and other existing fusion methods as mentioned above. After obtaining all the 240 fused images with 10 different fusion methods, shape features of pig-bodies were extracted by the automatic threshold method. To optimize the segmentation results, the morphological operation was utilized to remove holes and noise. To assess the performance of pig-body segmentation results based on 4 image pairs in different situations, the VI and IR pig-body images with different illumination were segmented in Fig. 9a-b, respectively. The binarization of fused images with different situations based on different fusion methods are shown in Fig. 9c-l. The extracted binary images in Fig. 9, reveal that the proposed pig-body segmentation model is capable to segment higher integrity of pig-body regions, especially the ears, legs, and tail, based on dimmer situations.

Fig. 9
figure 9

The segmentation results of different methods based on different illumination situations (a) IR pig-body image, b VI pig-body image, c DWT, d DCT, e ST, f CT, g NSCT, h NSCT-PCNN, i NSST, j CNN, k GAN and l Proposed

Full size image

The proposed pig-body detection method achieved higher segmentation accuracy than all the considered fusion methods, as shown in Table 5 above. As per Table 5, the average performance of the proposed segmentation was 94.452%, which is 2.175–5.129% higher than other considered methods. To prove the segmentation performance of proposed method, the results were represented in line charts and shown in Fig. 10. The x-axis of Fig. 10 represented the number of multi-source image pairs of breeding pigs. The figure shows that the proposed detection method achieves highest segmentation rate among 10 different methods.

Table 5 The comparative accuracy of the proposed pig-body detection method with other fusion models
Full size table
Fig. 10
figure 10

The comparison of detection accuracy in homemade database

Full size image

4.5 Pig-body temperature detection results

To eliminate the influence of ambient temperature on pig-body temperature detection, the highest temperature of pig-body was computed based on pig-body shape segmentation and the pixels of infrared pig-body image. Firstly, the shape segmentation results were normalized and multiplied with infrared pig-body image. Then, because of each pixel value was the temperature of pixel in infrared images, the highest temperature was extracted by comparing pixel by pixel. Finally, the position of reference temperature in different situations were obtained by FLIR Tools, as shown in Fig. 11. To detect the health degree of pig in a homemade database, the highest temperature of pig-bodies were represented in line charts as shown in Fig. 12. This figure shows that the highest temperature of pigs are all in the normal temperature range in the homemade database.

Fig. 11
figure 11

The detection and position of highest temperature and the position in different situations

Full size image
Fig. 12
figure 12

The detection and position of highest temperature and the position in different situations

Full size image

5 Conclusion and future works

A novel infrared and visible image fusion method for pig-body detection was proposed based on different illumination, named NSCT-GF-IAG. First, the NSCT was adapted to decompose multisource images into a series of multi-scale and multi-directional sub-bands. Then, Gabor filter with even-symmetry and IAG were used to fuse low-frequency and high-frequency sub-bands, respectively. Next, the fused coefficients were directly fused into final pig-body fusion image. Finally, the binary images of pig-body were presented by the automatic threshold method and morphological operation. Further, the highest temperature of pig-body was obtained based on segmentation results. The experimental results show that the proposed detection method had a superior performance in improving the average detection rate (94.452%) based on pig-body images in four different situations, which is 2.175–5.129% higher than other considered methods. It lays a foundation for accurately measuring the maximum temperature of pig-body.

This work mainly focused on improving the problem of pig-body shape and temperature detection under different situations. Some parameters of the proposed algorithm were not selected automatically for optimal fusion performance. Future studies should focus on the automatic selection of algorithm parameters, which may replace the process of parameter determination with empirical values and experimentation. Moreover, the running time of the proposed algorithm needs to be further reduced.