1 Introduction

The fusion of multi-modal medical images is the combination of useful information from multiple images individually to create a single image. This allows the fusion image to not only contains various types of information but also enhanced significantly in terms of quality. Currently, there are various medical images, such as single photon emission tomography (SPECT), computed tomography (CT), positron emission tomography (PET), and magnetic resonance imaging (MRI). Each kind of medical image can only contain one type of information. For instance, SPECT image provides functional and metabolic information with low spatial resolution, while MRI images contain anatomical information with high spatial resolution.

Currently, approaches proposed to fuse multi-modal medical images are grouped into two primary domains, namely spatial and transform domain [30]. The spatial domain-based approaches process directly on the pixels, blocks, or regions of the input images without making the transformation from the spatial domain to the transform domain. In this category, there are various proposed methods, such as pixel-based [24], region-based [34, 65], and block-based methods. The spatial approaches utilize various techniques, such as the maximum selection rule (MSR), the minimum selection rule, weighted-average rule [44]. The advantage of these methods is often implemented simply, and they have low computational complexity. The drawback of these methods is the loss of useful information in the composite image as they produce spectral distortion and color distortion. This disadvantage is overcome by transform-domain based approaches.

The transform domain-based methods are more commonly used in recently proposed methods, and they consist of three main stages, including image transformation, the fusion of components in the transform domain, and inverse transformation [30]. Firstly, the input medical images are transformed into a transform domain by adopting a certain method of image transformation. Secondly, the components in the transform domain are fused by a certain fusion rule. Finally, the fusion image is reconstructed by utilizing the corresponding inverse transformation on the fusion components. In general, transform domain-based approaches can be further divided into Pyramid methods, Wavelet transform methods, and Multi-scale geometric analysis (MGA) methods.

There are some disadvantages of the above transform domain-based methods. Pyramid-based approaches as the Laplacian pyramid (LP) [10, 14, 51] can cause the loss of considerable sourced information and suffer from the blocking effect since they supply only spectral information without having the directional information. Wavelet-based methods, such as Discrete wavelet transform (DWT) [58] and Dual-Tree Complex Wavelet Transform (DTCWT) [62], do not use phase information, so the main disadvantage of them is unable to preserve the edges and texture regions. MGA- based methods, such as Contourlet transform [57], Curvelet transform [37], Non-subsampling contour transform (NSCT) [23, 50, 64], Non-subsampled shearlet transform (NSST) [6, 27], provide both spectral and directional information, so they can transcend the limitations of pyramid-based and wavelet-based approaches. Nevertheless, the disadvantage of MGA-based methods is high computational complexity.

In recent years, many meta-heuristic optimization algorithms have been introduced and applied to medical image fusion. Some typical meta-heuristic optimization algorithms can be mentioned as Particle swarm optimization (PSO) [42], non-subsampled shearlet transform and particle swarm optimization (NSST-PSO) [47], quantum-behaved particle swarm optimization (QPSO) [55], gray wolf optimization [5], hybrid genetic–grey wolf optimization (HG-GWO) [4], modified central force optimization (MCFO) [12], chaotic grey wolf optimization (CGWO) [3], binary crow search optimization (BCSO) [39], the total variation (TV-L1) based cartoon-texture decomposition with particle swarm optimization (TV-L1-PSO) [38], and modified shark smell optimization (MSSO) [54].

The equilibrium optimizer algorithm was inspired by the dynamic mass balance in physics, and it was a recently proposed meta-heuristic algorithm. This algorithm was originally introduced by Faramarz [13] in 2020. Currently, this algorithm has been applied successfully in various problems such as feature selection [16], biological data classification [48], and vehicle routing problem [15]. However, according to our observation, there are no studies utilizing the EOA for medical image fusion. This encouraged and motivated us to propose a novel approach based on the EOA for medical image fusion.

In addition, according to our current knowledge, there are some limitations of current image fusion approaches. The first one is that the use of a weighted average rule for the fusion of low-frequency components. Some recent approaches have this limitation such as convolutional sparse representation (CSR) [28], convolutional sparsity with morphological component analysis (CSMCA) [29], two-scale image decomposition with sparse representation (TSID-SR) [33], two-scale image decomposition with structure tensor (TSID-ST) [11], and two-scale image decomposition with guided filtering and sparse representation (TSID-GF-SR) [40]. This limitation leads to a decrease in the intensity of the brightness of the fused image. The second limitation is that the use of fusion rules for high-frequency coefficients cannot be optimal because the detailed information preservation indexes are not really high. Some fusion rules can be mentioned as Max selection [44], Local variance [2], and Parameter-adaptive pulse coupled neural network (PA-PCNN) [61]. This is likely to result in the loss of detailed information in the fused image. In this paper, we propose two novel algorithms to address the above-mentioned limitations. Some advantages of the proposed approach are highlighted as follows:

  • The first algorithm makes use of the Equilibrium Optimizer Algorithm (EOA) to find optimal parameters with the aim of fusing the base layers. This allows the fusion image to have good contrast.

  • The second algorithm is based on the sum of local energy functions using the Prewitt compass operator to create an efficient rule for the fusion of the detail layers. This allows the fusion image to preserve significantly details transferred from input images.

The remaining of this paper is organized as follows: Some background knowledge, such as the TSD method, YCbCr color space, and the EOA algorithm, is introduced briefly in Section 2. Section 3 introduces two novel algorithms in the proposed approach: the first one based on the sum of local energy functions using the Prewitt compass operator is proposed to fuse detail layers, and the second algorithm based on the EOA is introduced to combine base layers. Section 4, the quality of fused images is evaluated by different indexes. Finally, the conclusion and future work are given in Section 5.

2 Background

Some background knowledge, such as TSD method, YCbCr color space, and EOA algorithm, are introduced in this section.

2.1 TSD method

There are a number of different methods for decomposing an image into two scales. These methods have been utilized in many medical image fusion approaches such as Convolutional sparse representation (CSR) [28], Two-scale image decomposition with statistical comparisons [9], Two-scale image decomposition with structure tensor (TSD-TS) [11], Two-scale image decomposition and sparse representation (TSD-SR) [33], and Two-scale image decomposition with guided filtering and sparse representation (TSD-GF-SR) [40]. In this section, we present the TSD method used in TSD-SR and CSR approaches. According to this method, an input image (Iin) is decomposed into two components. The first one is the base layer (Ib) having large scale variations, and the second one is the detail layer (Id) having small scale variations.

Figure 1 illustrates the use of TSD method for a input medical image.

Fig. 1
figure 1

Illustration for the two-scale image decomposition: Iin, Ib, and Id

Full size image

Ib and Id components can be calculated as follows:

Firstly, the component Ib is found by addressing the following optimization problem (see Eq. (1)):

$$ \underset{I^{b}}{argmin} \begin{Vmatrix} I_{in}-I^{b} \end{Vmatrix}_{F}^{2} +\gamma (\begin{Vmatrix} d_{x}*I^{b} \end{Vmatrix}_{F}^{2} + \begin{Vmatrix} d_{y}*I^{b} \end{Vmatrix}_{F}^{2} ) $$
(1)

Where,

  • γ is the regularization parameter.

  • \(d_{x} = \begin {bmatrix} -1 & 1 \end {bmatrix}\) is a gradient operator in the horizontal direction.

  • \(d_{y} = \begin {bmatrix} -1 & 1 \end {bmatrix}^{T}\) is a gradient operator in the vertical direction.

The (1) is a Tikhonov regularization problem, and it can be addressed efficiently by making use of Fast Fourier Transform (FFT).

Secondly, the detail layer (Id) is calculated from input image Iin and the base layer (Ib) as shown in Eq. (2):

$$ I^{d} = I_{in} - I^{b} $$
(2)

2.2 YCbCr color space

There are different types of color models proposed for the fusion of color medical image, such as IHS [25, 38, 46], HSV [20], YIQ [49], YUV [26, 31, 59], and YCbCr. Among these color models, YCbCr color space is effectively applied in various image processing problems, such as Watermarking [22], Multi-Focus Image Fusion [1], and Encryption for color images [60]. Therefore, this color space is opted for the composition of medical images in the proposed model.

The RGB-YCbCr transformation is represented as shown in Eq. (3):

$$ \begin{bmatrix} Yy\\ Cb\\ Cr \end{bmatrix} = \begin{bmatrix} 0.257 & 0.564 & 0.098 \\ -0.148 & -0.291 & 0.439 \\ 0.439 & -0.368 & -0.071 \end{bmatrix} \begin{bmatrix} Rr\\ Gg\\ Bb \end{bmatrix} + \begin{bmatrix} 16\\ 128\\ 128 \end{bmatrix} $$
(3)

The conversion of YCbCr colour model to RGB colour model is represented as shown in Eq. (4):

$$ \begin{bmatrix} Rr\\ Gg\\ Bb \end{bmatrix} = \begin{bmatrix} 1.164 & 0.000 & 1.596 \\ 1.164 & -0.392 & -0.813 \\ 1.164 & 2.017 & 0.000 \end{bmatrix} \begin{bmatrix} Yy - 16\\ Cb - 128\\ Cr - 128 \end{bmatrix} $$
(4)

2.3 EOA Algorithm

The EOA algorithm was inspired by the dynamic mass balance in physics, and it was originally introduced by Faramarzi [13] in 2020. More details about the EOA algorithm can be found in Algorithm 1. Generally, the EOA algorithm is described by the following six steps:

  • Step 1: The population of particles is initialized

  • Step 2: The fitness of particles in the population is estimated.

  • Step 3: The equilibrium pool is constructed.

  • Step 4: Each exponential term (F) and the generation rate (G) are updated using Eqs. (5) and (6), respectively.

    $$ F = {\upbeta}_{1} sign(k-0.5)[e^{-\gamma t}-1] $$
    (5)
    $$ G = \left\{\begin{array}{ll} 0.5.k_{1}.(K_{eq}-\gamma K).F & k_{2}\geq GP \\ 0 & k_{2} < GP \end{array} \right. $$
    (6)

    Where

    $$ t =\left (1-\frac{Iter}{Iter_{max}}\right)^{{\upbeta}_{2} \frac{Iter}{Iter_{max}}} $$
    $$ {\upbeta}_{1} = 2, {\upbeta}_{2} = 1, GP = 0.5 $$

  • Step 5: Each particle concentration is updated using Eq. (7):

    $$ K = K_{eq} + (K-K_{eq}).F + \frac{G}{\gamma V} (1-F) $$
    (7)

  • Step 6: Go to step 3 until the stopping criteria is achieved.

figure a

3 The proposed approach

In this section, two novel algorithms are proposed. The first one, based on the sum of local energy functions using the Prewitt compass operator, is proposed to fuse detail layers. The second algorithm is designed to fuse base layers based on Equilibrium Optimizer Algorithm.

3.1 Fusion rule based on the sum of local energy function with Prewitt Compass Operator (SLE_PCO)

The detailed information of an image is contained in the high-frequency components, and the energy of detail layers of a sharp image is much larger than that of a blurred one. Therefore, some approaches, such as NSCT transform and local energy [2], Empirical wavelet decomposition and maximum local energy (EWT-MLE) [41], and Maximum local energy with compass operator [7, 8], have applied the fusion rules based on local energy.

The local energy LE(x,y) is calculated using Eq. (8).

$$ LE(x,y) = \sum\limits_{s_{1} = 0}^{u-1} \sum\limits_{s_{2} = 0}^{u-1} W(s_{1},s_{2})*{I_{m}^{2}}(x+s_{1},y+s_{2}) $$
(8)

Where

  • W is the unit matrix of size u x u.

  • Im is the image matrix.

Prewitt operator is one of the compass operators used to detect edge, and it was first introduced by Russell Prewitt. This method uses eight kernel masks in eight directions (see details in Fig. 2). Currently, this operator is widely and effectively applied in image processing algorithms such as Auto-focusing approach on multiple micro objects [32] and Quantum image edge extraction [63].

Fig. 2
figure 2

Illustration for 8 masks of Prewitt compass method

Full size image

We propose a novel rule based on the sum of local energy function and Prewitt operator for fusing detail layers as Algorithm 2. Figure 3 illustrates functions (Ei, \(i = \overline {1,8}\) and Esum) using Prewitt compass method.

figure b
Fig. 3
figure 3

Illustration for functions (Ei, \(i = \overline {1,8}\) and Esum) based on Prewitt compass method

Full size image

3.2 The proposed algorithm based on SLE_PCO and EOA

There are some main steps in the proposed method. In the first step, RGB-YCbCr transformation is implemented to convert colour medical images to YCbCr color space, and only the Y channel is utilized for the next step. In the second step, the TSD method is implemented to decompose gray medical images with the aim of receiving the base and detail layers. Then, the EOA algorithm is applied to find optimal parameters for the fusion of base layers, and the SLE_PCO method is implemented to compose the detail layers. Finally, two components (the detail layer and base layer) are summed to get the reconstructed image, and this fused image, together with Cb and Cr channels, is transformed to RGB color space. Figure 4 illustrates the diagram of the proposed approach utilizing TSD and EOA.

Fig. 4
figure 4

The diagram of the proposed approach using TSD and EOA

Full size image

4 Experimental results

Quality indexes and the data set using for experiments are presented in this section. In addition, the results received from experiments are displayed and discussed.

4.1 Objective evaluation metrics

Currently, there are many image fusion quality indexes introduced to evaluate the fused images. In this section, eight metrics are selected, including Mean (intensity index), Standard deviation (SD - contrast index), average information of an image (Entropy), Sharpness index (S), Edge-based similarity measure (QAB/F) [56], Visual Information Fidelity for Fusion (VIFF) [19], Feature Mutual Information (FMI) [18]), and Mutual Information (MI) [17].

4.2 Experimental setup

Experimental data consists of three pairs of medical images (MRI-PET), namely Data-1 (a,b), Data-2 (c,d) and Data-3 (e,f) with a size of 256 × 256 pixels (See Fig. 5 for details). The above-mentioned images were obtained from the online source as follows: http://www.med.harvard.edu/AANLIB/.

Fig. 5
figure 5

Illustration for medical images in input image dataset: Data-1 (a,b), Data-2 (c,d), and Data-3 (e,f)

Full size image

Several different experiments were conducted to evaluate the effectiveness of the proposed approach. The first experiment: to evaluate the effectiveness of the EOA algorithm in the proposed method, some well-known metaheuristics optimization algorithms such as Particle Swarm Optimization (PSO) [21], Artificial Bee Colony (ABC) Optimization, Biogeography-based optimization (BBO) [43], Multi-Verse Optimizer (MVO) [36], and Whale Optimization Algorithm (WOA) [35] have been selected for comparison. 30 runs were performed with different optimization algorithms for each pair of images. Two indexes (Average and Standard deviation) are utilized to compare the overall performance of algorithms. In addition, the Wilcoxon rank-sum test [52], a non-parametric statistical test, was used to determine the significance of the results.

The second experiment: :

Various compass operators such as Isotropic, Robinson, Kirsch, and Prewitt, were used in our approach to find the right compass operator.

The third experiment: :

With regard to qualitative analysis, the results of our approach are compared with those of other latest approaches, namely: convolutional sparsity based morphological component analysis (CSMCA) [29], Non-subsampled contourlet transform (NSCT) [64], convolutional sparse representation (CSR) [28], Non-subsampled shearlet transform with parameter-adaptive pulse coupled neural network (NSST-PA-PCNN) [61], and Non-subsampled shearlet transform and multi-scale morphological gradient with pulse-coupled neural network (NSST-MSMG-PCNN) [45].

figure c

In terms of the performance evaluation of proposed approach, eight indexes were utilized, including Mean, SD, Entropy, Sharpness, QAB/F [56], VIFF [19], FMI [18], and MI [17]. Tools for experimenting with all the aforementioned methods are online available. The proposed algorithm is implemented with the following parameters in EOA algorithm:

  • The size of population: n = 50.

  • β1 = 2, β2 = 1,

  • The generation probability: GP = 0.5, V = 1.

  • The maximum number of iterations: Itermax = 50

4.3 Image fusion evaluation

Firstly, the three optimal parameters (α1,α2, and α3) of the proposed model are illustrated in Table 1.

Table 1 Illustration of optimal parameters
Full size table

Secondly, from Table 2, the average and standard deviation indexes obtained from the EOA algorithm are the lowest. This indicates that the EOA algorithm is effective in the proposed approach. Furthermore, from Table 3, the p-values obtained from the Wilcoxon rank-sum test are less than 0.05. This shows that the results are statistically significant. Therefore, these explain why we have opted for the EOA algorithm in the proposed approach.

Table 2 Average and Standard deviation obtained from 30 different runs
Full size table
Table 3 P-values obtained from Wilcoxon rank-sum test
Full size table

Thirdly, from Table 4, experimental results show that the use of Prewitt compass operator in the proposed method is to bring the best results. This explains why the Prewitt compass operator is selected in Algorithm 2.

Table 4 Experimental results of the proposed method by using different compass operators
Full size table

Fourthly, from the Tables 5, 6, 7 and the Figs. 6, 7, 8, 9, 11, and 13, it is clear that the CSMCA [29] and CSR [28] methods create the composite images having low contrast, while the contrast of the NSCT [23], NSST- PA-PCNN [61], NSST-MSMG-CNN [45] approaches are greater than those of the above-mentioned approaches. In particular, the proposed method gives the best contrast. For instance, from Table 5, contrast index (SD) of the proposed method was highest, at 0.3780, while the figures for other methods (CSMCA [29], NSCT [23], CSR [28], NSST-PA-PCNN [61], and NSST-MSMG-CNN [45]) were lower, at 0.2093, 0.3194, 0.1847, 0.3411, and 0.1621, respectively. Similarly, the proposed method also gives the best intensity, entropy, and sharpness. This shows that the proposed approach not only enhances significantly the contrast but also improves the amount of information and sharpness of composite image (Figs. 10, 11, 12, 13, and 14).

Table 5 Experimental results of latest approaches and the proposed method on dataset (“Data-1”)
Full size table
Table 6 Experimental results of latest approaches and the proposed method on dataset (“Data-2”)
Full size table
Table 7 Experimental results of latest approaches and the proposed method on dataset (“Data-3”)
Full size table
Fig. 6
figure 6

Illustration of indexes from dataset (“Data-1”) by using different methods: CSMCA [29], NSCT [23], CSR [28], NSST-PAPCNN [61], NSST-MSMG-CNN [45], and the proposed method

Full size image
Fig. 7
figure 7

Illustration of indexes from dataset (“Data-2”) by using different methods: CSMCA [29], NSCT [23], CSR [28], NSST-PAPCNN [61], NSST-MSMG-CNN [45], and the proposed method

Full size image
Fig. 8
figure 8

Illustration of indexes from dataset (“Data-3”) by using different methods: CSMCA [29], NSCT [23], CSR [28], NSST-PAPCNN [61], NSST-MSMG-CNN [45], and the proposed method

Full size image
Fig. 9
figure 9

Illustration of the fused images from dataset (“Data-1”) by using different methods: CSMCA [29], NSCT [23], CSR [28], NSST-PAPCNN [61], NSST-MSMG-CNN [45], and the proposed method

Full size image
Fig. 10
figure 10

Illustration of frames extracted from the composite images in Fig. 9

Full size image
Fig. 11
figure 11

Illustration of the fused images from dataset (“Data-2”) by using different methods: CSMCA [29], NSCT [23], CSR [28], NSST-PAPCNN [61], NSST-MSMG-CNN [45], and the proposed method

Full size image
Fig. 12
figure 12

Illustration of frames extracted from the composite images in Fig. 11

Full size image
Fig. 13
figure 13

Illustration of the fused images from dataset (“Data-3”) by using different methods: CSMCA [29], NSCT [23], CSR [28], NSST-PAPCNN [61], NSST-MSMG-CNN [45], and the proposed method

Full size image
Fig. 14
figure 14

Illustration of frames extracted from the composite images in Fig. 13

Full size image

Finally, to assess the preservation of detailed information carried to the composite image from the input images, frames extracted from the fused images (Figs. 9, 11, 13) are shown in Figs. 10, 12, 14. It is easy to see that our method conserves detailed information from the input images, while remained methods produce much redundant information in the composite images. For instance, the QAB/F index of the proposed method is highest, at 0.8561 as shown in Table 7. Whereas, the figures for methods (CSMCA [29], NSCT [64], CSR [28], NSST-PA-PCNN [61], and NSST-MSMG-CNN [45]) were lower, at 0.7735, 0.7709, 0.7156, 0.7681, and 0.7112, respectively. Similarly, the results show that the values of VIFF [19], FMI [18], and MI [17] indexes are higher than those of the other approaches, this indicates that the proposed approach is highly effective in the fusion of medical images.

5 Conclusion

In this paper, two novel algorithms are proposed to address some limitations of current medical image fusion approaches. The proposed approach takes advantage of the TSD method to decompose the input medical images into two components, namely base and detail layers. The first algorithm allows the composite image to have good quality because the EOA algorithm is applied to find the optimal parameters for the fusion of the base layers. This overcomes the limitation of using a weighted average rule. The second algorithm (called SLE_PCO), based on the sum of local energy using the Prewitt compass operator, is implemented to fuse detail layers. This allows composite images to significantly preserve details, which overcomes the drawback of using fusion rules for detail layers.

The experiments were conducted on the dataset (Data-1, Data-2, and Data-3) to compare the result of the proposed method with those of other latest methods, such as CSMCA [29], NSCT [64], CSR [28], NSST-PA-PCNN [61], and NSST-MSMG-CNN [45]. Eight evaluation indexes, including Mean, SD, Entropy, Sharpness, QAB/F [56], VIFF [19], FMI [18], and MI [17], are utilized to assess the quality of the composite images. The experimental results show that our approach can significantly enhance the quality of the fusion image, and it is likely to ensure suitability for the human visual system. In addition, this proposed approach can also effectively conserve the edge information in the composite image.

In the future, there are some issues that we plan to address to improve the performance of current approaches. The first problem to be solved is to improve the quality of input images. This is because input medical images may be of low-quality, such as low-contrast, blur, and noise. This significantly affects the performance of image fusion methods. Therefore, image quality improvement plays a vital role in improving the quality of the fused images. For instance, the Non-Parametric Modified Histogram Equalization (NMHE) algorithm was used in the TSD-RS method [33] to enhance the low-contrast images. The second problem to be tackled is to preserve detailed information transferred to the fusion image. This can be explained by the fact that some existing decomposition methods can cause loss of information. Therefore, some novel image decomposition methods need to be applied to limit information loss in the fused images. For example, the Taylor Expansion algorithm was introduced to decompose input images into many intrinsic components in the Taylor expansion and convolutional sparse representation (TE-CSR) method [53].