Keywords

1 Introduction

With the development of modern science and technology, medical image fusion has been an important technology in the field of information fusion. Medial image fusion refers to the application of image fusion technology to medical images, that is, acquiring original images from different medical imaging equipments, using some special image processing algorithms, and then fusing a new medical image that contains more disease information of the target objects. There are many kinds of medical images, such as computer tomography (CT) images, Magnetic Resonance Imaging (MRI) images, and so on. Different kinds of medical images can show different information of human organs and sickness tissues, and each kind of images has their strong points and weak points.

CT images cannot display the parenchyma lesions clearly, but as they have high resolution ratio, they can display clearly bones which have large difference in density, which can help doctors to locate the lesions efficiently. The resolution ratio of the MRI images is not very high, and the MRI images can hardly image the bone tissues, but it can image the parenchyma clearly. If we synthesize the strong points of each kind of images, merging a new medical image by modern computer technology, then the medical value of each kind of images will be enhanced, thus assisting the doctors to make a right decision.

The commonly used medical image fusion algorithms at present include the weighted average, Laplace multi-scale pyramid, wavelet transform, and so on [1–8]. Among these algorithms, the weighted average algorithm is simplest, but it ignores the difference of the important messages between the target area from different sensor images, and the apparent jointing marks will be generated in the fusion image, thus not conducive to the following image processing [1]. Compared with the multi-resolution pyramid algorithm, image fusion based on wavelet transform can achieve better image fusion effects [2–7], but there are some defaults in the directional and anisotropy while applying the 2D separable wavelet basis, and only the horizontal, vertical, and diagonal directions can be used. So the wavelet transform cannot describe the image direction information perfectly. In 2002, a “really” 2D image representing algorithm contourlet transform was put forward by Do, M.N [9]. Comparing to the wavelet transform, the contourlet transform not only has the features of multi-scale and regional time–frequency, but also has the feature of multi-direction. But the image should be subsampled while transforming by this algorithm, so the contourlet transform does not have the feature of translation invariance, the pseudo-Gibbs phenomenon will occur by this algorithm, thus causing the image distortion. Aiming at these defects, nonsubsampled contourlet transform (NSCT) algorithm was put forward by Arthur L Cunha and Minh N. Do; this algorithm has the feature of translation invariance and enough redundance information; therefore, the direction information of the being fused image can be extracted effectively, and thereby getting better fusion effect [10, 11].

In this paper, aiming at the features of CT images and MRI images, NSCT algorithm is applied to fuse images and different fusion algorithms are applied to fuse the low-frequency subband and high-frequency subband separately. Experiment shows that by this algorithm, the useful information can be extracted and added to the fusion image effectively and better fusion result can be obtained.

2 NSCT Algorithm

The decomposition of the NSCT algorithm can be done by two steps. Firstly, the image is decomposited into band-pass subimages through multi-scale by nonsubsampled pyramid filter bank (NSPFB), and then the band-pass subimages are decomposited through directional nonsubsampled directional filter bank (NSDFB); thus, different scales and different directional subband images can be obtained. The nonsubsampled pyramid decomposition has the feature of multi-scale, and the NSDFB has the feature of multi-directional; the combination of these two steps makes the NSCT multi-scale and multi-directional. Besides, NSCT cancels the down-sample and the up-sample step while decompositing and reconstructing an image, only sampling to the filter, the NSCT has the feature of the translation invariance. The flow chart of the NSCT decomposition is shown in Fig. 1; including, Fig. 1a shows the NSCT filter group, and Fig. 1b shows the ideal frequency domain distribution of the origin image after the NSCT decomposition.

Fig. 1
figure 1

The flow chart of the NSCT: a filter group, b ideal frequency distribution

Full size image

After the image is decomposited by NSPFB, all the filters at the current level will be up-sampled to form as the filter group for the next level and each level satisfies the reconstruction condition. Let J be the decomposition scale number, then J + 1 band-pass subband images with the same size as the original image will be generated after NSPFB decomposition.

NSDFB module contains a dual-channel nonsubsampled filter group, which can decompose the band-pass images through directional. Using this filter group iteratively, up-sampling the filter at previous level to form as the filter group of current level, more precise directional decomposition will be obtained. Figure 2 shows a four-channel output of NSDFB structural diagram and frequency distribution diagram. Including, up-sampling matrix Q = [1 1; 1 −1] is applied to the filter; passing the first scallop filter, image is analyzed into horizontal and perpendicular two directions. Passing the second quadrant filter, image is analyzed into four pennant direction subbands as shown in Fig. 2b.

Fig. 2
figure 2

Four-channel output NSDFB a NSDFB (Nonsubsampled Directional Filter Banks) b frequency decomposition diagram

Full size image

Let l j be the directional decomposition number of the jth decomposition scale, then after NSCT decomposition, \( 1 + \sum\nolimits_{j = 1}^{j} {2^{{l_{j} }} } \) subband images with the same size as the origin image will be generated.

3 Research on the Fusion Rules about CT Images and MRI Images Based on NSCT Algorithm

Suppose that the CT and MRI images have been matched strictly before fusing. After the NSCT transform, each image will be decomposed into two parts. The low-frequency part contains the approximate information of the image, that is, the result of lowering the original image's resolution. The fusion image will have better visual effects if the low-frequency coefficients are fused efficiently. The high-frequency part represents the edge and the contour information correspondingly. According to the above characters, in this paper, aiming at the low-frequency part and high-frequency part of the transformed image, different fusion rules are applied separately to get the NSCT coefficients of the fusion image, and then the NSCT coefficients are inversely transformed to rebuild the fusion image.

3.1 Fusion of Low-Frequency Subband

After NSCT, the low-frequency part is the approximate description of the original image, including the original image’s average gray scale, texture information, etc. Fusion of the original image’s low-frequency part will affect the final fusion result directly. Traditional fusion algorithm for the low-frequency part includes weighted average algorithm, although this algorithm is simple, it will cause some problems such as the fusion image’s edge blur, which will interfere with the visual effect of the final fusion image. To some extent, the image’s local energy can represent the amount of the information in that area, so the larger the local energy is, the larger the amount of the information will be. Therefore, a fusion algorithm based on local energy is applied to fuse the low-frequency part of the image in this paper.

Let (i, j) be the coordinate for the pixel that is to be fused, \( W(m,n) = \frac{1}{16}\left[ {\begin{array}{*{20}c} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \\ \end{array} } \right] \) be the weighted coefficient matrix, and \( A_{J}^{0} (i,j) \), \( B_{J}^{0} (i,j) \) be the low-frequency subband pixel value at \( \left( {i,j} \right) \) of original image A and B separately, take \( \left( {i,j} \right) \) as center, the regional energy \( E_{A} (i,j) \) and \( E_{B} (i,j) \) in the \( 3 \times 3 \) regional window can be calculated as follows:

$$ E_{A} (i,j) = \sum\limits_{m = - 1}^{1} {\sum\limits_{n = - 1}^{1} {W(m,n)\left[ {A_{J}^{0} (i + m,j + n)} \right]^{2} } } $$
(1)
$$ E_{B} (i,j) = \sum\limits_{m = - 1}^{1} {\sum\limits_{n = - 1}^{1} {W(m,n)\left[ {B_{J}^{0} (i + m,j + n)} \right]^{2} } } $$
(2)

If there is not much difference between \( E_{A} (i,j) \) and \( E_{B} (i,j) \), we can conclude that the difference at \( \left( {i,j} \right) \) between the two images is not evidence, weighted algorithm can be applied to get the fusion result. Otherwise, the pixel value that has larger regional energy should be selected as the fusion result. Defining the normalized local energy \( M_{A} (i,j) \) and \( M_{B} (i,j) \) as follows:

$$ M_{A} (i,j) = \frac{{E_{A} (i,j)}}{{E_{A} (i,j) + E_{B} (i,j)}} $$
(3)
$$ M_{B} (i,j) = \frac{{E_{B} (i,j)}}{{E_{A} (i,j) + E_{B} (i,j)}} $$
(4)

Then, final fusion result can be calculated as:

$$F(i,j) = \left\{ {\begin{array}{*{20}l} {A_{J}^{0} (i,j){\text{ if}}\;{{M}}_{{A}} (i,j) \, > {{ T}}} \\ {B_{J}^{0} (i,j){\text{ if}}\;M_B (i,j) > T} \\ {M_{A} (i,j) \times A_{J}^{0} (i,j) + M_{B} (i,j) \times B_{J}^{0} (i,j)\quad{\text{otherwise}}} \\ \end{array} } \right. $$
(5)

where \( T \) is a threshold value, and in this paper, the value of \( T \) is 0.6, and \( F(i,j) \) is the fusion result.

3.2 Fusion of High-Frequency Subband

The high-frequency subband coefficients obtained from NSCT decomposition represent the geometrical features and edge features of two original images at different resolution ratios. Between the two images, the one that has larger high-frequency coefficient absolute value corresponds with some sudden change, such as the image’s edge, texture, and so on. In this paper, a fusion algorithm based on maximum regional deviation is applied to fuse the high-frequency subband coefficients. Let \( \left( {m,n} \right) \) be the coordinate for the pixel that is to be fused at the high-frequency subband, taking \( \left( {m,n} \right) \) as center, to the region with size \( M \times N \)(in this paper the size is \( 3 \times 3 \)), the regional deviation can be calculated as follows:

$$ V_{j}^{i} (m,n) = \frac{1}{M \times N}\sum\limits_{m = 1}^{M} {\sum\limits_{n = 1}^{N} {\left[ {f(m,n) - \overline{f} } \right]^{2} } } $$
(6)

where \( V_{j}^{i} (m,n) \) is the image’s regional deviation at the \( j \)th decomposition level and the \( i \)th decomposition direction. \( \overline{f} \) is the average deviation value of the region.

Let \( V_{A,J}^{i} \)and \( V_{B,J}^{i} \) be the high-frequency subimage’s regional deviation of the source image A and B separately at the \( j \)th decomposition level and the \( i \)th decomposition direction, then the normalized regional deviation can be defined as follows:

$$ \Updelta_{A,J}^{i} = \frac{{V_{A,J}^{i} }}{{V_{A,J}^{i} + V_{B,J}^{i} }} $$
(7)
$$ \Updelta_{B,J}^{i} = \frac{{V_{B,J}^{i} }}{{V_{A,J}^{i} + V_{B,J}^{i} }} $$
(8)

If there is not much difference between \( \Updelta_{A,J}^{i} \) and \( \Updelta_{B,J}^{i} \), we can conclude that the difference in the position \( \left( {i,j} \right) \) between the two images is not evidence; weighted average algorithm can be used to get the fusion coefficient, otherwise the value of the pixel that has the larger regional deviation should be applied as the fusion coefficient. The final fusion algorithm is shown as follows:

$$F(m,n) = \left\{ {\begin{array}{*{20}l} {A^{i} (m,n){\text{ if }}\Updelta_{A,J}^{i} - \Updelta_{B,J}^{i} \ge T} \\ {B^{i} (m,n){\text{ if }}\Updelta_{B,J}^{i} - \Updelta_{A,J}^{i} \ge T} \\ {\Updelta_{A,J}^{i} A^{i} (m,n) + \Updelta_{B,J}^{i} B^{i} (m,n)\quad{\text{otherwise}}} \\ \end{array} } \right.$$
(9)

where \( T \) is the threshold (in this paper, the value of \( T \) is 0.6) and \( F(m,n) \) is the fusion result.

4 Fusion Result and its Evaluation

Figure 3 shows the original CT image and MRI image applied in our experiment. The steps of our experiment are shown as follows:

Fig. 3
figure 3

original CT (a) and MRI (b) images in our experiments

Full size image
  1. (1)

    Firstly, the original CT and MRI images are transformed by NSCT algorithm separately to get the subband image correspondingly.

  2. (2)

    Secondly, each low-frequency subband and high-frequency subband are fused by the algorithm mentioned in this paper separately.

  3. (3)

    Finally, The NSCT inverse transform is done for the fusion coefficients to get the fused image.

To evaluate the effectiveness of the algorithm mentioned in this paper, we also fused the CT and MRI images by some custom fusion algorithms such as the weighted average, the Laplace Pyramid, and the NSCT custom algorithm. The fusion results are shown in Fig. 4, including, Fig. 4a is the result of weighted average algorithm, Fig. 4b is the result of the Laplace pyramid algorithm, Fig. 4c is the result of the NSCT custom algorithm, and Fig. 4d is the result of the fusion algorithm mentioned in this paper.

Fig. 4
figure 4

Experiment results a result of weighted average algorithm, b result of the Laplace pyramid algorithm, c result of the NSCT custom algorithm, d the result of the fusion algorithm mentioned in this paper

Full size image

From Fig. 4, we can see that the image obtained from weighted average algorithm is lack of resolution, in which the edge is blurry, and some detail edge information is hard to be recognized. The result of the fusion image obtained from the Laplace pyramid algorithm is better than the result of the image obtained from the weighted average algorithm, but some detail information of the fusion image is still not very clear. The resolution ratio of the image obtained from NSCT custom algorithm has raised a lot, but the detail contrasts still remain some defaults in the image. The fusion result of the algorithm mentioned in this paper is better than others, in the fusion image, the detail contours are clearly visible, and the resolution ratio has raised evidently.

To evaluate the fusion effect objectively, we took information entropy, mutual information and average gradient as evaluation indexes, and the weighted average, the Laplace Pyramid, the custom NSCT algorithm and the algorithm mentioned in this paper are evaluated by these indexes. The final results are shown in Table 1. From Table 1, we can see that all the indexes obtained from the algorithm mentioned in this paper have raised evidently, which sufficiently proves the effectiveness of the algorithm mentioned in this paper.

Table 1 The objective evaluation to the image fusion result of the CT image and MRI image
Full size table

5 Conclusions

Aiming at the features of the CT and MRI images, on the basis of NSCT algorithm, NSCT fusion algorithm is studied in this paper, and an image fusion algorithm on which low-frequency coefficients are fused on the basis of maximum local energy and high-frequency coefficients are fused on the basis of maximum region deviation is proposed. Experiments show that in this algorithm the detail information of the CT and MRI images can add to the fusion image accurately, and comparing with the custom image fusion algorithm such as the weighted average algorithm, Laplace pyramid algorithm, and NSCT custom algorithm, better fusion results can be obtained at each fusion indexes. So the algorithm mentioned in this paper is an efficient image fusion algorithm.