Keywords

1 Introduction

Multispectral images (MSIs) capture more information about the scene as compared to standard color images. Therefore MSIs are widely used in different areas like medical imaging, food industry, remote sensing, or identifying materials [8,9,10]. In the past years, few single-sensor-based multispectral imaging systems [3, 11, 17, 22] have been proposed based on MSFA, similar to the standard consumer digital camera, which is based on CFA. MSFA has more than three spectral bands compared to CFA and empowers us to develop low-cost and portable multispectral cameras. MSFA based multispectral camera captures only one spectral band information as each pixel location depending on the filter element in the MSFA covering the pixel location. This captured image where only one spectral band information is available at each pixel location is called a mosaicked image (raw image). The process of estimating missing spectral band information at each pixel location in the mosaicked images is called multispectral image demosaicking. The quality of the MSI generated depends on the efficiency of the multispectral demosaicking algorithm. However, multispectral image demosaicking is more challenging than color image demosaicking due to the highly sparse sampling of the spectral band in the MSFA.

As the applications of MSIs are diverse, different multispectral imaging systems [15, 17, 21] are introduced with varying numbers of spectral bands. So there is a need for an efficient generic multispectral image demosaicking method that can be used to generate varying band-size MSI depending on the applications. Many generic multispectral image demosaicking algorithms [1, 2, 13] are proposed, which fail to generate good quality MSIs. Here, we propose a generic multispectral image demosaicking algorithm based on simple non-redundant MSFAs. It uses both spatial and spectral correlation present in the mosaicked image to generate the complete multispectral image. First, we generate a pseudo panchromatic image (PPI) from the mosaicked image. Later, we use the PPI to generate a multispectral image by utilizing spectral correlation between PPI and each undersampled band in the mosaicked image. The PPI has a stronger correlation with each spectral band than bands considered pairwise [15].

The main contributions of our work are as follows: (1) We propose a generic multispectral image demosaicking based on the PPI as it is strongly correlated with each spectral band. (2) We develop a new metric, ANMPSNR, which facilitates one to quickly estimate how better a particular image can be reconstructed from a sparse image acquired using a single sensor camera and helps easily compare the relative performance of different demosaicking algorithms. (3) We also highlight the problem with simple non-redundant MSFAs when used for generic multispectral demosaicking algorithms.

The remaining paper is organized as follows. Section 2 reviews the existing multispectral image demosaicking algorithms. In Sect. 3 and Sect. 4, we describe our proposed algorithm and proposed metric, respectively. Section 5 presents our experimental results on two benchmark multispectral image datasets, and in Sect. 6, we present the conclusion and future work.

2 Related Work

In past years many demosaicking algorithms have been proposed for the different number of bands multispectral images. In this section, we discuss these different demosaicking algorithms [2, 4,5,6, 12, 14,15,16, 18, 19, 21]and their related MSFA patterns.

Miao and Qi proposed a first methodical generic MSFA formation method [12] based on a binary tree that can be used to create any number of band MSFA patterns. Miao et al. also proposed a binary tree based edge sensing (BTES) generic demosaicking algorithm [13]. BTES used edge correlation to estimate missing pixel values for each spectral band. BTES only considers spatial correlation to estimate missing pixel values. Therefore, BTES performs poorly, especially on the higher number of band images.

Brauers and Aach proposed a multispectral demosaicking image algorithm [2] which used 6-band simple non-redundant MSFA pattern arranged in \(2 \times 3\) grid. [2] applied a low-pass filter to each spectral band to estimate the missing pixel values. Further, the quality of generated multispectral images is improved using the inter-band differences method. The simple non-redundant MSFA design patterns and corresponding low-pass filters were later generalized by [6]. Mizutani et al. [16] extended [2] to develop a scheme for a 16-band MSFA image by iterating the interpolation process multiple times. [2] algorithm can be generalized to varying band size multispectral images using filters designed by [6]. In [1], Aggarwal and Majumdar proposed a generic demosaicking algorithm based on learning interpolation parameters based on uniform MSFA patterns. This method requires original multispectral images for learning interpolation parameters, which are practically impossible to obtain in real-time.

Monno et al. [17] proposed a 5-band MSFA pattern based on  [12] and a demosaicking algorithm. [17] used the concept of guide image (estimated from G-band), and later guide image is used as a reference image to interpolate remaining under-sampled bands. [17] is restrained to the MSFA patterns having probability of appearance (PoA) of G-band equals to 0.5 in the MSFA pattern, making other bands rigorously under sampled in higher band multispectral images.

Mihoubi et al. [14] proposed a multispectral demosaicking algorithm constrained to square-shaped simple MSFA patterns. Further, in [15], authors had improved their previous work by proposing a new estimation of intensity image. In [21], the authors proposed a 9-band multispectral imaging system based on binary tree based MSFA pattern having PoA of the middle band equals 0.5. The authors first estimate the middle band using image gradient in the demosaicking method and later used this estimated band as a guide image to interpolate other bands.

Rathi et al. [19] proposed a generic multispectral demosaicking approach based on spectral correlation present in the mosaicked image. The proposed approach first applied the bilinear interpolation and later used the spectral correlation differences progressively. However, Some deep learning-based image demosaicking algorithms [7, 20] also have been recently suggested. But these algorithms, including [1] require the complete information of multispectral images (not just mosaicked images) for training their model parameters. But, these images will not be available in real practice for MSFA based multispectral camera devices intended to be developed.

3 Proposed Multispectral Demosaicking Algorithm

This paper proposes a generic multispectral image demosaicking algorithm based on spatial and spectral correlation present in the mosaicked image. Our proposed algorithm uses simple non-redundant MSFAs where each band has an equal probability of appearance. Here, we have used the concept of the PPI to estimate the missing pixel values of each spectral band. PPI has a stronger spectral correlation with each band compared to the band considered pair-wise. The PPI is defined at each pixel location as the average over all the spectral bands of a multispectral image.

Fig. 1.
figure 1

Spatial filters used to generate the PPI from mosaicked image.

Full size image
$$\begin{aligned} I^{PPI}= \frac{1}{K}\sum _{k=1}^K{I^k} \end{aligned}$$
(1)

where, \(I^k\) is the \(k^{th}\) spectral band of multispectral image \(\mathbf {I}\) of size \(M \times N\times K\). The size of the PPI is \(M \times N\). However, to define the PPI from the mosaicked image, we use spatial filters H defined for each band-size mosaicked image as shown in Fig. 1. We calculate the PPI from mosaicked image for any k-band mosaicked image \(I_{MSFA}\) using corresponding filter H.

$$\begin{aligned} I^{PPI}= I_{MSFA}*H \end{aligned}$$
(2)

This spatial filter H is defined to consider at least one instance of each spectral band in the mosaicked image when centered at any pixel location. And the size of H should be the smallest and odd number size (e.g., \(3 \times 3\), \(5 \times 5\), \(7 \times 7\), etc.). Each element of H (e.g., take pixel location (i, j) in H w.r.t. to its center) is set to \(\frac{1}{x}\), where x is the number of times when the spectral band at the location (i, j) appears within the window of size of H. And finally, filter H is normalized to in order that all elements of H sum up to one.

The first part of our proposed algorithm estimates the PPI from the mosaicked image and uses it to generate the multispectral image. And in the second part, our algorithm estimates the PPI from the generated multispectral image and uses it to generate the improved version of demosaicked multispectral image. The second part of our proposed algorithm is iterated multiple times further to improve the quality of the generated multispectral image. Our proposed algorithm performs the following steps.

  1. 1.

    Generate the PPI \(I^{PPI}\) using Eq. 2.

  2. 2.

    For each band k, determine the sparse band difference \(\tilde{\mathcal {D}}^{k}\) between PPI (\(I^{PPI}\)) and band k at the locations of band k in the mosaicked image.

    $$\begin{aligned} \tilde{I}^{k}={I}_{MSFA} \odot m^{k} \end{aligned}$$
    (3)
    $$\begin{aligned} \tilde{\mathcal {D}}^{k}= \tilde{I}^{k}- I^{PPI} \odot m^{k} \end{aligned}$$
    (4)

    where, The binary mask \(m^k\) has value 1 only at locations where \(k^{th}\) band’s original values are present in the mosaicked image.

  3. 3.

    Now compute the fully-defined difference \(\hat{\mathcal {D}}^{k}\) for each band k using Weighted Bilinear (WB) interpolation [6].

  4. 4.

    Now estimate each band k as follow:

    $$\begin{aligned} \hat{I}^{k}=I^{PPI}+\hat{\mathcal {D}}^{k} \end{aligned}$$
    (5)
  5. 5.

    Repeat the following steps 6 and 7 a number of times \(\mathcal {T}\) to further improve the quality of generated image in step 4.

  6. 6.

    Compute new estimate of the PPI using Eq. 1.

  7. 7.

    Generate multispectral images using steps 2, 3 and 4.

Now, all K bands are fully-defined and together form the complete multispectral image \(\hat{\mathbf {I}}\).

4 New Proposed Metric: ANMPSNR

Peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) metrics are the most often used performance metrics for comparing multispectral image demosaicking algorithms, and these are computed using the original images and the images reconstructed using the given algorithm(s). However, PSNR has a wide range and is image content dependent, and therefore the relative comparison of different methods becomes challenging. It lacks in giving quick estimation about how better can a particular image be reconstructed from a sparse image acquired using a single sensor camera and some demosaicking algorithm. Here, we propose a new metric ANMPSNR (Average Normalized Multispectral PSNR) which facilitates one to perform such estimation faster and also helps in easily comparing the relative performance of different demosaicking algorithms. For some particular K band n multispectral images and m different demosaicking algorithms considered for comparison, let \(P^K\) be a 2-D matrix of size \(n\times m\) s.t. \(P^K(i,algo)\) denotes the PSNR value obtained for the \(i^{th}\) image using demosaicking algorithm algo for the reconstruction. We define ANMPSNR as follows:

$$\begin{aligned} ANMPSNR_{algo}^K = \left( \frac{1}{n}\right) \sum _{i=1}^{n} \left( \frac{P^K(i,algo)}{\textit{max} \left( P^K(i,:)\right) } \right) \end{aligned}$$
(6)

For every image, it considers the relative performance, in terms of PSNR, in reference to the best performing algorithm for that image. It may also be noted that ANMPSNR is always positive, and its maximum value will be 1.

5 Experimental Results and Discussions

In this section, we evaluate the performance of the proposed algorithm and compare it with different generic multispectral image demosaicking algorithms on two benchmarks multispectral image datasets: TokyoTech [17], and Cave [23]. The TokyoTech dataset has 30 images of 31-band captured in the range from 420 nm to 720 nm at the equal spectral gap of 10 nm. The Cave dataset has 31 images of 31-band captured in the range of 400 nm to 700 nm at the equal spectral gap of 10 nm.

Fig. 2.
figure 2

Performance of different demosaicking algorithms based on simple non-redundant MSFAs. (a, b) Average PSNR and SSIM values, respectively on the TokyoTech dataset. (c, d) Average PSNR and SSIM values, respectively on the Cave dataset.

Full size image

We evaluate different algorithms on 5-band to 15-band multispectral images. To generate the ground truth K-band multispectral image, we take K-bands at equal spectral gap starting from the first spectral band from the 31-band original image. Then we convert the K-band multispectral image to the mosaicked image based on the MSFA pattern corresponding to band K. We use the demosaicking algorithm to generate the multispectral image from the mosaicked image. To evaluate the efficacy of the demosaicking algorithm by comparing the demosaicked multispectral images with corresponding ground truth multispectral images on different image quality parameters. The value of \(\mathcal {T}\) is chosen experimentally. We try different values of \(\mathcal {T}\) in range 1 to 20 for different band size images and select the value of \(\mathcal {T}\), which results in maximum PSNR value. We compare our proposed algorithm with the existing generic multispectral demosaicking algorithms like WB [6], SD [2], BTES [13], LMSD [1], ISD [16], PBSD [19], and PCBSD [18].

Fig. 3.
figure 3

Simple non-redundant MSFA patterns. (a) Non-compact shaped MSFA patterns. (b) Compact shaped MSFA patterns.

Full size image
Table 1. Average PSNR values of different generic multispectral image demosaicking algorithms on both the datasets. Note: We consider only those bands where simple non-redundant MSFA patterns have compact shaped for a fair comparison.
Full size table

In Fig. 2, we compare different generic demosaicking algorithms which use simple non-redundant MSFAs to capture the mosaicked image. Clearly, our proposed algorithm performs better than all other algorithms in terms of PSNR and SSIM values. Overall, our algorithm shows the improvement of 1.13 dB, and 0.7 dB in the PSNR values average over 5-band to 15-band multispectral images to the second-best performing algorithm on the TokyoTech and the Cave datasets, respectively. We notice the interesting behavior of demosaicking algorithms based on simple non-redundant MSFAs. They perform better on the 6, 8, 9, 12, and 15 bands size images than 5, 7, 10, 11, 13, and 14 bands size images. This is due to the compact nature of simple non-redundant MSFA patterns on the 6, 8, 9, 12, and 15 bands shown in Fig. 3. So for a fair comparison of the efficacy of our proposed algorithm, we compare it with all other generic multispectral demosaicking algorithms only on bands where simple non-redundant MSFA patterns have a compact shape. In Table 1, we show the comparison of our algorithm with all other generic multispectral image demosaicking algorithms in terms of PSNR value. Our algorithm performs better on these bands and shows an improvement of 0.83 dB and 0.33 dB on average over these bands in the PSNR value than PCBSD on the TokyoTech and the Cave datasets, respectively.

Table 2. Comparison of different multispectral image demosaicking algorithms based on ANMPSNR on the Tokyotech and the Cave dataset.
Full size table
Fig. 4.
figure 4

Average PSNR values of each spectral band of different multispectral image demosaicking algorithms: (a, b) on 12-band multispectral images on the TokyoTech and Cave datasets, respectively. (c) on 9-band multispectral images. (d) on 6-band multispectral images.

Full size image
Fig. 5.
figure 5

Visual comparison of sRGB images generated from the 6 bands MSIs.

Full size image
Table 3. Comparison of PSNR(dB) values for different multispectral image demosaicking methods on 30 images of the TokyoTech dataset.
Full size table
Table 4. Comparison of PSNR(dB) values for different multispectral image demosaicking methods on 31 images of the Cave dataset.
Full size table

In Table 2, we show the performance of these algorithms on the new metric ANMPSNR, and our algorithm shows better performance than other algorithms. Further, it can be noted that the ANMPSNR metric makes it easy and faster to compare different methods relatively. Tables 3 and 4 show the image-wise performance of different algorithms on the multispectral images of 6 and 12 bands from the TokyoTech and the Cave datasets, respectively. Figure 4 shows the PSNR value for each band on 6, 9, and 12-band multispectral images. Our proposed algorithm show almost consistent performance on all bands, whereas other algorithms like LMSD, BTES PBSD, and PCBSD show varying performance on different bands.

Figures 5 and 6 show the visual comparison of the sRGB images generated by the different multispectral image demosaicking algorithms for 6 and 12 bands multispectral images. Our proposed algorithm reproduces the sRGB images more accurately than the other MSID algorithms with fewer artifacts. PBSD and PCBSD produce zipper artifacts around the edges, whereas WB and BTES produce blurry images.

Fig. 6.
figure 6

Visual comparison of sRGB images generated from the 12 bands MSIs.

Full size image

6 Conclusion and Future Work

We proposed a generic multispectral image demosaicking algorithm based on the PPI, which uses both spatial and spectral correlation present in the mosaicked image. The PPI is considered to have a stronger correlation with each band than bands are examined pair-wise. We show that compact-shaped simple non-redundant MSFA patterns achieve better demosaicking performance than non-compact-shaped simple non-redundant MSFA patterns. We also proposed a new performance evaluation metric for an easy and faster relative comparison of MSID methods. We evaluated the performance of the proposed MSID algorithm by comparing it with other existing generic MSID algorithms on multiple multispectral image datasets. Considering different performance metrics and also the visual assessment, the proposed algorithm’s performance is consistently observed better than the existing generic multispectral image demosaicking algorithm on simple compact-shaped non-redundant MSFAs. In the future, we plan to extend it to binary tree based patterns [12] which are considered compact as compared to simple non-redundant MSFA patterns.