1 Introduction

Multi-projector display systems can achieve high-resolution and high-brightness display with a set of low-cost projectors. Advanced multi-projector display techniques are capable to generate realistic visualization display using up to hundreds of projectors. By fully utilizing the diversity of display surfaces, e.g. curved surfaces and convex/concave surface, special artistic effects can be achieved, such as the immersive virtual reality (VR) effects.

Automatic color correction (ACC) is a challenging problem for multi-projector display systems, since the color differences, various projector properties and surface material differences create a large number of variables in computation [10, 22]. Traditional methods usually make several assumptions on the optical property of display surfaces. For example, intensity blending method assumes that the display surfaces are Lambertian; and the distances from the display surface to each projector are equal [16, 22]. It is however common that a practical multi-projector displaying system does not meet such assumptions.

In this study, we tackle the traditional ACC problem with a hierarchical framework consisting of a simple color correction approach and an advanced color correction approach. The choices of the approaches are made according to the specific projection conditions, avoiding unnecessary computational costs. The aim of this study is to achieve ACC with adaptable parameter estimations for different multi-projector systems. The parameters are able to converge globally. An iterative optimization scheme is used to adaptively estimate the model parameters. All projection properties are considered for modeling real-world situations. Experimental results show that our method outperforms traditional methods. The main contributions of this paper include:

  1. 1)

    A practical hierarchical calibration framework is proposed for various projection conditions. A selection scheme, based on peak signal to noise ratio (PSNR), is used to evaluate the output quality and select the ACC approaches.

  2. 2)

    A simple color correction algorithm is proposed based on a simple color model. The simple color model is a displacement dependent experimental model, which matches the dynamic response curves of each projector. The simple color model is used to implement an absolute color correction solution for the multi-projector display system.

  3. 3)

    An advanced color model is designed to calibrate projectors that have large property differences. A pre-processing step is designed to remove isolated abnormal sampling points and generate a wider gray level CAR. The output of the advanced color correction method is visually seamless and of higher quality than existing methods.

2 Related work

The geometry calibration is one of the major steps for ACC, which can be considered as a preliminary step of this study. Wang et al. [22] used horizontal and vertical lines to achieve feature point recognition with sub-pixel precision. Yang et al. [25] reviewed several existing techniques to build the multi-projector calibration systems using cameras. Bhasker and Majumder [3] presented a closed-form model to connect the projectors with possibly significant distortions for the lenses. Park et al. [12] presented an active calibration technique for camera-projector systems based on planar homography. Sergio and Joaquim [21] proposed a calibration method for the extrinsic and intrinsic parameters of the projector-camera pair. Wang et al. [23] employ the horizontal and vertical feature strips to calculate the corresponding points with sub-pixel decision between the input image space and the projection image space. Chen et al. [4] proposed a geometry calibration method based on the feedbacks from cameras. Babar et al. [2] introduced a FPGA (Field-Programmable Gate Array) based geometry calibration method that provides real-time correction of projected images. Park et al. [13] presented an automatic multi-projector calibration approach using a simple handheld camera.

In 1998, Raskar et al. [16] first proposed a color correction solution for multi-projector display system using image warping and intensity blending. The approach was so simple that they only normalize the intensities of the overlapping regions. Majumder and Stevens [10] deeply analyzed Raskar et al.’s work and proposed an extension work based on [16]. More recently, a series of methods were proposed to improve the robustness, accuracy, and efficiency for ACC of multi-projector systems [1, 19, 20]. Nevertheless, it was still difficult to obtain a universal high-precision color model for multi-projector systems due to the complicated optical properties of display surfaces, various projection characteristics of different projectors and other environmental factors. In Garcia-Dorado and Cooperstock’s work [7], a fully automated multi-projector calibration method was presented, which requires only one camera and several paper markers to delimit the boundaries of the desired display region. Majumder and Stevens [11] illustrated that absolute photometric uniformity was not always necessary to achieve photometric seamless display based on human perceptions. They addressed the photometric variation problem from a perceptual view point and generated seamless images with high dynamic ranges. However, the method was too complicated for implementation. Han et al. [8] proposed a multi-projector projection solution for seamless tiled display system. Their solution is built based on hardware image processors, such as FPGA. Pranav et al. [15] proposed a generalized color model of multi-projector ACC. The focus of their study is on geometry calibration. The color correction is done by adjusting the projection intensities in overlapping regions. The proposed method was hard to be generalized in real-world applications due to the high computation complexity. Sajadi and Majumder [18] presented a constrained gamut morphing algorithm that removed all the variations across the projectors, resulting in color seamless tiled multi-projector displays. But the method was required to adjust the intensities of light of each projector precisely to achieve smooth morphing. Moreover, the time complexity of the above method is still too high.

Recently, more ACC solutions are proposed [17, 24, 26, 27]. Zoido et al. [27] proposed a series of optimization techniques for large projection displays, which can be implemented with inexpensive hardware. Saeed et al. [17] proposed a unified scheme that handles geometric and photometric transformations using a single, combined Look-up Table. But it usually requires multiple cameras, which is inconvenient in practical usage. Wang et al. [24] proposed a software-hardware cooperative method for seamless multi-projector system with an optimization algorithm. Zhong et al. [26] presented a weighted average optimization algorithm for the color correction of multi-projector system. Full consideration of the depth reconstruction ability of the display system must be taken to reconstruct the 3D scenes.

In this study, we employ both simple and sophisticated color correction methods from the literature, and propose an ACC hierarchical framework that contains two sub-methods adapting various conditions. The experimental results show that the proposed hierarchical framework outperforms traditional ACC methods.

3 Overview of the proposed solution

Given a multi-projector display system (Fig. 1), our objective is to establish the color correspondence mapping between the input image space and the projected output image space for each pixel. There are generally three sources of color difference: 1) color variation within each projector; 2) color variation across different projectors; 3) nonlinear color additions in the overlapping regions between adjacent projectors. The color mapping between the input and output of the projector is possibly nonlinear, which can further increase the difficulty for color correction.

Fig. 1
figure 1

Experimental bench with 3 × 5 projectors

Full size image

The geometry calibration is a necessary step to obtain the geometric relationship between the images before and after projection. The horizontal and vertical feature strips are projected and automatically captured by a hanging pan-tilt camera to calculate the transmission matrices for each projector. We employ the four-step method in [9] to calculate the transmission matrices. We optimize the method in [11] by introducing a boundary-adaptive pattern that has an uneven division of the projected image (Fig. 2). Utilizing the boundary-adaptive pattern, the boundary regions are divided into denser grids than the central regions, as shown in Fig. 3, which can improve the correction accuracy for the overlapping region among different projectors.

Fig. 2
figure 2

Boundary-adaptive pattern for geometry calibration

Full size image
Fig. 3
figure 3

Uneven subdivision of each projected image

Full size image

In the color correction stage, we employ a practical hierarchical ACC framework for various working environments. Two color correction methods are proposed, namely, the simple color correction and the advanced color correction. The peak signal to noise ratio (PSNR) is used to evaluate the projection image quality and select the most suitable method. The PSNR is defined as

$$ PSNR=10\times { \log}_{10}\left(\raisebox{1ex}{${\left({2}^n-1\right)}^2$}\!\left/ \!\raisebox{-1ex}{$ MSE$}\right.\right) $$
(1)
$$ MSE=\frac{1}{H\times W}\sum_{i=1}^H\sum_{j=1}^W{\left(\mathbf{X}\left( i, j\right)-\mathbf{Y}\Big( i, j\Big)\right)}^2 $$
(2)

where n is the number of bits per pixel; X and Y are the images before and after projection; MSE is the mean square error between X and Y; H and W are the height and width of X and Y. If the PSNR is less than a pre-defined threshold value, the simple color correction method is used; the advanced color correction method is used otherwise (Fig. 4). The simple color correction method uses an experimental parameter model to build the transportation function for each projector, and achieves absolute color consistency by brightness compensation. The advanced color correction method builds the advanced color model by overall considering all properties of projectors, display surface and the projection distances.

Fig. 4
figure 4

Hierarchical framework of the proposed color correction solution

Full size image

4 Simple color correction

Simple color correction method is designed for situations while the projectors have similar properties, such as similar ages, small color differences and etc. The homography matrix generated by the uneven grids techniques distorts the input images, which actually improves the display quality of the whole system. However, the intra-projector and inter-projector colorific inconsistencies, as well as the artificial high brightness in the overlap regions, remain as difficulties for ACC. In this section, we present a simple color correction method to solve the above problems.

4.1 Simple color model

Projector color modeling is designed to model the dynamic response function that maps the input pixel values to the corresponding output radiant intensities on the display surfaces. If the color model of each projector is known, the projecting images can be calibrated with pixel mapping and color compensation. Otherwise, the translation and distortion effects of the projectors will produce uncertain pixel intensities on the display surface, and increase the difficulty for accurate compensation.

We calculate the dynamic response curve of each projector by the displacement dependent experimental model using Eq. (3):

$$ B\left( x, y,{v}_i\right)={k}_i\times \arctan \left({c}_i\times {v}_i+{l}_i\right)+{m}_i\sqrt{{\left( x- W/2\right)}^2+{\left( y- H/2\right)}^2}+{n}_i, $$
(3)

where I denotes the current color channel; (x, y) is the location of the sampling point with a left-top origin of coordinates; v i is the input value of the current color channel; W and H are the width and height of the current image respectively; c i is the contribution factor of the current color channel; k i is the energetic coefficient of the current color channel; l i is the configuration coefficient, involving brightness, contrast and etc.; n i is the diffusion coefficient of the current color channel of the display surface; and m i is the deviation ratio. The color of each sampling point is stored to estimate the parameters in the experimental model, using the high dynamic range method [6].

4.2 Parameter estimation

The parameter estimation of the simple color model starts from a set of non-linear equations. We take the red channel as an example:

$$ \left\{\begin{array}{l}{f}_{R1}\left({k}_R,{r}_R,{b}_R,{v}_R\right)-{k}_R \arctan \left({r}_R\cdot {x}_{R1}+{b}_R\right)+{v}_R=0\\ {}{f}_{R2}\left({k}_R,{r}_R,{b}_R,{v}_R\right)-{k}_R \arctan \left({r}_R\cdot {x}_{R2}+{b}_R\right)+{v}_R=0\\ {}\dots \\ {}{f}_{R n}\left({k}_R,{r}_R,{b}_R,{v}_R\right)-{k}_R \arctan \left({r}_R\cdot {x}_{R3}+{b}_R\right)+{v}_R=0\end{array}\right. $$
(4)

It is obvious that Eq. 4 is over-constraint, since the number of experiment data set is usually greater than the unknown coefficients. A reasonable set of parameters can be obtained by applying the least square method to minimize the energy function:

$$ {E}_R(X)=\sum_{i=1}^n{\left|{f}_{R i}\left({k}_R,{r}_R,{b}_R,{v}_R\right)-\left[{k}_R \arctan \left({r}_R\cdot {x}_{R i}+{b}_R\right)+{v}_R\right]\right|}^2 $$
(5)

The detailed parameter estimation algorithm for the simple color model is described as follows. Only the algorithm for the red channel is shown. The solutions for the other two channels (blue and green) are similar.

figure c

4.3 Absolute color correction

After calculating the color model parameters of each projector, the simple color correction algorithm is proposed as follows:

figure d

5 Advanced color correction

In conditions that the projector properties, such as the bulb age, color differences and etc., differ drastically, an advanced color correction algorithm is proposed. In this section, we present the proposed advanced color correction method using an advanced color model.

5.1 Advanced color model

In the advanced color correction method, a generalized color model is established for each projector with overall consideration of the properties of projectors, display surface and the projecting distances. A pre-processing step is applied to eliminate the isolated abnormal sampling points and obtain the visually seamless results.

Suppose {(R i , f(R i ))| i = 0, 1,  … , p + q}is a set of points while the given projector is configured with RGB value (R i, 0, 0); and f (R i) is the corresponding output brightness. The advanced color model is defined by:

$$ {C}_{p, q(R)}=\frac{M_p(R)}{N_q(R)}=\frac{a_0+{a}_1 R+\dots +{a}_p{R}^p}{b_0+{b}_1 R+\dots +{b}_q{R}^q} $$
(8)

with the interpolation condition defined as follows:

$$ {C}_{p, q\left({R}_i\right)}=\frac{M_p\left({R}_i\right)}{N_q\left({R}_i\right)}= f\left({R}_i\right) $$
(9)

where C p , q(R) is the interpolation function; and f (R) is the advanced color model. The interpolation function can be solved by the optimal rational approaching algorithm [3].

5.2 Parameters estimation

The solution of Eq. (8) does not always exist [5]. However, there exists an approximated optimal solution w p , q (R), if R i and f(R i ) take values in [0,255]. The error function is defined as:

$$ E(R)=\underset{0\le R\le 255}{ \max}\left| f(R)-{C}_{p, q}(R)\right| $$
(10)

The problem of parameter estimation for the advanced color model can be solved by finding two sets: A k  = {a i | i = 0, 1,  … , p},B k  = {b j | j = 0, 1,  … , q}, with the condition:

$$ {E}_{A_k,{B}_k}(R)=\underset{0\le R\le 255}{ \min}\left| E(R)\cdot W(R)\right| $$
(11)

where W(R)is the weighting function, and \( {E}_{A_k,{B}_k}(R) \)is the weighted error function. Based on Eqs. 10 and 11, the optimal approximation solution w p , q (R)can be calculated recursively by algorithm 3.

figure e

5.3 Advanced color correction

In this subsection, we show the details of the advanced color correction algorithm in Algorithm 4. A novel pre-processing step (Step 4 in Algorithm 4) is proposed to eliminate the isolated abnormal sampling points and obtain a wider gray level CAR for better quality outputs.

figure f

6 Experimental results

In this section, we test the proposed solution on two experimental platforms. One is a front projection multi-projector system of resolution 3000 × 768 consisting of a 1 × 3 array of three projectors, which is denoted as platform A. The other platform is a rear projection multi-projector system of resolution 5000 × 2250 consisting of a3 × 5 array of 15 projectors, which is denoted as platform B. The Projector model is EPSON EB-CS500Wi. The front projection system uses a white plastic soft curtain; and the rear system uses a white glass fiber curtain. An open source library named CxImage is utilized to the process the images [14].

6.1 Evaluation criteria

To evaluate the performance of the proposed ACC framework, we employ the mean structural similarity (MSSIM) and PSNR to evaluate the performance of our method. MSSIM is a comprehensive image quality evaluation index, which measures the similarity of two images from three aspects: brightness, contrast and structure. The range of MSSIM is [0, 1]; and greater values represent better color correction results. The MSSIM is defined as:

$$ \overline{S}=\frac{1}{N}\sum_{i=1}^N S\left({I}_k,{O}_k\right) $$
(14)

where N is the number of projectors;I k and O k are the input image and output image of the kth projector, respectively. S(I k , O k ) is the structural similarity between I k and O k , which is defined as:

$$ S\left( I, O\right)= b\left( I, O\right)\bullet c\left( I, O\right)\bullet s\left( I, O\right) $$
(15)
$$ b\left( I, O\right)=\frac{2{\mu}_I{\mu}_O+ C1}{\mu_I^2+{\mu}_O^2+ C1} $$
(16)
$$ c\left( I, O\right)=\frac{2{\sigma}_I{\sigma}_O+ C2}{\sigma_I^2+{\sigma}_O^2+ C2} $$
(17)
$$ s\left( I, O\right)=\frac{\sigma_{I O}+ C3}{\sigma_I{\sigma}_O+ C3} $$
(18)

Where I and O are input and output images of each projector, respectively; b(I, O),c(I, O)and s(I, O) are the similarity between IandO in brightness, contrast and structure, respectively; μ I andμ O are the mean values of IandO, respectively; σ I andσ O are the variance of IandO, respectively;σ IO is the covariance between IandO; C1, C2 and C3 are all constants.

6.2 Experiment 1

In experiment 1, we perform the experiment on platform A, and compare the simple color correction method with the traditional absolute CAR method. In platform A, all three projectors are in close ages. Therefore, the brightness values of their output images are similar. Under this condition, the simple color correction method is selected by the hierarchical framework.

Figure 5 shows the experimental result comparison between the simple color correction method and the absolute CAR method. No obvious difference can be found by human vision. However, it is noted that the absolute CAR method involves a more costly algorithm compared to the simple color correction method. The result is valid while the display surface has high diffuse reflection coefficients with all similar property projectors.

Fig. 5
figure 5

Different simple color correction results

Full size image

Figure 6 illustrates a MSSIM curve comparison between the proposed simple ACC algorithm and the absolute CAR method. The X-axis represents different pixel values of the input images. The collection of data in Fig. 6 includes four steps:

  1. 1)

    apply black images (all pixels with zero values) to all projectors.

  2. 2)

    use the two color correction methods to calibrate the output.

  3. 3)

    calculate the MSSIM values of each method comparing the input images and the corresponding calibrated images.

  4. 4)

    increase the pixel values of input images by one step value. If the new pixel value is less than 255, go to step 2.

    Fig. 6
    figure 6

    MSSIM comparison for experiment 1

    Full size image

According to Fig. 6, the simple color correction achieves slightly better image quality than the absolute CAR method, especially for the pixel values of input images ranging from 125 to 175. The reason is that, in platform A, the projection screen uses plastic curtain with high diffuse reflection coefficients.

The peak signal to noise ratio (PSNR) is employed to evaluate the proposed simple ACC algorithm. The parameter n in Eq. 1 is set to 8. In Fig. 7, the PSNR values of the proposed ACC algorithm and the absolute CAR method are shown with pixel values ranging from 0 to 250. Generally speaking, both methods achieve good calibration results, since in platform A, all three projectors are in close ages with similar brightness values. However, the absolute CAR method involves a more complex computing process than the simple color correction method.

Fig. 7
figure 7

PSNR value comparisons of experiment 1

Full size image

6.3 Experiment 2

In platform B, projectors properties differ drastically. While the proposed algorithm is applied to platform B, the hierarchical framework automatically selects the advanced color correction method. It is noted that the number of abnormal output points increases gradually along with the aged bulbs, making the CAR of the whole system narrow. We show great quality loss for the output images using the absolute CAR method (Fig. 8a). The brightness distribution of the absolute CAR method is shown in Fig. 8b. The output brightness of all projectors is artificially reduced to achieve uniform brightness. Figure 8c, d show the calibration result and the brightness distribution using the advanced color correction method. Since our method successfully excludes the isolated abnormal points and creates a wider ranged CAR, the calibrated outputs are visually seamless and with obviously higher image quality.

Fig. 8
figure 8

Different advanced color correction results

Full size image

The MSSIM metric is employed to evaluate the calibrated results generated by the advanced color correction method. Two traditional absolute CAR methods are implemented for performance comparison purposes, namely, the optimized color correction method [27] and the color correction method combining look-up Table [17]. The pixel values of the input images range from 0 to 255; and different color correction methods are applied sequentially to obtain the experimental results. Figure 9 shows the comparison results, verifying that the proposed advanced color correction method generates higher quality images than the traditional methods. Additionally, PSNR is used to further demonstrate the effectiveness of the algorithm. The experimental results with the values of PSNRs against the pixel values of input images are shown in Fig. 10. The number of bits per pixel n is still set to 8. In Fig. 10, the proposed advance color correction method generally achieves higher PSNR values against the other three methods, representing better quality color correction results. The reason is that the proposed method excludes isolated distortion points, which results in wider CAR. Moreover, it is noted that the traditional absolute CAR method has smaller PSNR value than the other three methods in Fig. 10, since the traditional absolute CAR method uses the narrowest CAR, resulting in the smallest color gradation in the calibrated output images. Comparing Fig. 10 with Fig. 7, the PSNR values of Experiment 2 is smaller, since in Experiment 2, the platform B is composed by projectors with different properties.

Fig. 9
figure 9

MSSIM comparisons for experiment 2

Full size image
Fig. 10
figure 10

PSNR value comparisons of experiment 2

Full size image

7 Conclusion

In this study, we proposed a novel hierarchical framework consisting of two color correction schemes to deal with the ACC problem for multi-projector display systems. The hierarchical framework is proved to be effective while a simple correction method and the advanced correction method are automatically selected according to the specific conditions. Both methods include three major steps. First, we build a general color model for each projector based on the projector properties, the display surface optical properties and other factors. Second, the transmission matrices and color models are built to generate the color look-up tables. Last, we combine the color look-up tables (CLTs) with corresponding brightness blending tables. The improved CLTs can facilitate the GPU rendering, since the pixel shaders can retrieve pixel information directly from the CLTs. According to the experimental results, both methods have advantages over traditional ACC approaches. The limitation of our solution includes: 1) a pre-assumption of planar display surfaces and 2) ignoring the anisotropy problems with rear-projection. Our future work will be along this line to accomplish ACC for general surface display systems, and eliminate the anisotropy phenomena by integrating more human computer interaction techniques into the hierarchical framework.