1 Introduction

The rapid development of digital media technology has enriched the forms of digital media through the combination of multi-projection, computer vision, pattern recognition, artificial intelligence, and other fields [1,2,3]. In particular, multi-channel spatial augmented reality technology enables real-time superposition of virtual content with the scene environment, integrating real and virtual objects to present diverse and natural forms. This technology has promising applications in fields such as medical rehabilitation, commercial experience, and science and technology exhibition halls [4, 5].

It is well-known that virtual reality headset display devices aim to generate immersion by emulating human stereoscopic vision, displaying virtual images separately for the left and right eyes, and combining them with audio cues to evoke a sense of immersion. Among various virtual reality display devices, the Cave Automatic Virtual Environment (CAVE) is considered one of the most immersive spatial augmented reality systems [6, 7]. Unlike headset display devices, CAVE-type systems, which can display multiple images simultaneously on multiple screens, alleviate the issue of vertigo commonly associated with headset displays [8].

The CAVE-type system structure not only enhances the artistic appeal of multi-projection displays but also provides observers with a stronger sense of immersion and interaction [9,10,11]. As CAVE-type systems rely on mutual reflection between projectors, several issues arise in real-world applications. Firstly, the quality of projected images and the fidelity of rendered scenes may be reduced due to mutual reflections between projectors. Secondly, the speed of model rendering is crucial for ensuring the quality of the multi-projection immersive environment. Thirdly, inconsistent optical parameter configurations between projectors may result in inconsistencies in color and brightness in the multi-projection display. Finally, inaccurate deployment and placement of multiple projectors can lead to image distortion. Moreover, manual adjustment of projector image content is a complex task in practice. Considering these challenges, this paper proposes a set of rear projection orthogonal multi-screen multi-projection display system platforms that incorporate geometric and inter-reflection compensation to achieve a highly immersive and seamless splicing fusion consistent display system.

To summarize, the contributions of this work are as follows:

  • To ensure the higher quality of multi-projection display while still realizing fast projection scattering compensation processing, the parallel accelerated mutual scattering compensation algorithm based on GPU is proposed. We demonstrate the effectiveness of our inter-reflection compensation model through comprehensive experiments, showing that it effectively addresses the issue of light scattering and achieves high immersion and high-resolution display.

  • Our approach allows for recovery of the display shape using a camera without the need for markers, making it scalable to displays of any resolution and significantly improving system efficiency.

  • Our approach does not rely on assumptions about the projection surface or require prior knowledge of the geometric shape, which is expected to facilitate future research in this direction.

The rest of this paper is organized as follows. Section 2 provides a review of related works. Section 3 presents the multi-projection distortion correction. Section 4 describes the scattering compensation for CAVE-type systems. Section 5 reports the discussion of this work. Finally, Section 6 concludes the paper and discusses future directions for research.

2 Related works

Despite the availability of various augmented reality display devices, the Cave Automatic Virtual Environment is still considered one of the most immersive spatial augmented reality systems. CAVE is capable of rendering images in real time, creating visually vivid and novel effects that overcome the spatial limitations of traditional displays [12,13,14]. In this section, we review two types of related content: geometric distortion correction of multi-projector images and inter-reflection compensation technology.

Geometric distortion correction of multi-projector images is a common research area in the field of visual effect correction for multi-projection display systems. Cameras are commonly used as sensing devices to provide feedback for adjusting the projected image and achieving automatic fusion correction of visual content [15]. Geometric distortion correction involves adjusting the geometric shape of the projected image and nonlinearly transforming it to match the target projected image.

To achieve online geometric correction, magnetic trackers are used to measuring the attitude and position of the rigid projection surface. Computer vision techniques are commonly used to estimate geometric information without requiring additional equipment, such as using captured surface projection textures [16], printed visual marks, or depth cameras [17, 18]. In the case of deformable surfaces, infrared cameras, and reflective markers [19] are used to map onto the projection surface. For example, Siege et al. [20] proposed a fully dynamic multi-projection mapping system that used multiple projectors to draw high-quality dynamic content on the surface of objects. Later, they further optimized the system to enable real-time tracking of the movement of the target projector using RGB-D cameras without visible marks. However, this method is prone to error as it requires a 6DOF (degrees of freedom) geometric transformation from the camera to the projector. Jones et al. [21] proposed the IllumiRoom system, which used projection light to enhance the TV game screen, and later the RoomAlive system, which explored a completely independent multi-projection augmentation system for screen games [22]. The RoomAlive system used optical flow-based particle tracking for interactive intelligent projection, allowing observers to immerse in seamless fusion projection presentations and achieve an immersive dynamic adaptive projection mapping experience.

In the context of multi-channel imaging space enhancement display systems, digital image content or virtual model information is projected onto a real physical medium using multiple projectors, and visual effect correction methods are used to adjust the imaging display information to enhance the visual effect environment of the projection space and provide a dynamic and immersive interactive experience for users. Many immersive multi-projection systems are based on tiled display systems, with the key to immersion being the uniform distribution of illumination [23]. However, in practical applications, the illumination of the projection may exhibit brightness attenuation from the center to the edge, overlapping brightness, and superimposed interference, which can affect the immersion of multi-projection content. Sheng et al. [24] proposed a global illumination elimination algorithm, but this method requires prior knowledge of the geometric information and assumes that the projection surface is a Lambertian surface, limiting its applicability in mutual reflection compensation of high-resolution projected images.

In the context of immersive multi-projection systems that involve the collection and storage of optical transmission information, Li et al. [25] proposed a method for mutual reflection compensation of multi-projection images using nonlinear optimization. Takeda et al. [26] proposed a method to optimize the reflection model and reduce the reflectivity of the projection screen to achieve mutual reflection compensation for the illumination of multiple projection contents [17]. The perception of content by the human eye is primarily influenced by the brightness of the projector, with chromaticity information of the projected content having weaker sensitivity to human eye perception relative to brightness. Steimle et al. [27] analyzed and explained the degree to which human vision is affected by light brightness and chromaticity.

In view of the above, the mutual reflection compensation algorithm for multi-channel projection images is proposed, which has successfully realized seamless fusion consistency display of CAVE-type projections with high immersion, which is scalable to displays of any resolution and has significantly improved the practical efficiency of our system. Meanwhile, our approach does not require any assumptions about projection surface, and geometric shape does not need to be known in advance, making our approach scalable to displays of any resolution and significantly improving our system’s practical efficiency.

3 Multi-projection distortion correction

The placement of projectors at random locations often results in nonlinear distortion of the projected image on the display screen, leading to poor continuity and weak immersion of the display. Therefore, it is necessary to correct the distortion of the image on each projection display screen in advance. Manual adjustment of the directional freedom of the projectors to correct the distortion is challenging due to the relationship between adjacent projection display screens and external factors such as the mechanical structure of the projector bracket and external force interference [28, 29]. To address these issues, we propose a geometric distortion correction approach based on color-coding structured light to establish a one-to-one correspondence between the projected coding features and the modulation coding features of the projection display, enabling correction of the distortion of the projection display screen.

3.1 Color structured light coding

To establish the corresponding relationship between the projected coding circle feature and the camera-collected feature, a geometric distortion correction approach based on color coding sampling is proposed, which has used color coding structured light to generate a color circle feature pattern, as shown in Fig. 1. In this process, we utilize the pseudo-random characteristics of the De Bruijn sequence to encode the color elements to make the code unique and maximize the information value of the color code. We generate a cyclic symbol sequence of length \(q^m\) in the color symbol alphabet of q-symbol for the De Bruijn sequence of q-symbol m level, where any subsequence of length m in the sequence of the elements is unique. We utilize red, green, blue, cyan, magenta, yellow, and white as the color code elements, resulting in a 7-element 3-level De Bruijn color-coded structured light pattern.

To enable quick matching of the code features, we propose a code feature matching method based on the repeatability of sequence features. We first extract the left gradient value sequence and the right gradient value sequence of the projected original stripe code sequence and the modulation stripe code sequence acquired via the camera. We then identify all the gradient codes that satisfy certain conditions in the four gradient value sequences mentioned above, and refer to the code corresponding to the gradient code as a kernel code. We determine the minimum repetition degree of the left and right domain code sequences formed by the kernel code. We gradually expand the repetition of the code sequence by combining adjacent kernel codes, and identifying the maximum sequence repetition corresponding to the code feature sequence. We match the leftmost sequence code of the leftmost sequence code in the sequence of maximum repetition, and if it matches, we add it to the sequence. Otherwise, we stop matching. We repeat the same process for the rightmost sequence code in the sequence of maximum repetition.

Fig. 1
figure 1

Design color structured light coding feature pattern

Full size image

Through this process, we can further expand the repetition of the code character sequence. We remove the matched code sequence from the sequence of projected original codes and the sequence of codes captured by the camera. We process all adjacent kernel codes until a separate kernel code remains. We then continue to extend the code feature sequence repeatability of the independent kernel codes, and also match the left gradient value sequence code and the right gradient value sequence code corresponding to the leftmost and rightmost sequence code search of the code sequence. If it matches, it is added to the sequence, and if it does not match, it is stopped. The stripe code feature matching algorithm based on sequence feature repeatability effectively addresses the problem of non-continuous code matching, and has low time complexity compared to dynamic matching methods.

3.2 Geometric mapping

We collect the modulated encoded image through the camera, and use the ellipse fitting method to extract the center of the ellipse as the feature point of the modulated encoded image, and assign color to the feature point of the image. We use the feature matching method of color structured light to establish the correspondence f(m), between the encoded feature points \(I({p_{xi}},{p_{yi}})\) and the feature points \(\mathrm{{C(}}{c_{uj}},{c_{vj}}\mathrm{{)}}\) of the modulated image, defined by:

$$\begin{aligned} f(m):I({p_{xi}},{p_{yi}}) \rightarrow \mathrm{{C(}}{c_{uj}},{c_{vj}}\mathrm{{)}} \end{aligned}$$
(1)

The feature point \(\mathrm{{C(}}{c_{uj}},{c_{vj}}\mathrm{{)}}\) as a set of vertices to mesh the triangle sub-region by the Delaunay algorithm, that to obtain the description set \({\Upsilon _C}(\mathrm{{C(}}{c_{uj}},{c_{vj}}\mathrm{{)}},{\Psi _C})\), where \({\Psi _C}\) is the connection description of the triangle sub-region grid edges. Through the mapping relationship f(m) and \({\Psi _C}\), the description set \({\Psi _P}\) of the coding feature point is obtained, the mapping parameters \({A_k} = \{ a_k^1,a_k^2,...a_k^9\} ,k = 1,2...\mathrm{{n}}\) of the n triangular sub-regions to the description set \({\Psi _P}\) and \({\Psi _C}\) are calculated by:

$$\begin{aligned} \left\{ {\begin{array}{c} {p_{{x_k}}^w{S_k}\mathrm{{ = }}a_k^1c_{{u_k}}^w + a_k^2c_{{v_k}}^w + a_k^3}\\ {p_{{y_k}}^w{S_k}\mathrm{{ = }}a_k^4c_{{u_k}}^w + a_k^5c_{{v_k}}^w + a_k^6}\\ {{S_k} = a_k^7c_{{u_k}}^w + a_k^8c_{{v_k}}^w + a_k^9} \end{array}} \right. ,w = 1,2,3 \end{aligned}$$
(2)

Where \((p_{{x_k}}^w,p_{{y_k}}^w)\) is the coordinates of the w-th vertex of the k-th triangular subregion, that the corresponding coordinate information is \((c_{{u_k}}^w,c_{{v_k}}^w)\) . The mapping relationship of each triangle sub-region is calculated by the above method, and the multi-region mapping relationship set parameters \({A_{k = 1,2...n}}\) can be obtained.

Fig. 2
figure 2

Geometric consistency correction of multi-projector

Full size image

To validate our approach for multi-projector geometric distortion correction, we conducted an experiment using a multi-projector display set up in a distorted environment. Figure 2 illustrates the image displayed after applying our geometric distortion correction method to the multi-projector setup. As evident from the image, there is no noticeable distortion in the display, and the projected image aligns seamlessly with the plane screen projection. This demonstrates that our method minimizes the need for manual adjustment of the projector position, thereby simplifying the installation and adjustment process of the multi-projection display system. Our approach relies on the observation that each triangular sub-region exhibits an approximately linear mapping relationship. By leveraging this mapping relationship, we can calculate the pixel coordinate values corresponding to the triangular sub-regions, enabling effective distortion correction of the projected image. This eliminates the need for complex manual adjustments, making our method efficient and practical for real-world multi-projector display systems.

4 Scattering compensation for CAVE-type systems

In immersive multi-projection CAVE-type systems, the illumination information emitted by the projectors is not uniformly transmitted to the projection screen. This is due to the fact that the brightness information transmitted in different directions varies, resulting in a cosine radiator mode phenomenon [30, 31]. As a result of this non-uniform illumination, there can be variations in image quality and clarity across different areas of the projection screen. Additionally, the projected image is not only affected by ambient light, but also by mutual scattering between adjacent projection screens, further impacting the brightness of the image [32]. Consequently, the contrast of the projected image is reduced, leading to a decline in the overall picture quality of the system [33, 34].

4.1 Problem conceptualization

The light source information output by the projector can be regarded as a surface light source. The light energy generated by the radiation around in unit time is called luminous flux information \(\overline{\Phi }\). Its expression is shown in Eq. (5), where \(d\overline{Q}\) is the unit energy transmitted in dt time.

$$\begin{aligned} \overline{\Phi }{{ = }}{{d\overline{Q}}/{dt}} \end{aligned}$$
(3)
Fig. 3
figure 3

Light intensity characteristics of cosine radiation

Full size image

For the plane light source, the value of the illuminance \(\overline{E} \) generated on the face patch dS of the distance r is as shown in Eq. (6), where \(\overline{L}\) is the illuminance generated by the unit face patch in the face light source at the distance r , \({\theta _1}\) is the angle between the face element normal of the light source and the face of the light source, \({\theta _2}\) is the angle between the face element normal of the light source and the face of the light source, and \(d{S_e}\) is the face patch area of the light source.

$$\begin{aligned} \overline{E}{{ = }}{{d\overline{\Phi }}/{dS}}( {\overline{L} d{S_e}\cos {\theta _1}\cos {\theta _2}})/{r^2} \end{aligned}$$
(4)

Since the illumination information output by the projector is not evenly transmitted to the projection screen, the brightness information transmitted in different directions is inconsistent, so the light transmission process in CAVE-type system can be approximately regarded as cosine radiation mode, as shown in Fig. 3.

The luminous intensity \({\overline{I} _\theta }\) of the slow-reflecting object in the cosine radiation model in any angle \(\theta \) direction formed from the normal can be defined by Eq. (7), Where \({\overline{I} _N}\) is the reflected light intensity of the face patch in the normal direction. For the luminance \(\overline{L}\) generated by the panel patch dS in the range of \(\Omega \) solid angle range in any direction in space, it is shown in Eq. (8), and the luminance \(\overline{\Psi }\) generated by the cosine radiation model in the range of \(2\pi \) solid angle range is as shown in Eq. (9).

$$\begin{aligned} {\overline{I} _\theta }\mathrm{{ = }}{\overline{I} _N}\cos \theta \end{aligned}$$
(5)
$$\begin{aligned} \overline{L} {{ = }}{{d\overline{\Phi }}/{(\cos {\theta }dSd\Omega )}}=I_N/{(\cos {\theta }dS)} \end{aligned}$$
(6)
$$\begin{aligned} \overline{\Psi }{{ = }}{\Phi }/dS=(\overline{L}\pi dS)/dS=\overline{L}\pi \end{aligned}$$
(7)

4.2 Parallel scattering photometric compensation

In practical applications, it is challenging to completely eliminate light scattering from projectors, which can result in mutual scattering effects between multiple projection screens [35, 36]. This phenomenon can significantly degrade the quality and clarity of projected images. Therefore, it is crucial to develop a robust mathematical model that accurately characterizes the intricate interplay of these effects. By doing so, we can ensure that multi-projection systems deliver optimal performance and provide viewers with an immersive visual experience that fully engages their senses.

It is noteworthy that several mathematical strategies exist for compensating for scattering photometric effects. However, these compensation strategies often require excessive manual intervention and exhibit high time complexity. Therefore, it is imperative to develop a more efficient and automated approach to overcome these challenges and achieve accurate photometric compensation in multi-projection systems. In the context of immersive multi-projection systems, the posture relationship between the projector and the projection display screen can be categorized into two modes: forward projection to the projection screen and back projection to the projection screen. The back projection mode involves the light transmittance of the projection screen, which adds an additional layer of complexity to the photometric compensation process, as shown in Fig. 4. We assume that \({\omega _{re}}\), \({\omega _{fro}}\) are transmittance of projection screen, defined by:

$$\begin{aligned} \omega _{re}={\overline{\Phi }_2}/{\overline{\Phi }_1},\omega _{fro}=({\overline{\Phi }_3}-\overline{\Phi }_4)/{\overline{\Phi }_3} \end{aligned}$$
(8)

Where \({\overline{\Phi }_1}\) is luminous flux when a tiny light spot is projected onto the screen, \({\overline{\Phi }_2}\) is luminous flux that a tiny light spot through the screen, \({\overline{\Phi }_3}\) is the luminous flux scattered by the illumination to the adjacent screen, \({\overline{\Phi }_4}\) is the luminous flux \({\overline{\Phi }_3}\) through the adjacent screen.

Fig. 4
figure 4

Scattering photometric problem of CAVE-type system

Full size image

\({\overline{IP} _i}\) is a pixel of the projected image on the projection screen, the image pixels presented on the projection screen have \(r{e_{pix}}\), then the screen transmittance is \(\overline{Ef}_i=\overline{Er} _i/\omega _{re}\), that the brightness value \({\overline{Ef} _i}\) corresponding to the illuminance is \(\overline{L}_{\overline{Ef}_i}=\overline{Ef}_i/(re_{pix}\pi )\), When the illuminance of the pixel \({\overline{IP} _i}\) is transferred to the pixel \({\overline{IP} _j}\) of the adjacent projection screen, the corresponding reflected illuminance is \({\overline{Ea} _{ij}}\), defined by:

$$\begin{aligned} {\overline{Ea} _{ij}}\mathrm{{ = }}{\omega _{fro}}{\mu _{ij}}{\overline{Ef} _i} \end{aligned}$$
(9)
$$\begin{aligned} \begin{array}{l} {\overline{Ea} _{ij}}\mathrm{{ = }}{{\left( {{\omega _{fro}}{{\overline{L} }_{{{\overline{Ef} }_i}}}r{e_{pix}}\cos {\theta _1}\cos {\theta _2}} \right) } / {r_{{{\overline{IP} }_i}{{\overline{IP} }_j}}^2}}\\ \;\;\;\;\;\; = {{\left( {{\omega _{fro}}{{\overline{Ef} }_i}\cos {\theta _1}\cos {\theta _2}} \right) }/{\left( {r_{{{\overline{IP} }_i}{{\overline{IP} }_j}}^2\pi } \right) }} \end{array} \end{aligned}$$
(10)

Where \({\theta _1}\) and \({\theta _2}\) are the angels between the ray connecting the two patches’ centers and their normal, \({r_{{{\overline{IP} }_i}{{\overline{IP} }_j}}}\) is the two patches distance, we can get the value of \({\mu _{ij}}\mathrm{{ = }}{{\left( {\cos {\theta _1}\cos {\theta _2}} \right) } / {\left( {r_{{{\overline{IP} }_i}{{\overline{IP} }_j}}^2\pi } \right) }}\) through formulas (9) and (10).

Because the scattering effect involves the reflection of illuminance at multiple points, the point \({\overline{IP} _i}\) on the screen, its corresponding illuminance is \({\overline{EM} _i}\) , which is calculated as

$$\begin{aligned} {\overline{EM} _i}\mathrm{{ = }}{\overline{Ef} _i}\mathrm{{ + }}\sum \nolimits _j^{Set} {{{\overline{Ea} }_{ji}}} \mathrm{{ + }}\sum \nolimits _j^{Set} {{{\overline{Es} }_{ij}}} \end{aligned}$$
(11)

Where \({\overline{Ef} _i}\) is the illuminance obtained by itself, \(\sum \nolimits _j^{Set} {{{\overline{Ea} }_{ji}}}\) is the illuminance reflected by set of points of the adjacent screen, \({\overline{Es} _{ij}}\mathrm{{ = }}{\omega _{fro}}{\mu _{ij}}{\overline{Ea} _{ji}}\) is the brightness of light radiated to this point by secondary scattering. However, in the actual scattering process, multiple screen scattering will be involved. According to the above process, multiple cross scattering process is an iterative recursive process, so we can get the mathematical form of k-th secondary scattering defined as

$$\begin{aligned} \left\{ \begin{array}{l} \overline{EM} _i^0\mathrm{{ = }}{\overline{Ef} _i}\\ \overline{EM} _i^1 = {\overline{Ef} _i}\mathrm{{ + }}\sum \nolimits _j^{Set} {\overline{Ea} _{ji}^1} \mathrm{{ + }}\sum \nolimits _j^{Set} {\overline{Es} _{_{ij}}^1} \\ \overline{EM} _i^2 = \overline{EM} _i^1 + \sum \nolimits _j^{Set} {\overline{Ea} _{ji}^2} \mathrm{{ + }}\sum \nolimits _j^{Set} {\overline{Es} _{_{ij}}^2} \\ ...\\ \overline{EM} _i^k = \overline{EM} _i^{k - 1} + \sum \nolimits _j^{Set} {\overline{Ea} _{ji}^k} \mathrm{{ + }}\sum \nolimits _j^{Set} {\overline{Es} _{_{ij}}^k} \end{array} \right. \end{aligned}$$
(12)

Where \({\mu _{ji}}\mathrm{{ = }}{\mu _{ij}}\), \(\overline{Es} _{_{ij}}^k\mathrm{{ = }}\sum \nolimits _j^{Set} {\omega _{fro}^2\mu _{_{ij}}^2\overline{Es} _{_{ji}}^{k - 1}}\), \(\overline{Ea} _{ji}^k\mathrm{{ = }}\sum \nolimits _j^{Set} {{\omega _{fro}}{\mu _{ij}}\overline{Ea} _{_{ji}}^{k - 1}}\), these two mathematical expressions are equal ratio series, when k is infinite, the final mathematical form of multistage scattering is obtained defined as

$$\begin{aligned} \overline{EM} _i^\Theta = {\overline{Ef} _i}\mathrm{{ + }}\sum \limits _j^{Set} {(\frac{{\omega _{fro}^2\mu _{ij}^2}}{{1 - \omega _{fro}^2\mu _{ij}^2}})} {\overline{Ef} _i} + \sum \limits _j^{Set} {(\frac{{{\omega _{fro}}{\mu _{ij}}}}{{1 - {\omega _{fro}}{\mu _{ij}}}})} {\overline{Ef} _j} \end{aligned}$$
(13)

When the projector outputs illuminance to the projection screen, the actual illuminance generated on the projection screen is \(\overline{EM} _i^\Theta \), \(E{c_i}\) is the compensation value of cross scattering between multi-level screens.

$$\begin{aligned} \begin{array}{l} E{c_i} = ({\overline{Ef} _i}\mathrm{{ - }}\sum \limits _j^{Set} {(\frac{{{\omega _{fro}}{\mu _{ij}}}}{{1 - {\omega _{fro}}{\mu _{ij}}}})} {\overline{Ef} _j})/(1\mathrm{{ + }}\sum \limits _j^{Set} {(\frac{{\omega _{fro}^2\mu _{ij}^2}}{{1 - \omega _{fro}^2\mu _{ij}^2}})} )\\ \;\;\;\;\;\; = ({\overline{Ef} _i}\mathrm{{ - }}\sum \limits _j^{Set} {m{d_{ij}}} {\overline{Ef} _j})/\mathrm{{m}}{\mathrm{{c}}_i} \end{array} \end{aligned}$$
(14)

Where \(\mathrm{{m}}{\mathrm{{c}}_i}\mathrm{{ = }}1\mathrm{{ + }}\sum \limits _j^{Set} {(\frac{{\omega _{fro}^2\mu _{ij}^2}}{{1 - \omega _{fro}^2\mu _{ij}^2}})}\), \(m{d_{ij}} = \frac{{ - {\omega _{fro}}{\mu _{ij}}}}{{{\omega _{fro}}{\mu _{ij}} - 1}}\), we can see that \(\mathrm{{m}}{\mathrm{{c}}_i}\) and \(m{d_{ij}}\) are constants.

The mutual scattering effect in multi-projection systems can significantly impact image quality, and direct CPU-based processing can be time-consuming. To address this challenge and achieve fast projection scattering compensation while maintaining image quality, we propose a parallel accelerated mutual scattering compensation algorithm based on GPU. The GPU (Graphics Processing Unit) is a hardware architecture that facilitates data parallelism through a multitude of graphics functions, enabling concurrent execution of multiple independent threads for data processing. Unlike CPUs, GPUs are optimized for graphics-intensive tasks and exhibit inherent parallel processing capabilities. Leveraging this characteristic, we can convert data processing into multiple threads that can be executed concurrently on the GPU. Furthermore, the cross-scattering effect between projection screens typically occurs only in certain regions of the adjacent screens, while other regions remain unaffected. Therefore, during the scattering compensation process, identifying the regions of mutual scattering effect in advance can significantly reduce the computational overhead and improve the overall calculation efficiency of the scattering compensation algorithm.

To determine the scattered regions R1 and R2 between adjacent projection screens S1 and S2, we establish a mapping relationship between the projectors P1 and P2 and a camera using color structured light. In this description, we assume that relevant camera parameters are adjusted and kept fixed. The process of finding the scattered region R1 is as follows: First, we turn off projectors P1 and P2, and the camera captures a background image B of screen S1. Next, projector P2 projects a white image onto screen S2, and the camera captures the scattered image W of screen S1. The difference between the background image B and the scattered image W is then processed to identify the pixel positions where the gray level of the image changes. These pixel positions are subsequently converted into the coordinate space of the projected image using the mapping function, resulting in the determination of the scattered region R1. Similarly, the scattering region R2 can be obtained through a similar process.

To accelerate the scatter compensation process via GPU parallel processing. One-dimensional arrays M1 and M2 are constructed to store the constant values \(\mathrm{{m}}{\mathrm{{c}}_i}\) corresponding to adjacent projection screens. The Scattering regions R1 and R2 are converted into one-dimensional linear arrays, and the corresponding pixel gray information values are obtained by combining the arrays M1 and M2, which are stored in arrays V1 and V2, respectively. The two-dimensional array \(\mathrm{{FD[i][j]}}\) is constructed to store the constant value \(m{d_{ij}}\), (t+1)×(t+1) parallel threads is created on the GPU, where t+1 is the number of pixels in scattered region. Execute \(\mathrm{{FD[i][j]}} \times {\mathrm{{V}}_1} \times {\mathrm{{V}}_2} \rightarrow \mathrm{{FD[i][j]}}\) in parallel for (i, j) thread, execute \((1\mathrm{{ + }}\sum \nolimits _{j = 0}^{j \le t} {\mathrm{{FD[i][j]}}} )/(\mathrm{{G}}{\mathrm{{V}}_1}[i] \times \mathrm{{M}}{\mathrm{{C}}_1}[i]) \rightarrow \mathrm{{G}}{\mathrm{{V}}_1}[i]\) in parallel for (i, 0) thread, execute \((1\mathrm{{ + }}\sum \nolimits _{j = 0}^{j \le t} {\mathrm{{FD[i][j]}}} )/(\mathrm{{G}}{\mathrm{{V}}_2}[i] \times \mathrm{{M}}{\mathrm{{C}}_2}[i]) \rightarrow \mathrm{{G}}{\mathrm{{V}}_2}[i]\) in parallel for (0, j) thread, where \(\mathrm{{G}}{\mathrm{{V}}_1}[i]\) and \(\mathrm{{G}}{\mathrm{{V}}_2}[i]\) are the gray values of scattering compensation solved via parallel processing. To obtain the ultimate visual consistency with human eye, set the threshold \(\xi \), compare whether \(\mathrm{{G}}{\mathrm{{V}}_1}[i]\) and \(\mathrm{{G}}{\mathrm{{V}}_2}[i]\) values and gray values of original projected images are within the range of threshold \(\xi \). If the value is within the threshold range, the parallel computation is stopped; otherwise, the execution continues. Calculate the difference between the solved \(\mathrm{{G}}{\mathrm{{V}}_1}[i]\) (or \(\mathrm{{G}}{\mathrm{{V}}_2}[i]\) ) and the gray value of the original projected image, and compare whether they are within the range of threshold \(\xi \). If they are within the range of threshold \(\xi \), stop the parallel calculation; otherwise, continue the execution.

4.3 Method validation

The problem of light scattering between adjacent projection screens poses a significant challenge in achieving high-quality projection display images in immersive multi-projection display systems. Experimental tests were conducted to evaluate the effectiveness of the proposed scattering photometric compensation solution, as depicted in Fig. 5. The results demonstrate that after correcting for projection distortion, the projected display picture can be presented completely and continuously on the projection screen, enabling seamless splicing of multi-projection displays, thereby achieving an immersive visual experience. In Fig. 5(a) and (d), where no illuminance scattering compensation is applied, a noticeable difference in brightness is observed at the intersections of the projection screens. However, upon applying the illumination mutual scattering compensation algorithm for compensation and correction, as shown in Fig. 5(b) and (e), the phenomenon of light scattering in the projected image is mitigated. The difference in illuminance between the compensated and uncompensated images is illustrated in Fig. 5(c) and (f). These experimental results provide evidence of the effectiveness of the proposed scattering photometric compensation approach in improving image quality and mitigating the impact of light scattering in multi-projection display systems.

Fig. 5
figure 5

Cross-scattering compensation for illumination of projected images

Full size image

To visually assess the effectiveness of illuminance cross-scatter compensation between adjacent projection screens, a pseudo-color compensation experiment was conducted. The results are presented in Fig. 6, where the first and third rows shows the images without illuminance scattering compensation, and the first column displays the corresponding pseudo-color images. Following compensation and correction using the illumination mutual scattering compensation algorithm, the results are shown in the second and fourth rows of Fig. 6. Upon comparing the compensation and without scattering compensation of images via Fig. 6, it can be observed that after applying illuminance cross-scatter compensation, the pseudo-color compensation images exhibit greater consistency with the visual perception of the viewer. This indicates that the proposed compensation algorithm effectively mitigates the impact of illuminance cross-scatter, resulting in improved pseudo-color images that align with human visual perception. These findings further validate the efficacy of the proposed scattering photometric compensation approach in enhancing image quality in multi-projection display systems.

Fig. 6
figure 6

Pseudo-color experiment of illuminance cross-scatter compensation

Full size image

5 Discussion

In this section, we have developed an immersive stereoscopy multi-projection Cave Automatic Virtual Environment display system, as depicted in Fig. 7. Our setup includes high-performance graphics workstations, multiple projectors, and sound processing equipment. The high-performance graphics workstation is utilized for various tasks such as distortion correction, scattering compensation correction, chromaticity consistency correction, collaborative control, and rendering control for the multi-projection display screen. The display screen covers the viewer’s visual field, allowing for interaction with the virtual scene and enabling the viewer to roam and immerse themselves in the virtual environment.

However, the display screen may exhibit non-linear distortion caused by mechanical loosening of the projector bracket and external interference factors, resulting in poor continuity and weak immersion of the displayed images, as well as increased visual fatigue for the viewer. To address this issue, we establish the corresponding relationship for each projection display image using our proposed structured light coding method in order to correct the distortion of the displayed images. One notable feature of our approach is that it does not rely on any assumptions about the projection surface, and the geometric shape of the projection screens does not need to be known in advance. This characteristic enables our method to promote the rapid development of multi-projection display technology towards the direction of intelligence, practicability, and efficiency.

Fig. 7
figure 7

Immersive multi-projection display system

Full size image

The human vision system is truly remarkable, as it can adjust the line of sight of our eyes to automatically align with an object point. This incredible feat allows us to generate depth information about that particular object point, which is crucial for our understanding and perception of the world around us. To replicate this process, we can create left and right scene images through a process known as human-like visual perception. By doing so, we can create a sense of stereo vision that mimics what we experience in real life. To further enhance this experience, virtual cameras can be constructed to simulate real cameras and obtain even more detailed scene information. Through perspective projection techniques, we are then able to transform these three-dimensional scenes into two-dimensional representations that accurately reflect how they would appear in reality. To display the virtual space, the two-dimensional scene plane composite of virtual scene under all viewpoints, as shown in Fig. 8. The stereo parallax generator is used to convert the left-eye scene image and the right-eye scene image. A special auxiliary device (e.g. stereoscopic glasses) is used to enable the viewer’s eye to see the two-dimensional virtual scene image of the projected picture.

Fig. 8
figure 8

The two-dimensional scene plane composite of virtual scene under all viewpoints

Full size image

The immersive multi-projection CAVE display system developed in this work represents an advanced and immersive data visualization and man-machine interactive roaming system. This system has the potential to be applied in various virtual simulation fields that require high levels of immersion, such as virtual assembly, simulation training, driving simulation drills, etc. Our system integrates interactive devices that are capable of perceiving user instructions, and transmits the perceived and understood interactive instructions to the computer for processing. In response, real-time adjustments are made to the virtual scene to enhance the user’s roaming and interactive experience within the immersive virtual environment, as depicted in Fig. 9.

Fig. 9
figure 9

User roam and interact in immersive virtual scenes

Full size image

Through thorough experimentation, our approach has successfully realized seamless fusion consistency display of multi-channel projections with high immersion, which is scalable to displays of any resolution and has significantly improved the practical efficiency of our system. Therefore, our system has demonstrated not only practical convenience, but also an enhancement in the realism and visual impact of the virtual scenes. The combination of interactive devices, real-time processing of user instructions, and dynamic adjustments to the virtual scene has resulted in a compelling and immersive experience for users.

6 Conclusions

We propose an inter-reflection compensation algorithm for multi-projection systems to address the issue of immersion and achieve high levels of visual fidelity. Our approach has been experimentally validated, demonstrating seamless fusion consistency across multiple channels with scalable resolution capabilities. Notably, our method significantly improves practical efficiency. Our approach stands out for its independence from assumptions about the projection surface and lack of requirement for prior knowledge of the geometric shape of projection screens. This unique characteristic facilitates the advancement of multi-projection display technology towards greater intelligence, practicality, and efficiency. Furthermore, our method has potential to extend support to multi-user interactive high-reality displays. In future research, we plan to investigate the application of our multi-projection display technology for immersive naked eye displays. This direction of research holds promise for further enhancing the immersive experience of users in virtual environments, and we anticipate that it will contribute to the advancement of the field of multi-projection display technology.