1 Introduction

The authors have reported the image resolution under the effect of color speckle using white-and-black line pairs (grille patterns) for raster-scan RGB mobile laser projectors using an MEMS (micro electro-mechanical systems) scanner [1,2,3]. The color speckle is known as random interference patterns on the human retina. That is, the partially coherent R (red), G (green), B (blue) laser lights projected on the screen generate the color speckle noise on retina. Therefore, a speckle measurement device with optics simulating the MTF (modulation transfer function) of the human-eye optics must be used. For this purpose, a pinhole (iris) is usually set in front of the lens of the measurement device.

In theory, the color speckle chromaticity coordinates for a uniform white pattern except its edges are distributed randomly around the white chromaticity point (W). However, we found that the color speckle at the pattern edges was distributed more complicatedly with randomness of color speckle noise on an underlying color shifting trajectory, which greatly affects the image resolution [1]. It is necessary for analyzing the edge color shift to extract pure color shift data by minimizing the effect of the color speckle noise. One of the methods for minimizing the color speckle effect is to open the iris of the speckle measurement device [3,4,5]. We can make speckle grain size smaller than the pixel size of the two-dimensional (2D) image sensor when the iris opens wider. The smaller speckle grains are thus averaged in the sensor pixel area. The iris diameter is originally designed to reproduce the speckle grain sizes equivalent to those on the human retina. Therefore, it is an unusual usage for the speckle measurement device to open the iris wider. However, this method was quite useful for extracting the underlying behavior of the edge color shift [6]. It is also useful for the comparison between the edge color shift and the color speckle effects. Examples of the edge color shift under the full effect of color speckle and under the minimized color speckle effect are shown in Fig. 1 (a) and (b), respectively. The other feature of this measurement method is that the pure behavior of the R, G, B laser light sources can be independently analyzed by changing the color filters equipped with the speckle measurement device. Therefore, the pure R, G, B color behaviors of each laser sources can be analyzed using only white patterns. The new measurement method in this work comprehensively covers the effects of the color speckle, the edge color shift, the performance of R, G, B laser sources, and the performance of the MEMS scanner.

Fig. 1
figure 1

Examples of the edge color shift under the full effect of color speckle (a) and under the minimized color speckle effect (b)

Full size image

In this work, the edge color shift behavior of three different RGB mobile laser projectors was measured and compared. As a special issue paper of LDC 2021 [6], the measurement results of the pure color shift behavior at image pattern edges are analyzed more in detail.

2 Theory

The color speckle has been discussed theoretically and experimentally in our previous works [1,2,3,4,5,6,7,8,9,10,11,12]. The measurement method of color speckle based on these works was already standardized as IEC 62,906–5-4 [13].

The colorimetric metrics of a white pattern projected on a standard diffusive reflectance screen by the RGB laser mobile projector can be obtained by a single wavelength value for each R, G, B laser output spectrum and the corresponding single value of CIE color matching functions,\(\overline{x}(\lambda_{Q} ),^{{}} \overline{y}(\lambda_{Q} ),^{{}} \overline{z}(\lambda_{Q} )\).

The tristimulus values X, Y, Z are calculated by the following equations.

$$\begin{gathered} X = \overline{x}(\lambda_{R} )E_{e - R} (i,j) + \overline{x}(\lambda_{G} )E_{e - G} (i,j) + \overline{x}(\lambda_{B} )E_{e - B} (i,j), \hfill \\ Y = \overline{y}(\lambda_{R} )E_{e - R} (i,j) + \overline{y}(\lambda_{G} )E_{e - G} (i,j) + \overline{y}(\lambda_{B} )E_{e - B} (i,j), \hfill \\ Z = \overline{z}(\lambda_{R} )E_{e - R} (i,j) + \overline{z}(\lambda_{G} )E_{e - G} (i,j) + \overline{z}(\lambda_{B} )E_{e - B} (i,j), \hfill \\ \end{gathered}$$
(1)

where, Ee-Q (i, j) is the measured irradiance of each color (Q = R, G, B). The R, G, B spectral power ratios are implicitly included the irradiance. The symbols, i, j indicate the pixel position of the 2D image sensor of the speckle measurement device.

CIE 1931 chromaticity coordinates, (x, y) are calculated by the following equations:

$$x = \frac{X}{X + Y + Z},^{{}} y = \frac{Y}{X + Y + Z}.$$
(2)

Illuminance, Ev is simply obtained by the following equation:

$$E_{v} = 683 \times Y.$$
(3)

Monochromatic speckle contrast, Cs-Q, is given by the following equation:

$$C_{s - Q} = \frac{{\sigma_{e - Q} }}{{\overline{E}_{e - Q} }},$$
(4)

where \(\overline{E}_{e - Q}\) is the average of the distribution of Ee-Q (i, j), and σe-Q is the standard deviation. The speckle contrast must be calculated for uniformly illuminated regions. The probability density function for the monochromatic speckle can be statistically expressed as a function of speckle contrast, Cs-Q.

$$p\{ E_{e(norm) - Q} (i,j) = \frac{{E_{e(norm) - Q}^{{C_{s - Q}^{ - 2} - 1}} (i,j)}}{{\Gamma (C_{s - Q}^{ - 2} )}}\exp \{ - E_{e(norm) - Q} (i,j)\} ,$$
(5)

where Γ is the gamma function, and Ee(norm)-Q(i, j) is the irradiance values normalized by the average.

Photometric speckle contrast, Cps, for white color is expressed as follows, using Eqs. (1)–(3):

$$C_{ps} = \frac{{\sigma_{v} }}{{\overline{E}_{v} }},$$
(6)

where \(\overline{E}_{v}\) is the average of the illuminance distribution of Ev (i, j), and σv is the standard deviation.

The CIE 1931 chromaticity coordinates, (x, y), are distributed around the original white chromaticity point, W. The distribution range depends on the R, G, B speckle contrasts, Cs-R, Cs-G, and Cs-B. The color speckle is defined as random speckle grains with various colors in the white-illuminated area on the retina.

3 Measurement preparation

3.1 Raster-scan RGB mobile laser projectors

Three raster-scan RGB mobile laser projectors, #1, #2, and #3 were used for the measurements in this work. The R, G, B laser beams are combined and raster-canned by an MEMS scanner. High-performance MEMS scanners with ultra-flat high-frequency performance for high-resolution laser projection applications were developed for the projector applications [14].

The R, G, B wavelengths and CIE1931 white chromaticity coordinates of the three projectors are shown in Table 1. The measured R, G, B speckle contrasts and the photometric speckle contrast for the white pattern are also shown in Table 2. The signal pixel resolution of the three projectors was nominally specified as 1280 × 720 (16:9). However, the signal pixel of the raster-scan RGB mobile projectors are virtually reproduced on a screen. The R, G, B signals created in the time domain by switching the R, G, B laser diodes are converted into the spatial domain by MEMS scanning of the R, G, B combined white laser beam. That is, the actual minimum signal pixel is virtually created depending on the beam shape/size, scanning speed, rise/fall time of the R, G, B laser diodes, etc.

Table 1 R, G, B wavelengths and white chromaticity of the three mobile laser projectors
Full size table
Table 2 Speckle contrast (R, G, B) and photometric speckle contrast (W) of the three mobile laser projectors
Full size table

This procedure requires the following conditions for creating the perfect virtual signal pixels.

  1. 1.

    R, G, B laser beams should be perfectly combined.

  2. 2.

    R, G, B output signals should rise and fall in the same timing, synchronized with scanning.

  3. 3.

    R, G, B output signals should rise and fall in the same manner.

3.2 Image patterns

A white-and-black wide line pattern and grille patterns (much narrower white-and-black line pairs) were used for analyzing the edge color shift. The wide line pattern was a single line of a width of 18.85 pixels. The grille patterns of a width of 3.77 pixels and a width of 1.88 pixels were also used. (The nominal unit pixel value was calculated from the pixel resolution of 1280 × 720 (16:9). The above three patterns are shown in Fig. 2.

Fig. 2
figure 2

Three image patterns

Full size image

3.3 Measurement setup

The measurement setup is shown in Fig. 3. The line patterns were projected on a standard diffusive reflectance screen, SRT-99–120 (Labsphere). The measured 2D data (350 × 300 sensor pixels) were obtained by the speckle measurement device, SM01VS11, provided by OXIDE Corporation [15,16,17]. The schematic internal structure of the measurement device is illustrated in Fig. 4. The 2D data were captured by the 2D image sensor (EMCCD: electron multiplying charge-coupled device) with a pixel size of 16 × 16 μm2. A pinhole (iris) with a diameter of 1.2 mm was inserted in front of an imaging lens with a focal length of 135 mm to have optics with the same MTF as the human eye. The measuring device was also equipped with color filters. The 2D R, G, B monochromatic speckle data could be extracted by filtering out the other color data from the white data. The 2D color speckle data of the white patterns could be calculated by the R, G, B monochromatic data using Eq. (1). The wavelengths and the relative optical powers of the R, G, B laser diodes were measured directly without the screen by an RGB laser illuminance meter, TM6102 provided by HIOKI E. E. Corporation [18]. The illuminance meter was set at the position of the measured patterns. The illuminance meter has a function of synchronizing its sampling time with an integral multiple of the refresh rate of the projector.

Fig. 3
figure 3

Measurement setup

Full size image
Fig. 4
figure 4

Schematic internal structure of the speckle measurement device

Full size image

The projectors were set normal to the screen and the speckle measurement device was set slightly shifted by angle θ from the projector axis to avoid the mutual shadowing problem. The dimensions were D1 = 710 mm, D2 = 390 mm, and θ = 20.6º.

3.4 Minimization procedure of the speckle effect

The speckle effect was minimized by opening the iris of the speckle measurement device [5]. This method can reduce speckle contrast down to the values less than 0.02, as the same level as LEDs. To eliminate the residual noise completely, the data in the same column group of the 2D sensor were averaged.

An example of the speckle minimization procedure is shown in Fig. 5. The data were obtained using the #3 projector. The graph on the left shows the speckle illuminance distribution data normalized by the illuminance value of the original white point. The graphs in Fig. 5 were plotted in the spatial domain. The horizontal axis represents the column number of the 2D image sensor. The white chromaticity point was measured in the wide uniform white region. The sensor column direction was aligned exactly with the direction of the line pattern. The plotted speckle data in the same column group was supposed to be projected at the same illuminance level. The photometric speckle contrast Cps was measured to be 0.045. The graph in the middle shows the case for the iris opening. The Cps value was measured to be 0.018, still affected by the background optical noise. The graph on the right shows the result of eliminating the noise effect completely by averaging the residual distribution in each column group.

Fig. 5
figure 5

Example of the speckle minimization procedure

Full size image

4 Measurement results

4.1 Data plot in the color space domain

In the color space domain, the color speckle and the edge color shift data were plotted in the 3D color coordinate system with the CIE 1931 chromaticity coordinates (x, y) in the horizontal plane and the vertical axis representing the normalized illuminance values. The illuminance values were normalized by the illuminance value at the original white point. We tentatively call it as the xyYNORM color space throughout this paper. In general, the color speckle distribution of a uniform white region has been plotted on the 2D chromaticity coordinates such as CIE 1931 or CIE 1976. However, the illuminance axis is necessary for the measuring the edge color shift and the grille pattern contrast (contrast modulation) because illuminance rapidly changes between the bottom black level and the top white level along the pattern edges. Half of the illuminance data in the uniform white regions statistically exceed the original white point (YNORM = 1.0). From this viewpoint, CIELAB space used in our previous work [1] is not appropriate for the color speckle analysis because the values exceeding the white point (L* = 100) are not allowed in this color space.

4.2 Edge color shift of pattern I

Figure 6 shows the spatial data plots of the illuminance of the white, and those of the R, G, B laser outputs using Pattern I for the projectors, #1, #2, and #3. The color speckle effect was eliminated by the procedure in 3.4. As in Fig. 5, the illuminance data were normalized by the value at the original white point. They are plotted in the spatial domain, with respect to the column number of the 2D image sensor.

Fig. 6
figure 6

Spatial plots of color shift for projectors #1, #2, and #3 (Pattern I)

Full size image

The R, G, B spectral power ratios were kept constant in the black-bottom and the white-top regions. However. the R, G, B laser outputs were not synchronized with each other in the edge regions, showing a unique complicated color shifting behavior. For the projector #1, the edge color shift was purple at the left edge and blue-green at the right edge. On the other hand, the color was blue-green at the left edge and purple at the right edge for the projectors #2 and #3. The range of the edge color shift was widest for projector #2, and narrowest for projector #3.

As the next step, the color shift behaviors were expressed more in detail in the color space domain using the xyYNORM color space. Figure 7 shows the color space plot of the color shift behavior with the color speckle effect for projector #1. The viewgraph on the left shows the 3D bird’s eye view of the data plot in the xyYNORM color space. The viewgraphs on the right show the data projections on the x–y plane, the x-YNORM plane, and the y-YNORM plane, respectively. As in Fig. 7, the color speckle distribution makes the color shifting behavior blur seriously even if the monochromatic speckle contrasts Cs-R, Cs-G, Cs-B are less than 0.1 (See Table 2).

Fig. 7
figure 7

Color space plot of color shift behavior with color speckle effect (Pattern I for projector #1)

Full size image

Figure 8 shows the data plot in the xyYNORM color space of the pure color shift without the color speckle effect obtained by the speckle minimization procedure in 3.4. By eliminating the color speckle effect, we could successfully make the pure color shift behavior perfectly visible as in Fig. 8. The color shift trajectory on the x–y plane (CIE1931 chromaticity coordinates) is almost linear along the line between the purple and the blue-green points through the white point. The maximum color shift on both sides occurs around at the normalized illuminance level of 0.4.

Fig. 8
figure 8

Color space plot of pure color shift behavior (Pattern I for projector #1)

Full size image

Similar data plots in the xyYNORM color space are shown in Figs. 9 and 10 for projector #2. For projector #3, they are shown in Figs. 11 and 12. Projectors #2 and #3 showed the edge color shift behaviors similar to that of projector #1 except that the shifted color on the left and the right side was reversed, and the maximum edge color shift occurred around the normalized illuminance level of 0.2. Projector #2 showed the largest color shift range.

Fig. 9
figure 9

Color space plot of color shift behavior with color speckle effect (Pattern I for projector #2)

Full size image
Fig. 10
figure 10

Color space plot of pure color shift behavior (Pattern I for projector #2)

Full size image
Fig. 11
figure 11

Color space plot of color shift behavior with color speckle effect (Pattern I for projector #3)

Full size image
Fig. 12
figure 12

Color space plot of pure color shift behavior (Pattern I for projector #3)

Full size image

4.3 Edge color shift of Pattern II and Pattern III

The edge color shift behaviors of Patterns II and III are shown only for projector #2 which showed the widest range of color shift.

Figure 13 shows the spatial data plots of the white illuminance and the R, G, B laser outputs, using Pattern II and III for projector #2. The color speckle effect was eliminated by the procedure described in 3.4 as well. The grille patterns consisting of repeating white-and-black line pairs (e.g., Patterns II and III) have been widely used for image resolution measurements. The white and the R, G, B outputs for a single period of a black-and-white line pair is shown in Fig. 13. The center of the graph is set at the white line center. On the left of Fig. 13, the spatial data plot for Pattern II of a width of 3.77 pixels is shown. The data could not reach the top white level of 1.0 or the bottom black level of 0,0, because the output responses of the R, G, B laser diodes were not fast enough to catch up with the narrow line width. That is, the spatial domain resolution is related to the response of the laser diode in the time domain via scan speed. On the right of Fig. 13, the spatial data plot for Pattern III of a width of 1.88 pixels is shown. The plotted data exist within the much smaller normalized illuminance range of 0.1, implying that the image resolution is much lower. That is, it is much more difficult for the width of 1.88 pixels to distinguish the white-and-black signals.

Fig. 13
figure 13

Spatial plots of color shift of Patterns II and III for projectors #2

Full size image

Figure 14 shows the xyYNORM color space plot of the color shift behavior with the color speckle effect for Pattern II. The same graph format as in Figs 7, 8, 9, 10, 11, 12 is used. The color speckle distribution took the shape of a tilted torus thicker in the higher illuminance region. Figure 15 shows the xyYNORM color space plot of the color shift behavior without the color speckle effect. The trajectory of the pure color shift on the x–y plane (CIE1931 chromaticity coordinates) was elliptical, although that for Pattern I was linear.

Fig. 14
figure 14

Color space plot of color shift behavior with color speckle effect (Pattern II for projector #2)

Full size image
Fig. 15
figure 15

Color space plot of pure color shift behavior (Pattern II for projector #2)

Full size image

Figure 16 shows the xyYNORM color space plot of the color shift behavior with the color speckle effect for Pattern III. The color speckle distribution was shrunk and was not like a torus anymore. However, the xyYNORM color space plot of the pure color shift behavior without the color speckle effect showed a smaller but clear elliptical trajectory on the x–y plane, as in Fig. 17.

Fig. 16
figure 16

Color space plot of color shift behavior with color speckle effect (Pattern III for projector #2)

Full size image
Fig. 17
figure 17

Color space plot of pure color shift behavior (Pattern III for projector #2)

Full size image

5 Discussion

Firstly, the linear trajectory on the x–y plane of the edge color shift behavior of the wide white line (Pattern I) is discussed. The trajectory started from the black line region where the chromaticity coordinates are same as the original white point. Then the color shift occurred along the pattern edge on the left side with increasing illuminance. The trajectory moved to the furthest chromaticity point and moved back linearly to the original white point of the highest illuminance value. After that, it returned to the black region along the pattern edge on the right side in the opposite chromaticity direction with decreasing illuminance via the furthest chromaticity shift.

On the other hand, the elliptical trajectories for Patterns II and III (grille patterns) varied between a non-return-to-zero black level and a non-reach-to-one white level. As in Fig. 13, the red laser power of the previous grille line remained with delay at the start point on the left side, and the green laser power slightly advanced at the start point. Therefore, the chromaticity point at lower illuminance level moved toward yellow from the original white point on the x–y plane. Around the illuminance peaking level, the red power was not peaking yet, which implies that the chromaticity point moved toward blue-green from the original white point. As a result, the trajectory took an elliptical shape.

The spatial image patterns of the raster-scan RGB mobile projectors are created as always moving virtual pixels by the spatial domain MEMS scanner and the time domain response of the R, G, B diode lasers. As already shown in 3.1, the following conditions are necessary for creating the perfect virtual signal pixels.

  1. 1.

    R, G, B laser beams should be perfectly combined.

  2. 2.

    R, G, B output signals should rise and fall in the same timing, synchronized with scanning.

  3. 3.

    R, G, B output signals should rise and fall in the same manner.

It is difficult for the human eye to recognize the color shift inside a large-area uniform white pattern, because the imperfectly combined R, G, B laser beams are averaged by the continuous beam scanning at a high speed. Therefore, the color shift is usually observed at the pattern edges. Furthermore, the pattern edges are created by the rise or fall of the R, G, B laser diodes at higher speed in the time domain, which is converted into the spatial domain by scanning as in Figs. 6 and 13.

If the white beam had a spatially asymmetric color structure, the color on the beam left would be emphasized along the left pattern edge, and the color on the beam right would be emphasized along the right pattern edge. It is against condition (1). The other possible cause of the edge color shift is asynchronized rise/fall timing among the R, G, B laser diodes, which is in the time domain. It is against condition (2). However, we cannot determine which cause is dominant only by this measurement method in this work. The color-by-color time domain measurement using high-speed oscilloscope [19] might be useful for further analysis of determining the root cause of the edge color shift.

The complicated chromaticity variation in the edge regions occurs because of the continuous variation of the R, G, B irradiance values (the R, G, B spectral power ratios). The white illuminance and the chromaticity were calculated by Eqs. (1)–(3) using the R, G, B irradiance values. That is, the change of the R, G, B irradiance values is closely related to the rise/fall response behaviors of each laser diode. The response of laser diodes depends on semiconductor material, wavelength, cavity structure, transverse/longitudinal mode stability, bias level, and temperature characteristics, which are, in general, different for each R, G, B laser diode. It is against condition (3). Therefore, we can conclude that the larger edge color shift behavior is dependent on how much it is further from the ideal conditions (1), (2) and (3). As a result, projector #2 showed the largest color shift, and projector #3 showed the smallest color shift and the best resolution.

The edge color shift of raster-scan RGB mobile projectors are clearly observed for the wide line patterns. However, for the low-resolution narrow line patterns, the edge color shift becomes vague by the averaging effect of low grille pattern contrast for Pattern III in Fig. 17. For the zero resolution with much narrower grille pattern, in fact, the edge color shift was confirmed to converge on the chromaticity coordinates close to the original white point because no pattern edges were observed (not shown here).

Compared with the longest axis of elliptical-shaped color shift trajectory for Pattern II in Fig. 15, the linear length of color shift for Pattern I in Fig. 10 looks slightly shorter. This is considered as the result of competition between the color shift increasing effect and the color shift blurring effect. The color shift increasing effect is caused by the delay/advance at the non-return-to-zero black level and the non-reach-to-one white level, whereas the color shift blurring effect is caused by the averaging effect of low grille pattern contrast.

6 Conclusion

Color shift behavior at the edges of white-and-black line patterns was measured and analyzed for three RGB raster-scan mobile projectors #1, #2 and #3. The 2D data of the projected line patterns on a standard diffusive reflectance screen patterns were captured by a speckle measurement device for analyzing the effects of both the color speckle noise and the edge color shift. Three patterns were used for the measurement. Pattern I is a single white line with a width of 18.85 pixels, Pattern II is a grille pattern of a line width of 3.77 pixels, and Pattern III is a grille pattern of a line width of 1.88 pixels.

The color speckle noise blurs the behavior of the underlying edge color shift. To extract the underlying color shift behavior, a new procedure for eliminating the color speckle effect was introduced. The color speckle noise was minimized by opening the iris of the speckle measurement device as the first step. Then the residual noise was averaged as the second step. The color shift trajectories were plotted in the color space consisting of CIE1931 chromaticity coordinates and normalized illuminance (the xyYNORM color space).

The edge color shift moved to purple color at the left edge and to blue-green color at the right edge for projector #1. On the other hand, it was reversed for projectors #2 and #3 (blue-green at the left edge and purple at the right edge). The range of the edge color shift was largest for projector #2 and smallest for projector #3.

In the analysis using the xyYNORM color space, the color speckle makes the color shift behavior blur even if the R, G, B monochromatic speckle contrasts Cs-R, Cs-G, Cs-B are less than 0.1. By eliminating the color speckle effect using the new procedure, we could successfully plot the pure color shift data with perfectly high visibility. For Pattern I, the color shift trajectory on the x–y plane (CIE1931 chromaticity coordinates) was almost linear along the line between the purple and the blue-green points through the white point. On the other hand, the pure color shift trajectories for Pattern II and III on the x–y plane (CIE1931 chromaticity coordinates) took elliptical shapes. The elliptical trajectories are caused by non-return-to-zero black level and non-reach-to-one white level output signals for the narrower grille patterns.

As a result of the analysis, we conclude that the edge color shifts are possibly caused by white beam with an asymmetrically combined R, G, B beam structure, asynchronized driving of the R, G, B laser diodes, or rise/fall response characteristics different for each R, G, B laser diode. Time domain measurements would be necessary for further analysis of the edge color shift.