Keywords

1 Introduction

Mixed reality extends a physically accessible environment with three-dimensional virtual objects [3]. Thanks to Microsoft (MS) HoloLens [11], the mixed reality is more accessible for common use and virtual collaboration [12]. Its built-in depth camera technology allows to scan surrounding environments for physically available objects. The spatial depth detection can be performed by precise real-time mapping using an RGB-D camera [1].

The scan sensitivity of the surrounding environment is affected by global illumination [10]. Therefore, the quality of image detection and spatial mapping depends on the intensity of the light source placed in the physical space [13]. In the case of previous research works [7, 9, 14], MS HoloLens has been evaluated in environments with achromatic (white) light sources, which provide ideal conditions for the detection and scanning [5].

However, with the advent of RGB lighting, it is important to consider using mixed reality equipment under other than achromatic lightning conditions. The light of a specific color may be used for various purposes such as entertainment or research. With respect to the latter purpose, we plan to conduct experiments focused on the human cognition, which we currently perform in a CAVE environment [8], in a more flexible collaborative mixed reality one. Before constructing such an environment, it is necessary to evaluate the performance of the corresponding devices under unusual lighting conditions, provided by RGB light sources.

In this paper we report on a set of experiments where the spatial scanning performance of MS HoloLens has been evaluated under the red, green and blue monochromatic illumination. As it has been observed [6] that the performance of depth cameras in a mixed reality setting decreases significantly when the global illumination is dimmed, we decided to emulate different light intensities by using one, two or three independent light sources. We focused on the luminous flux metering accuracy, maximum scanning distance and physical object detection performance of MS HoloLens. The rest of the paper starts with a short review of related researches in Sect. 2. Section 3 describes the object of the experiments, MS HoloLens, and Sect. 4 the corresponding setup. Section 5 presents the procedures and results of the individual experiments and Sect. 6 summarises the most important findings and outlines plans for more comprehensive experiments.

2 Related Work

The most related work is the study [9], where a series of experiments to quantitatively evaluate MS HoloLens performance has been carried out. These experiments focused on the accuracy and stability of MS HoloLens head posture estimation, its capability to reconstruct a real environment, spatial mapping and speech recognition. While the range of these experiments is greater than of the ones reported here, they all have been conducted under normal ambient indoor lighting with 25 W power output. Another difference is the measurement area size, which is \(5\,\mathrm{m} \times 8\,\mathrm{m}\) in [9] and \(3\,\mathrm{m} \times 3\,\mathrm{m}\) in our experiment. There is one methodological similarity between [9] and this work, namely a use of a reference measuring device for comparison: The work [9] uses OptiTrack motion tracking system to evaluate the head localization process of MS HoloLens while we use a professional luminous flux meter to evaluate similar measurements performed by MS HoloLens. The authors of [9] later focused on an evaluation of MS HoloLens from the user perspective and developed an appropriate quality-of-experience model [14]. This evaluation has been, again, performed under normal indoor lightning conditions.

The work [7] evaluates MS HoloLens with a specific goal in mind, namely to improve the spatial perception for humans with visual impairments by means of the depth mapping. It uses MS HoloLens as a tool for distance-based vision, which can show the user a colored depth scan in a real-time mixed visualization. Each recognized object is colorized in high-contrast with respect to its distance from the user. As a part of [7], the performance of MS HoloLens spatial mapping and object recognition was evaluated under a typical indoor lighting with a finding that the recognition is reliable up to the distance of 3 m from the object. The measurement area was a room with the size \(5.3\,\mathrm{m} \times 3.6\,\mathrm{m}\).

3 MS HoloLens

MS HoloLens is a mixed reality device, combining a head-mounted stereoscopic see-through display with a set of input devices and a processing computer running MS Windows 10 operating system. The input devices are used to sense the surrounding environment, including user actions. They consist of an inertial measuring unit to determine the user position, four microphones for sound capturing, a two megapixel RGB camera for photo and video capturing, an ambient light sensor, a depth camera and a set of four gray-scale cameras. The experiments described in this paper focus on the last three devices.

The depth camera and the gray-scale cameras are used for the spatial mapping, i.e. a reconstruction of the surrounding environment in a form of a 3D model. The depth camera is also used for the collision detection between the real and virtual objects in a mixed environment and has the resolution of \(1024\times 1024\) pixels and the \(120^\circ \times 120^\circ \) field of view. It utilizes an active infra-red illumination for more accurate measurements. The depth is estimated using the time-of-flight (TOF) approach with the speed of 1 to 5 frames per second (FPS) for far-depth sensing and up to 30 FPS for near-depth sensing [2].

4 Experimental Setup

In our experiments we tested MS HoloLens performance using three RGB light sources and a measurement area with the size of \(3\,\mathrm{m} \times 3\,\mathrm{m}\). Our goal was to test the depth camera together with the gray-scale cameras and find out how they perform under monochromatic red, green and blue illumination. To be able to capture the data directly from MS HoloLens sensors in real time we used a special developer mode called HoloLens Research modeFootnote 1. Two experiments have been carried out. The first experiment evaluated the luminous flux metering accuracy of MS HoloLens by comparing it to a professional flux meter. The second one focused on the MS HoloLens performance under a limited lighting, where it measured the maximum spatial mapping distance and the physical object detection speed.

Fig. 1.
figure 1

Arrangement of the cubical indoor space used for the experiments

Full size image

The experiments have been conducted within a cubical indoor space of \(3\,\mathrm{m}\times 3\,\mathrm{m}\times 3\,\mathrm{m}\). The exact arrangement of the space can be seen in Fig. 1, where L1, L2 and L3 are the light sources, PU is the position of HoloLens and P1, P2 and P3 are specific points, used in the experiments. For each of them, 3D Cartesian coordinates are given in meters in the format [x;y;z]. The orientation of the axes is given in the bottom left corner of Fig. 1.

The cubical space has one wall only. It is the rear wall, formed by the front side of a rear projection screen. The white area of the projection screen is \(2.1\,\mathrm{m}\) high and thus covers \(70\%\) of the rear wall. The length between the white area and the floor or ceiling is 0.45 m. The three identical light sources have been placed immediately in front of the screen, so the homogeneous matte surface of the screen provides a diffuse area for the lights. The light sources are LED PAR 56 Floor 36x1W Black RGB spotlights, manufactured by Ignition. As the name suggests, one source contains 36 RGB LEDs, each with 1 W power output. The pulse width modulation of the light sources is 500 Hz, which is more than 16 times higher than the FPS rate of HoloLens cameras. The chair (P3) was used in the second experiment only.

Fig. 2.
figure 2

Experimental setup under all light configurations used

Full size image

Three separate light sources were utilized to be able to adjust the light intensity. While it is possible to dim the lights, we used them on full power, because the color reproduction is negatively affected otherwise. This resulted in 12 different light configurations, shown in Fig. 2. To clearly see the configurations, the individual photos in Fig. 2 have been shot with different negative exposure compensation. Therefore, they do not represent the actual illumination of the area as perceived by the user. From the user point of view, the whole space has been as illuminated by a diffused light. The overlapping borders between the light sources were only slightly visible on the projection screen. To eliminate the effect of external light sources, all experiments have been conducted after dark with darkened windows and all the lights on the surrounding hallways turned off.

5 Experiment Procedures and Results

Each experiment utilized the setup in a slightly different way. The experiment procedure descriptions in this section use the designations from Fig. 1 to identify the position of MS HoloLens and other important locations.

5.1 Luminous Flux Measurement Accuracy

The objective of the first experiment was to compare the light power metering of MS HoloLens with a professional digital luminous flux meter under various RGB lighting conditions. The flux meter used was TES-133, manufactured by TES Electrical Electronic Corp.

Procedure. Both the MS HoloLens and TES-133 were placed at point PU and pointed to the point P1 on the projection screen (Fig. 1). PU was 1.7 m above the ground to emulate the position of MS HoloLens on the head of a person. The position of P1 was 2.5 m above the light L2, where the outputs produced by individual lights merged into one. This merging can be also observed in Fig. 2.

Nine sets of nine measurements were performed. The first three sets used only the red parts of the LEDs of the lights, the sets 4 to 6 used only the green ones and the last three used only the blue ones. During the measurements of the sets 1, 4 and 7 the only light turned on was L2. The sets 2, 5 and 8 used L1 and L2 and the sets 3, 6 and 9 used all three lights. Some measurements were also carried out with the lights L2 and L3 but the results were similar to L1 and L2, so it has been decided to continue with the latter configuration only. During all the measurements the lights that were turned on used all the LED components of the corresponding color at \(100\%\) intensity.

Table 1. Comparison of luminous flux measured by MS HoloLens and TES-133
Full size table

Results. The final results of the experiment are shown in Table 1. The luminous flux values in the \(4^{th}\) and \(5^{th}\) row are averages of all the measurements in the given set. Both TES-133 and MS HoloLens consider the green light as the most intense and the red light as the least intense. For each set s, \(1\le s \le 9\), the difference is computed in percent as the value \(\delta _s\):

$$\begin{aligned} \delta _s = \frac{\sum _{i=1}^{9}({\frac{|\varPhi _i^T - \varPhi _i^H|}{\varPhi _i^T}\times 100})}{9}. \end{aligned}$$
(1)

In (1), \(\varPhi _i^T\) is the luminous flux value from the i-th measurement of the set measured by TES-133 and \(\varPhi _i^H\) is the flux value measured by MS HoloLens. We consider \(\varPhi _i^T\) the accepted value and \(\varPhi _i^H\) the observed value.

Fig. 3.
figure 3

Relation between the number of lights turned on and the difference between the luminous flux metering of TES-133 and MS HoloLens

Full size image

As Fig. 3 shows, in the case of the blue and green lights the difference is decreasing with the increasing light intensity, i.e. the number of lights turned on. In the case of the red lights there is an opposite trend. The measurement sets with the red lights were also the only ones containing cases where the flux value measured by MS HoloLens was higher that the one measured by TES-133. There were three such cases in the set 1, one in the set 2 and two in the set 3.

5.2 Performance Under Low Light

In the second experiment we focused on the MS HoloLens performance under low light. Only one light source, namely L2, was turned on. We were interested in the influence of the light color on the maximum distance, detectable by the 120\(^{\circ }\) spatial sensing system of MS HoloLens and the spatial mapping speed of an object, positioned 3 m far from MS HoloLens.

Procedure. To measure the maximum detectable distance, we placed MS HoloLens at point PU and pointed it to the point P2 (Fig. 1). The chair at P3 was removed, so there were no objects between PU and P2. The distance from PU to P2 was 4.57 m. Three sets of 30 measurements were carried out. The first set used all the red, the second one all the green and the third one all the blue LED components of L2 at \(100\%\) intensity.

For the spatial mapping speed evaluation we placed a conference room chair at the point P3 (Fig. 1). The chair (Fig. 5 left) has got a black frame and dark blue seat and back cushions. MS HoloLens was placed at PU and pointed to the chair. The distance between MS HoloLens and the chair was 3 m. Three sets of measurements with the same light settings as in the maximum detectable distance case were conducted. There were 10 measurements in each set.

Fig. 4.
figure 4

Maximum detectable distance under low light (Color figure online)

Full size image

Results. As Fig. 4 shows, the distance metering was most successful under the red light with the average of 4.21 m, minimum of 4.11 m and maximum of 4.24 m. However, even in this case, it didn’t manage to reach P2 (4.57 m). The performance under the green and blue light was considerably worse with the averages of 3.45 m and 3.62 m. The variance of the values was higher, too. They ranged from 3.14 m to 3.84 m under the green light and from 3.41 m to 3.87 m under the blue light.

Fig. 5.
figure 5

The chair as detected and seen by MS HoloLens under the red light (left) and the object detection time measurement results (right) (Color figure online)

Full size image

The spatial mapping speed evaluation results (Fig. 5 right) also show the best performance under the red light with the average detection time of 2.4 s. Again, the performance under the green and blue light was similar with the average times being 3.7 s and 3.5 s. However, in this case the variance was highest for the red light where the measured times ranged from 2.07 s to 2.94 s. For the green color the range was from 3.51 s to 3.95 s and for the blue one it was from 3.14 s to 3.74  s.

6 Conclusion

While several experiments evaluating the performance of the MS HoloLens holographic computer have been conducted and documented, none of them considered specific, monochromatic light conditions. The uniqueness of the experiments presented here dwells in the evaluation under monochromatic red, green and blue light. They were performed inside a \(3\,\times \,3\,\times \,3\,\mathrm{m}\) cubical indoor space, which was considered sufficient according to the findings of the previous studies [9] and [7].

The results obtained show that MS HoloLens performs better under the red light. The performance under the green and blue light was worse than under the red one but similar to each other. There were also other variations. Considering the first experiment, the difference between the luminous flux value measured by MS HoloLens and the TES-133 professional meter decreased with the increasing light intensity under the green and blue light, while under the red light it increased. In the case of the maximum detectable distance evaluation there was less variance in the values obtained under the red light than under the green and blue light. On the contrary, the variance in the spatial mapping speed evaluation was higher in the red set and lower in the green and blue set. The different results for the red light may be caused by the use of the infra-red illumination in the MS HoloLens depth camera.

To confirm the findings presented here, we plan to repeat the experiments with considerably more measurements. We also intent to perform similar evaluation in a larger indoor space with an ambient RGB illumination. Regarding the planned utilization of MS HoloLens, the results obtained find the device suitable for the human condition-related experimentation and also for an ongoing research [4] focusing on ambient user interfaces.