1 Introduction

The aerial display technology gives a sense of reality by displaying images in the air and enables us to perceive the real objects and scenery while observing the aerial images. Aerial displays are used as eye-catching content in digital signage, concerts, and attractions. An aerial imaging by retro-reflection (AIRR) is one of the methods to form such an aerial floating image [1]. An aerial image can also be formed by other aerial display methods using a dihedral corner reflector array (DCRA) [2] or a crossed slit mirrors array [3]. Several aerial display technologies include Pepper’s Cone [4] which observes a virtual image projected on a plastic cone, and devices which use fog as a screen to project images [5]. The AIRR is characterized by the use of a retro-reflector to form a real image that has a wide viewing angle of well over 120 degrees [6]. Various usages of AIRR have been proposed, mainly in the fields of amusement and advertising. For example, aerial display expressing contents with depth [7, 8], aerial display where the content changes depending on the viewpoint [9], multiple aerial images from one light source [10], image floating on a crystal ball [11] and omnidirectional aerial display [12, 13] can be mentioned. Using the polarization-processing display as the source display in AIRR, it is possible to apply it to secure display [14, 15]. In these years, from the viewpoint of preventing the spread of infection, a touchless interface that combines an aerial display with a camera or a sensor has attracted attention [16]. In addition to these desktop-sized devices, by surrounding users with a large aerial display, they can be highly immersive without special equipment, such as a head-mounted display or polarization glasses. AIRR has an advantage of being easy to realize such large aerial display by arranging the retro-reflective sheets in tiles. In the previous study [16, 17], we have actually assembled a life-sized device (two meters wide and 1.8 m high) to form large aerial display, and examined the materials suitable for such device and revealed some problems caused by the upsizing. Considering the brightness and contrast of the formed aerial image, it was concluded that a beam splitter, which is an essential component of AIRR, should be made of an acrylic plate compared to the sheet made of PVC (polyvinyl chloride) which is much cheaper in price. Compared to glass, acrylic plate is hard to break and weighs less than half, thus it is easy to increase the size of device safely. However, there was a problem that the formed aerial display was distorted, since the acrylic beam splitter installed at an angle of 45 degrees was deformed by its own weight. This problem has not been resolved yet.

In this study, two new improvements have been purposed. One is to propose a new optical design to reduce the deformation of the acrylic beam splitter of the large size. For this purpose, we have supported the beam splitter with guide wires from the back, and examined how the image distortion was affected by changing the diameter and number of the guide wires and thickness of beam splitter. Another improvement is to propose a new optical design to positively utilize the distortion of aerial display for visual expression. The deformability of the beam splitter can be seen as an advantage of using an acrylic plate, which leads to a breakthrough from flat aerial images. For this purpose, we have generated irregular deflection of beam splitter by pulling it upward with an adjust wire and partially lifting it, and examined how the appearance of aerial image was changed depending on the viewpoint.

A preliminary result of this study was presented at 27th International Display Workshops [18] and at The 10th Laser Display and Lighting Conference 2021 [19]. In this paper, the distortion of aerial image has been quantified and its shape difference was compared horizontally and vertically.

2 Principles

2.1 Optical see-through AIRR

The principle of AIRR is shown in Fig. 1. This arrangement is called optical see-through AIRR [20]. AIRR is composed of three components: a light source, retro-reflector and a beam splitter. Light emitted from the light source is split by the beam splitter into reflected light and transmitted light. The transmitted light is retro-reflected by the retro-reflector and then returns to the beam splitter. This retro-reflected light is also split by the beam splitter. The secondary reflected light converges to the plane symmetric position of the light source with respect to the beam splitter. In this arrangement, user #1 can see the aerial image as a real image. On the other hand, the user #2, who is on the opposite side of the user #1 with respect to the device, can see the same content as a virtual image.

Fig. 1
figure 1

Principle of optical see-through AIRR (aerial imaging by retro-reflection)

Full size image

2.2 Distortion of an aerial image due to deflection of beam splitter

Distortion of an aerial image due to deflection of beam splitter was explained in Fig. 2. When the beam splitter is slantly installed by 45 degrees, the light emitted from the light source follows the optical path drawn by blue solid line and forms an aerial image at the position shown by the blue dotted line. However, when the beam splitter is bent due to its own weight, the retro-reflected light follows the light path drawn by orange solid line, resulting the formation of a distorted aerial image at the position shown by the orange dotted line. From the user’s viewpoint, the horizontal deformation of beam splitter is mitigated by the device frame, thus the deformation of beam splitter is likely to be greater in the vertical direction. Originally, a flat aerial image should be formed by a flat light source, but due to the distortion of the beam splitter, the imaging position shifts, and the aerial image becomes concave seen from the user's viewpoint. As a result, the aerial image has a curvature in the vertical direction, which consequently looks like “the image has shrunk vertically” from the user’s viewpoint.

Fig. 2
figure 2

Distortion of an aerial image due to deflection of an acrylic plate

Full size image

In order to mitigate this deformation of the beam splitter, piano wires (guide wires) are installed so as to support the beam splitter from the back. Both ends of the guide wire were tied to the rings which are bolted to the device frame.

3 Experiment

Figure 3a shows the assembled device to form the aerial image. The size of the device was 1800 mm in height, 2000 mm in width and 1000 mm in depth, seen from the user's viewpoint. In the center of the photo, it can be seen that the beam splitter without a guide wire installed at an angle of 45 degrees bends downward. The user's hand is also shown so that the size of the device can be recognized. As the light source, LED panels are used (2.5 mm pitches, 160 mm × 160 mm). As the beam splitter, transparent acrylic plate (1000 mm × 1280 mm) was used with a thickness of 1 mm, 2 mm and 5 mm. We used a retro-reflective sheets developed for forming aerial display (Nippon Carbide Industries). We adopted a prism-type retro-reflective sheet which can form brighter aerial images rather than commercially used beads-type. 10 guide wires to support the beam splitter were arranged at equal intervals within a range of 1000 mm in width (Fig. 3b). As guide wires, we used piano wires with a diameter of 0.3 mm and a diameter of 0.8 mm. The number of guide wires were reduced according to the experimental condition. Table 1 shows the experimental conditions for the guide wire and beam splitter. To evaluate the distortion of aerial image, we showed a grid pattern on a light source (Fig. 4) and took photographs of its aerial image by a digital camera (Nikon D5500, 6000 × 4000 pixels) from the user’s viewpoint. Considering camera aberrations, 18 of 27 squares located near the center of the photo were used to evaluate the image distortion. The area in red frames in Fig. 4 was seldomly affected by the camera aberrations. Then, the height and width (pixels) of each square were measured from the images using the software “ImageJ”.

Fig. 3
figure 3

Assembled device to form a life-sized aerial display. a Side view of the device. b A photograph of 10 guide wires taken from the user’s viewpoint

Full size image
Table 1 Experimental condition of acrylic plate as beam splitter and guide wire to support it
Full size table
Fig. 4
figure 4

Grid pattern on LED panel

Full size image

Figure 5 shows the photographs of the aerial image taken under each condition in Table 1. From these photographs, the distortion of squares under each experimental condition was quantified. Figure 6 shows the aspect ratio (height / width) of squares in each condition. In all conditions, the aspect ratio is far less than 1, which means that aerial images are vertically compressed. In condition A, that is an aerial image without guide wires, squares were highly distorted, and their vertical size were reduced almost by half. In aerial images supported by guide wires, aspect ratios of squares were closer to 1, compared to condition A. Aspect ratio of squares was closest to 1 in condition C, which suggests that the distortion of squares was reduced as the number of guide wires was increased. Meanwhile, the variation of aspect ratio between squares was largest in condition C. The comparison of results between conditions B and D showed that the thicker the guide wire, the less the distortion of aerial image. Moreover, a comparison between conditions D and E showed that the aerial image would be distorted and have larger variation if the beam splitter was thin.

Fig. 5
figure 5

Aerial image taken under the experimental condition in Table 1. a no guide wire and 5 mm acrylic plate were used (Condition A). b 4 guide wires of 0.3 mm diameter and 2 mm acrylic plate were used (Condition B). c 10 guide wires of 0.3 mm diameter and 2 mm acrylic plate were used (Condition C). d 4 guide wires of 0.8 mm diameter and 2 mm acrylic plate were used (Condition D). e 4 guide wires of 0.8 mm diameter and 1 mm acrylic plate were used (Condition E)

Full size image
Fig. 6
figure 6

Comparison of aspect ratio (height/width) of aerial images for different conditions

Full size image

In Fig. 7, the spatial information of the distortion of the aerial image is visually expressed using a heatmap. Under each condition, the average value of the width of the square was set as the standard value in both heatmaps of the width and height of squares. The values in the heatmap were obtained as the difference between the measured length (width or height) and the standard value and then normalized by the standard value. From the heatmaps, it was found that the width of the grid near the center of observed area was shortened under condition C. From result of width in condition E, horizontal shrinking and stretching of squares appear alternately when the BS is thin. This is the difference between the area supported directly by the guide wire and the area between the guide wires, and this result is consistent with the comparison between condition D and E in Fig. 6. Under other conditions, there was no large variation or locational bias in width of the squares. As for the height of the squares, all the conditions showed a large negative value, but no apparent variation was observed depending on the location, except for condition E where the upper left corner has a higher deviation. In the comparison between conditions, the condition C had the smallest distortion of the height.

Fig. 7
figure 7

Heatmaps to show the spatial variation of aerial image distortion, focused on the width and height of the squares

Full size image

From these results, it was found that the aerial image appears to shrink in the vertical direction as the beam splitter bends by its own weight, and this distortion can be reduced by the guide wire. Among the conditions using the guide wire in this study, the condition D (4 guide wires of 0.8 mm diameter and 2 mm acrylic plate) seems to show the least variation and spatial bias in the square distortion of the aerial image. In condition C, where the number of guide wires was increased, the aspect ratio of the aerial image has been most improved but the distortion of the aerial image was not reduced as expected. The increase in number of guide wires may result in the reduction of distortion, but as we have manually installed the guide wires without measuring the tensions of each guide wires, the inequality of the tensions of guide wires may cause the additional unfavorable distortion. It may be because it becomes more difficult to equalize the tension between the wires as the number of guide wires increases.

4 Applications

4.1 Deformation of the aerial image by pulling up the beam splitter

In Sect. 3, we have shown that the use of guide wires which support the acrylic plate, which bends by its own weight, can reduce the distortion of aerial image. However, the use of multiple guide wires of unequal tension happens to cause the unfavorable three-dimensional distortion of the aerial image. This implies that there is a possibility of creating three-dimensional projection of the aerial image, which is theoretically focuses on plane, by manipulating the distortion of beam splitter. Deformation of the aerial image by pulling the beam splitter upward was explained by Fig. 8. As shown in Fig. 8, when the beam splitter is placed at 45 degrees, the light emitted from the light source theoretically forms an aerial image at the dotted gray line, which is symmetrical with light source to the beam splitter. Aerial image formed by the deflected beam splitter focuses at the position drawn by the solid orange line in Fig. 8. Here, the beam splitter was supported by guide wires to reduce the distortion caused by the deformation of beam splitter. By pulling up the beam splitter by another wire (adjust wire) from the top, the beam splitter was raised partially and it causes irregular deflection of the beam splitter, and then the resulting aerial image is distorted and moved to the position drawn by the solid blue line in Fig. 8. As explained in 2.2, this aerial image has the depth, therefore its appearance will be largely changed depending on user’s viewpoint.

Fig. 8
figure 8

Deforming the aerial image formed with AIRR by pulling the beam splitter

Full size image

4.2 Observation of deformed aerial image by pulling up the beam splitter

The principal parameters of the system are the same as in Sect. 3. In this experiment, condition D (4 guide wires of 0.8 mm diameter and 2 mm acrylic plate) was adopted. Then an adjust wire (0.8 mm diameter) was used to pull up the acrylic plate. Figure 9 shows photographs of lifting up the center of beam splitter by an adjust wire. A GIF animation of the fireball shown in Fig. 10 was displayed on the LED panel as the light source. Figure 11 shows the observation result of the aerial image when the acrylic plate is supported only by the guide wire from the back. Figure 12 shows the observation result of the aerial image when the acrylic plate is pulled up by an adjust wire. In Fig. 11, the fireball appears to be attached to a point on a flat surface, while in Fig. 12, it appears to be attached to a distorted surface. Therefore, the position of the aerial image also appears to change when the user moves left or right. By shifting the view position of the aerial image, the image was perceived as three-dimensional, while a simple aerial image gave a flat impression. This is an advantage of using acrylic plate, which is relatively easy to deform, for the beam splitter. It is possible to change the appearance of the aerial image depending on the artificially imposed deformation of the beam splitter.

Fig. 9
figure 9

Lifting beam splitter by an adjust wire. a An oblique view of lifting the center of the beam splitter by the adjusting wire. b Aerial image of grids created by the lifted beam splitter, seen from the user’s viewpoint

Full size image
Fig. 10
figure 10

Cropped GIF image of a fireball displayed on a light source LED panel

Full size image
Fig. 11
figure 11

Aerial image formed by use of the acrylic plate supported only from behind, viewed a from the right, b in front, and c from the left

Full size image
Fig. 12
figure 12

Aerial image using a piano wire for hanging the acrylic plate, viewed a from the right, b in front, and c from the left

Full size image

5 Conclusion

We have quantified the distortion of aerial image formed by the deformed beam splitter, by changing the experimental conditions such as the diameter and number of guide wires and the thickness of the beam splitter. The aerial image formed by the beam splitter without a guide wire was largely distorted in the vertical direction according to the principle, and such distortion was reduced by the use of the guide wire that supports the beam splitter from the back. It was found that the aspect ratio of the aerial image gets closer to 1 by increasing the number of guide wires, but at the same time, the spatial variation of the distortion may occur because of the inequality of the tensions of guide wires. Among the experimental conditions we examined in this study, the distortion of aerial image between locations seems to be smallest when 4 piano wires of 0.8 mm diameter and 2 mm acrylic plate were used. Moreover, we have succeeded in displaying pseudo three-dimensional aerial images that appear differently depending on the viewpoint position by utilizing the irregular deflection of the beam splitter. The irregular deflection was occurred by pulling the acrylic plate upward with an adjust wire and partially raising it. The pseudo three-dimensional aerial image can be formed using an easily deformable acrylic plate. One of the prospective applications of this work is to show a virtual fire flame in an electric heater in a three-dimensional form.