Keywords

1 Introduction

People are surrounded by various natural phenomena in the world, and they have found a beauty for many of them. Artists have produced many artworks starting from imitating those natural phenomena. Although what beauty is has been a fundamental question discussed for a long time [1], people have an intuitive understanding of what beauty is and incorporate it around them to enrich their lives.

At the same time, the beauty of nature is more than what usually is seen. People can find beauty in deep-sea and deep-space images that can be seen using the latest technology. This indicates that there is still hidden beauty in nature that is unknown to people. To take out the beauty hidden in nature and make it visible is what we have been working. The basic concept of this paper is based on the idea of how to extract hidden beauty in nature and how to provide it to people in new ways.

We have done the work of using a high-speed camera to extract beauty hidden in natural phenomena that are usually invisible to people and provide it to people in the form of video art. In particular, recently we have been working on filming various fluid phenomena using a high-speed camera and making the beauty hidden in fluid phenomenon into video art [2]. Although the natural phenomenon is three-dimensional, we have expressed it in the form of two-dimensional images. As a next step, we have tried to showcase the created beauty by visualizing it into three-dimensional materials.

This paper has the following structure. First, in Sect. 2, we will explain fluid art which is to be visualized as three-dimensional material. We also discuss beauty in fluid art and mention what Japanese beauty is. Next, in Sect. 3, we will discuss related research on the 3D reconstruction technology required to convert fluid art into 3D. In Sect. 4, the method of making fluid art into 3D material is described. Finally, Sect. 5 summarizes the paper including future research directions.

2 Fluid Art

2.1 Fluid Dynamics and Fluid Art

The behavior of fluids accounts for a large part of natural phenomena. Typical examples are airflow, water flow, wave behavior, etc. Research on the movement of fluid is an important research subject in physics called “fluid dynamics” [3]. Visualization of fluid phenomena makes it possible to intuitively understand how fluid behaves under various conditions. With visualization techniques, people know that fluids create extremely beautiful shapes under various conditions.

Beauty is a very basic element that constitutes art. Therefore, it is natural to think of using fluid dynamics as a basic methodology for art production. The art created based on fluid dynamics is named “fluid art” by us.

2.2 Generation of Fluid Art

2.2.1 Fluid Art Generation System

As a basic technique for creating fluid art, we have developed a method for shooting the liquid form that is created by being given the sound vibration with a high-speed camera. High-speed cameras have been used to shoot various phenomena such as explosions that occur in a very short time, but most of their uses are in scientific and technological experiments. On the other hand, we were interested in producing various beautiful organic shapes such as “milk crowns”. Then, we found that a fluid such as paint could be vibrated by sound to create a flower-like shape. Figure 1 shows the fluid art generation environment [4]. We put a speaker facing upwards, put a thin rubber film on its top, put fluid such as color paints on the rubber film, vibrate the speaker with the sound, and shoot the phenomenon with a high-speed camera. A high-speed camera with 2000 frames/sec is used here. A PC connected to the speaker produces various sounds and vibrates the speaker.

Fig. 1.
figure 1

Fluid art generation system.

Full size image

2.2.2 Fluid Art “Sound of Ikebana”

By using the fluid art generation system, we systematically changed the shape of the sound, the frequency of the sound, the type of the fluid, the viscosity of the fluid, etc. Based on these experiments we found that beautiful forms were generated. One of us, Naoko Tosa, created a video art called “Sound of Ikebana” by editing the video images thus obtained according to the colors of the Japanese seasons [2, 4]. Figure 2 shows a scene of the artwork. An interesting issue is that, while she exhibited the video art in Japan and abroad, many people expressed that “in her artwork, Japanese beauty is expressed” or “Japanese beauty is included in her artwork.”

Fig. 2.
figure 2

A scene from “Sound of Ikebana.”

Full size image

2.3 What Is Japanese Beauty?

What is the essence of Japanese beauty? As has been pointed out by many people such as Bruno Taut, Japanese artworks and architectures have always emphasized the sense of unification with nature [5]. Then, it can be said that Japanese beauty is not something created by the Japanese, but that it is the beauty hidden in nature and that the delicate and sophisticated parts that appeal to the Japanese sensitivity are taken out. The methodology of extracting the beauty hidden in natural phenomena using advanced technology based on the artist’s sensibility fits this Japanese beauty concept very well. This seems to be the reason why many people feel Japanese beauty in Naoko Tosa’s artworks.

At the same time, this is not only the characteristics of her artworks but also those of other Japanese artists. Many Japanese artists have found beauty in the natural phenomena such as the flow of the river and the waves that scatter, and have finished them as artworks. For example, in Ikebana the basic shape (“type”) is an asymmetric triangle [15] (Fig. 3 left). Interestingly, the shape of “Sound of Ikebana” created by Naoko Tosa is similar to the “type” of Ikebana. (Figure 3 right).

Fig. 3.
figure 3

Comparison of Ikebana type (left) and “Sound of Ikebana” (right).

Full size image

What does the similarity between these artworks or “types” of Japanese beauty and natural and physical phenomena mean? Perhaps Japanese artists used their genius or mind-eye to discover the beauty hidden in nature that the general public cannot see, which may have been evaluated as Japanese beauty.

2.4 3D Modeling of Sound of Ikebana and Its Application

When “Sound of Ikebana” was exhibited in Japan and abroad, it was evaluated as “expressing Japanese beauty.” At the same time, many people have requested that “I want to see ‘Sound of Ikebana’ as a three-dimensional object.”

”Sound of Ikebana” is based on natural and physical phenomena, and it contains organic shapes that were not created by conventional human designs and modeling. There are three-dimensional objects such as public art, daily necessities such as plates and cups, cars and aircraft, and even buildings around us. Up until now, artists, designers, engineers, architects, etc. have created appropriate shapes for these objects. Those shapes were created based on the artists’ and designers’ sensitivity and creativity, such as a car design that reduces air resistance, and an architectural design that matches the cityscape. All of these are shapes created by humans, and are based on straight lines and simple curves, and tend to give an inorganic or artificial impression. However, as the maturity of culture increases and the aging society arrives, it is requested to incorporate not only artificial shapes but also organic and diverse modeling of nature into various products around us. Realizing “Sound of Ikebana” as a 3D model enables the following applications (Fig. 4).

Fig. 4.
figure 4

Application examples of 3D modeling of “Sound of Ikebana” (Left: Art sculpture, Middle: Vase, Right: Future architecture).

Full size image
  1. (1)

    Unprecedented organic art sculptures can be created.

  2. (2)

    New modeling design can be introduced to daily necessities such as bowls and vases.

  3. (3)

    A new architectural design with an organic shape can be introduced to the current architecture that is based on straight lines.

Also, by further discussing with CAD/CAM related engineers, another possible application is,

  1. (4)

    3D modeling with a natural shape can introduce a new design method to the design of cars and aircraft, which is so far based on the use of spline curves, etc.

To achieve this, it is necessary to achieve the 3D model reconstruction of physical phenomena and the transformation of the 3D model into a solid object. Section 4 describes the current status of our research progress regarding these issues.

3 Related Research on 3D Reconstruction

To measure the shape of a three-dimensional object and reconstruct it in the form of a 3D model, etc., there are two methods; a passive method of photographing an object with cameras and an active method of projecting laser or structured light on a target object.

A typical passive method is to reconstruct the surface shape of the target object using multiple camera images taken from different viewpoints [6, 7]. In this case, it is possible to use a relatively simple system as compared with the active method. Three-dimensional reconstruction from multiple viewpoints usually consists of three steps: image capturing, camera calibration, and precise shape reconstruction.

3D reconstruction from multiple viewpoints is based on the method of shooting target images with multiple cameras fixed around the target. With this method, camera calibration can be performed in advance, and the internal and external parameters of the cameras can be obtained with high accuracy and stability. Furthermore, the three-dimensional reconstruction of the moving object is possible by synchronizing the cameras and taking images. There is also a method of fixing the camera to a robot arm and moving the robot arm or fixing the camera or target object to a rotary table and moving the rotary table to take multi-viewpoint images [6], but by this method, it is not possible to shoot a moving object. Also, recently, as a shooting method with fewer restrictions on shooting, free-moving shooting with a monocular camera has been used [7]. With this method, there are few restrictions on shooting, and it is possible to perform 3D reconstruction of a large scale object such as an entire building, or even a city, as well as a small object as a table. With this method, it is difficult to perform camera calibration in advance, but in recent years, feature-based image matching such as SIFT [8] and SURF [9] has enabled stable image matching between multi-view images.

Active three-dimensional shape measurement is a method of recovering the shape of a target object from a single image by projecting a fixed pattern light onto the target object and performing high-speed projection or high-speed shooting. This method has attracted attention as a method capable of measuring the 3D shape of a moving object.

As one of them, there is a method called TOF (Time-of-Flight), which measures three-dimensional information by measuring the reflection time of the pulsed light applied to the target object for each pixel. A typical example of this is Kinect [10], which Microsoft used to sell as a product until recently. Kinect irradiates pulse-modulated infrared rays and measures the distance to the target based on the time delay of the reflected light from the target. Since it is relatively inexpensive, it was widely used not only for applications such as games but also for research related to human-computer interaction as a device that can easily measure human movements.

4 Generation of 3D Model of Fluid Art

4.1 Basic Concept and Method

As mentioned in Sect. 2, the fluid art “Sound of Ikebana” created by one us, Naoko Tosa, has been evaluated as “Japanese beauty is expressed”. By exploring the reasons for this, it is necessary to investigate the essential problems of what beauty is and also what Japanese beauty is. As part of this research, one idea is to make “Sound of Ikebana,” which is currently obtained as video art, into a three-dimensional object. Then, as it is possible to observe its shape from various directions, it is possible to get impressions and comments from many people, including art critics. Also, as described in Sect. 2, there will be various application areas of “3D Sound of Ikebana.” Based on this idea, we decided to carry out the 3D reconstruction of “Sound of Ikebana” and the 3D materialization. This Chapter describes the method of 3D reconstruction and the results.

Regarding 3D reconstruction, there are passive methods and active methods as described in Sect. 3. Since the created form can be seen only by using a high-speed camera of 2000 frames/second, the problem we challenge is a three-dimensional reconstruction of an extremely fast-moving object. Regarding the three-dimensional reconstruction of a high-speed moving object, an active method, that projects structured images using a high-speed projector of 1000 frames/second and capturing them with a high-speed video camera have recently been proposed [11]. However, the 1000 frame/sec projector is not commercially available as a mass-produced product at this time, and the resolution is not sufficient. Furthermore, the phenomenon of “Sound of Ikebana” occurs in a small area of approximately 10 cm3, and it is difficult to accurately project in such a small area.

On the other hand, when using the passive method, there are methods: a method that uses multiple still cameras and a method that uses multiple high-speed video cameras. Although a method of performing synchronous shooting using multiple still cameras can build a shooting system relatively inexpensively, the timing of shooting becomes a problem. Forms in “Sound of Ikebana” can be seen by shooting with a high-speed camera of 2000 frames/second, and at the same time, the timing when beautiful forms that can be used as a basis of 3D reconstruction is extremely limited. It is necessary to select relevant images by watching a huge number of shot images. Furthermore, in the case of materializing from a 3D model, the moment that can be selected is more limited because it is necessary not only to consider the beauty of the selected image but also the preservation of the obtained 3D object. This means it is very difficult to use multiple still cameras and to decide the adequate shutter timing based on human intuition.

In fact, in this research, we first set up a system using multiple still cameras and conducted various experiments to shoot, but it was extremely difficult to obtain a satisfying shooting result necessary to obtain beautiful modeling depending on the timing of pressing the shutter.

Based on the above consideration and preliminary experiments, we decided to adopt a passive method based on shooting “Sound of Ikebana” by using multiple high-speed cameras to obtain its 3D model.

4.2 Camera Settings

Figure 5 illustrates the setting for shooting. Also, Fig. 6 shows the actual setting. The system consisting of multiple high-speed cameras is called MEMRECAM manufactured by NAC. The camera part (Mcam V004) has 2 M pixels and a shooting speed of 2000 frames/sec. We placed 6 to 8 cameras around the speaker, where color paints are jumped up by sound vibration. Our final target is to shoot from 360 degrees to perform a complete 3D reconstruction, but as a first step, we used 6 to 8 high-speed cameras. We placed the high-speed cameras so that the speaker was surrounded by them about 120 to 180 degrees.

Fig. 5.
figure 5

Shooting system using eight high-speed cameras.

Full size image
Fig. 6.
figure 6

Actual setting.

Full size image

4.3 Three-Dimensional Shape Reconstruction by Phase-Only Correlation Method

The reason why a passive method was adopted for 3D shape reconstruction from 2D images was explained in 4.1. Regarding the passive method, we surveyed the technology in Sect. 4, but here, as a new method compared to the conventional method, the Phase-Only Correlation method [12, 13] was adopted. Here, the method will be briefly described.

3D shape reconstruction by the passive method consists of (1) estimation of 3D points using image matching, and (2) generation of 3D models from 3D points. Among these processes, the image matching method for estimating 3D points has a significant effect on the accuracy, robustness, and computational cost of the multi-view stereo algorithm, and is an important factor that determines the algorithm performance.

In conventional multi-view stereo algorithms, image matching based on Normalized Cross-Correlation (NCC) [14] is used. In this algorithm, image matching is repeatedly performed while discretely changing the coordinates of the 3D points, and the 3D coordinates having the highest matching score are used as the points for reconstruction. To perform highly accurate 3D reconstruction, it is necessary to change the coordinates with a very fine step size, which causes a problem that the calculation cost becomes huge.

On the other hand, Phase-Only Correlation (POC) [12, 13] is an image matching method that uses only the phase information of the images. The two images are Fourier transformed and the POC function is calculated from the respective transfer spectra. If the transformation between the two images is only based on translation, it is possible to define an ideal peak model of the POC function. At this time, the height of the peak of the POC function corresponds to the similarity between the images, and the peak coordinate of the POC function corresponds to the parallel movement region between the images. When applying POC to multi-view stereo, k stereo pairs are made from one reference viewpoint and k neighboring viewpoints, and POC is used for local window matching between each stereo image.

At this point, by estimating the coordinates of the 3D points from the peak model of the POC function, the 3D coordinates can be estimated with higher accuracy than the conventional NCC-based method. This is particularly effective in a camera setting whose baseline is relatively narrow like the camera setting of our experiment. To perform 3D reconstruction of a very complicated shape such as the one in “Sound of Ikebana,” it is necessary to set the cameras with multi-viewpoints close to each other and shoot under the camera setting with a narrow baseline. On the other hand, if the baseline becomes narrow, the influence of the image matching error on the error of the 3D shape becomes large, so a highly accurate image matching method is required. Therefore, in this experiment, we decided to use a method based on the POC method. The POC method has already been commercialized with the name “TORESYS 3DTM” by Toppan Printing Co., Ltd., and in this experiment, we performed 3D reconstruction using the software. A normal object, which is mainly composed of a convex surface, can be reconstructed into a 3D model with relative ease, but “Sound of Ikebana” has a very organic and complicated shape, and it is difficult to reconstruct its 3D model. Specifically, it was necessary to cope with the following problems.

Problem 1: Restoration accuracy is low when the number of high-speed cameras is small.

Solution 1: It was confirmed that the accuracy was improved by increasing the number of cameras from 2 to 4 and 8. In the installation layout of the camera, the restoration width is minimized to increase the restoration accuracy. (This means more cameras are required to restore the entire image.) The parameters for processing were fine-tuned to make it easier to restore the 3D shape.

Problem 2: Restoration accuracy cannot be improved unless imaging can be performed with the fluid having fine feature points.

Solution 2: When a gold paint is used, the material of the paint appears on the surface like a pattern (roughness), and good restoration results are obtained. Moreover, To reduce the overexposure of the image, the lighting was covered with tracing paper to diffuse the light.

Problem 3: If the object in the image is blurred, the accuracy of 3D rendering decreases.

Solution 3: The restoration result is good for an image in which the entire surface of the subject is in focus. But it is difficult to obtain a good stable result. It is necessary to consider the adjustment such as the squeezing of the camera to increase the depth of field and the adjustment of the intensity of the illumination.

Figure 7 shows a 3D model of “Sound of Ikebana” obtained as a result of various trials considering the above issues. Further, Fig. 8 shows the mesh model.

Fig. 7.
figure 7

3D model of “Sound of Ikebana.”

Full size image
Fig. 8.
figure 8

Mesh pattern of “Sound of Ikebana” 3D model.

Full size image

4.4 The 3D Materialization of Fluid Art

As a next step, we attempted to make “3D Sound of Ikebana” by a 3D printer using the 3D model shown in Fig. 8. As the 3D printer, a commercial 3D printer (3DUJ-553), which is mainly used for the production of figures and miniatures was used. The obtained 3D object is shown in Fig. 9.

Fig. 9.
figure 9

“Sound of Ikebana” obtained as a 3D object.

Full size image

It can be seen that the three-dimensional shapes shown in Fig. 7 and Fig. 8 are reproduced with certain accuracy. The shape of “Sound of Ikebana” is extremely organic, and there are several thin and elongated columnar parts. Because of this, it was difficult to reproduce it with a 3D printer at once. Also, we had to consider the issue of safe transport. Therefore, we created the 3D shape of Fig. 8 by dividing the 3D model into several parts, created them as individual parts with a 3D printer, and combined them. This time, the work of dividing it into multiple parts was done manually. How the entire 3D model with a complex shape could be optimally divided for a 3D printer is an important research theme.

5 Conclusion

We have been making artworks called “fluid art” utilizing fluid dynamics based phenomenon. One of such artworks called “Sound of Ikebana” has been evaluated as “expressing Japanese beauty”, and at the same time as “if it is made into a 3D object, it may have various applications.”

This paper described our attempt to make “3D Sound of Ikebana.” First, we explained about fluid art and investigated why it is evaluated as “expressing Japanese beauty.” We proposed the assumption that fluid art is created by discovering beauty hidden in nature, which has been the traditional art creation process in Japan, and that is the reason why fluid art is evaluated to include Japanese beauty. Also, we explained that if fluid art represented by “Sound of Ikebana” could be made into a 3D object, it has a wide range of applications, not only it can be exhibited as a new art in public spaces, but it can be applied to design everyday items such as cups and vases. There is also the possibility of applying it to new organic designs such as architectures, cars, and trains.

Next, we described how to develop “3D Sound of Ikebana.” For 3D shape reconstruction, there are two methods; a passive method and an active method. Based on our survey on these technologies we decided to adopt the passive method for reconstructing a 3D model from multiple 2D images. Furthermore, among various passive methods, we have adopted the Phase-Only Correlation method, as it has several advantages over other methods. We carried out experiments using multiple still cameras but found it is difficult to shoot the phenomenon at relevant shutter timing. Therefore, for the actual experiment, we have developed a system consisting of multiple high-speed cameras and succeeded in capturing beautiful multiple scenes of “Sound of Ikebana.” Then using the Phase-Only Correlation method, the 3D model of “Sound of Ikebana” was reconstructed and then by using a 3D printer it was materialized as a 3D object.

Our research and experiment are still in its early stage and we will continue to study how to improve the quality of the obtained 3D fluid art. Also, we will pursue how to apply the 3D materialized fluid art to various areas in our society.