Abstract
This paper presents an advanced human-computer interaction system which supports the design of 3D model and makes 3D prototypes by merging technologies such as augmented reality (AR), hand gesture recognition, and 3D printing. This proposed development provides a system to enhance the user’s experience in designing of 3D model (3D prototypes). Beginners or novice designers can design 3D model intuitively. The proposed system provides an ability of manipulating with 3D model in 3D space such as translating, rotating, and scaling. Leap Motion was used to detect and recognize the hand gesture using the skeleton-based algorithm. This device sends the data of hand positions and gesture commands to display the virtual hands on the Microsoft HoloLens. The graphic manager manages the registration between the coordinate frame of Leap Motion and Microsoft HoloLens’ frame. The gesture-based modeling technique allows the user to design and manipulates 3D holographic objects. In addition, the HoloLens application is used to visualize holograms in the real environment’s scale. The designed holographic objects can be assembled, disassembled, interacted with the real environment surface. The holographic objects can be exported into the CAD file format from the mesh rendering to the ASCII STL structure which can be printed by a 3D printer automatically. In the experiment, the holographic objects can be modeled relatively to the physical objects. The system has been tested by eight participants. The purpose of this experiment was to explore the intuitive interaction techniques which facilitated in the designing and validated the relation between physical objects and holographic object. The output of the experiment was the real 3D printed prototype obtained from the designed holographics. The results of the system performance covered the operations such as translation, rotation, and scaling of holographic objects with respective to the actual object, the averaged time of designing, the precision of hand gesture interaction, and the usability.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
The augmented reality (AR) is a technology to display the overlaying of virtual content on the real environment. The AR applications have been utilized for the designing or assembling the real environment, reducing designing time, enhancing the manufacturing processes, and improving the quality. The user can interact with the augmented reality directly and intuitively. Several research works presented the applications on display devices, such as head-mounted displays (HMDs) [1,2,3], handheld devices [4, 5], and projectors [6]. In general, AR systems required the head-mounted displays (HMDs) that provided three main characteristics such as the visualization of combining the real and virtual object into the real environment, the interaction in real time, and the registration of virtual object in 3D space [7]. The optical see-through HMDs provide the AR overlay through a transparent screen without the delay of video image. In the past, the HMDs had limitations of field of view, low frame rate, size, weight, and low resolution. Especially, the low frame rate of rendering holographic objects causes eye fatigue. The Microsoft HoloLens HMD as self-contained holographic computer can render holographic objects at 60 fps that enables the 3D design in a highly realistic environment [8].
In general, 3D models or virtual objects are typically made by complicated 3D modeling software, e.g., SolidWork, AutoCAD, Blender, Maya, and Tinkercad, which provides a simple graphical user interfaces (GUI) and precision of 3D model. Usually, mouse and keyboard are used to interaction with those software but this kind of interaction has a lack of immersion and intuition during modeling. The interaction in AR applications helps the novice to understand the designing and creating virtual object. The hand gesture recognition is one of the interactions which the users can interact with the virtual object in mid-air [9,10,11,12]. The Leap Motion is an input device used to track hand gestures and describe the interaction in millimeter scale [13]. The Leap Motion device has been applied in the virtual reality application for designing of 3D model and manipulating the game objects.
The studies of 3D modeling can be divided into two main approaches. First, sketch-based modeling is the creation of 3D models from 2D sketches. This modeling applies the freeform strokes in mid-air and operations such as extrusion, bending, and cutting [14, 15]. Secondly, geometric modeling or clay sculpting is modification or manipulation of the initial 3D shapes [12, 16]. This approach manipulates 3D shapes by moving, increasing, and removing the components of 3D models such as vertex, line, and surface. In addition, the gesture interaction technique was applied in the sketch-based modeling [14, 15] and geometric modeling [12, 16,17,18]. This proposed research work applied the geometric modeling approach for modifying any 3D model shapes with gesture interaction technique. In addition, several 3D prototypes can be made by those 3D models.
Recently, the 3D printing technology has been utilized to produce 3D prototypes easily. The user required the software tool for designing and customizing variety of the 3D prototype models. The immersive modeling systems using virtual reality techniques have been developed for rapid prototyping of 3D objects. The systems were able to manipulate object’s position, orientation, and size. Furthermore, augmented reality can show the 3D model on a mobile device using AR-marker [19]. However, the previous work could not modify the 3D model intuitively before printing.
Hence, this paper presented the design of 3D model for making a 3D prototype using the augmented reality. The Microsoft HoloLens was used to execute the application, receive the data from the sensors and connect to the Leap Motion and 3D printer. The hand gesture interaction was used to manipulate the 3D model including the translation, rotation, scaling, and modifying of the holographic object. This paper explained the system setup and the interaction method for augmented object and physical object in the actual environment. The results showed the system performance and usability of the proposed system related to the designing of holographic object and its 3D printed prototype.
2 System Overview
In general, the designing of 3D prototype using 3D printer has complicated processes such as, learning how to utilize user interface in CAD software, exporting STL file format from CAD software to Slicer software, setting properties of 3D printer in Slicer software, exporting G-code file format from Slicer software to 3D printer controller, and calibration of 3D printer. The beginner, especially non-technical user, spent more time on understanding those processes. The proposed system simplifies those processes by synchronizing between the applications of 3D model designing and 3D printer. In the proposed system, the hologram was used to display the targeted 3D model for being merged with the real environment while the hand gesture recognition was used for user’s interaction with the 3D model. The Microsoft HoloLens, an optical see-through HMD, was used to display the hologram of 3D model. There are two core components of gestures such as air tap and bloom were used to provide natural interaction [20]. The air tap gesture is tapping index finger down similar to a mouse click. The bloom gesture is holding out the hand with palm up and moving fingertips together. In addition, the combination of gaze and air tap gesture was used as the gaze-and-commit function. The combination of air tap and holding gesture allows the function of click and drag. However, the hand gestures were unable to describe the location in space precisely. The Leap Motion was then chosen to operate the interaction with hologram in millimeter unit and increase more various hand gesture interaction.
The proposed system overview in Fig. 1 shows that the user can design any 3D printed prototypes naturally by visualizing the hologram of 3D model and operating with hand gesture recognition. The system devices consist of the Leap Motion, the Microsoft HoloLens, the actual 3D printed cube, a 3D printer, and a computer. The Leap Motion was attached to the Microsoft HoloLens in order to transfer the gesture commands via web socket. The Leap Motion was connected to the computer with 2.8 GHz Intel Core i7 CPU, 8 GB RAM, and NVIDIA Ge-Force GTX 1050 Ti graphics card via micro USB 3.0. It was mounted on the top of Microsoft HoloLens about 0.05 m with declining angle about 25°. The user wore the head-mounted display for visualization and interaction between the virtual hands and holographic objects. The designed holographic objects can be selected and then were printed through web server automatically. The 3D printer was connected to the computer which had a web server running backend application online. The backend application called the data and file from the web server and then opened the slicer program for printing 3D prototype automatically.
The system consists of three main software parts which are hand gesture manager, graphics manager, and 3D printing manager. The hand gesture manager was used to manage the connection between Leap Motion and the HoloLens application. Leap Motion API was used in hand gesture manager to manipulate 3D holographic object intuitively instead of Gaze-and-Commit interaction from Microsoft HoloLens API. In the preprocessing phase of Leap Motion, it provided the positions and orientations of finger and hand movements. Then, the hand gesture manager sent the virtual hands’ positions and orientations to the graphics manager which took care of the image registration of virtual hand. Vuforia [21, 22] is software for creating the augmented environment by tracking the AR marker in the physical environment for rendering any holographic objects to Microsoft HoloLens. The graphics manager was used to display or visualize the 3D model such as holographic object, virtual hand, and user interface on the Microsoft HoloLens and build a structure of 3D model. Then, the graphics manager sent the structure of 3D model to the 3D printing manager providing the CAD file generation in the form of ASCII STL format by applying the slicing model algorithm [23,24,25,26]. The system converted the structure of 3D model to ASCII Stereo Lithography (STL) standard. Next, the user saved and uploaded 3D printed prototype file from the 3D printing manager to the web server with the timestamp order. The backend application was an application for calling Cura slicer application and uploaded the latest STL file from the database into the Cura slicer application. The Cura slicer converted that file to the G-code and ordered to print the 3D prototype on the 3D printer as shown in Fig. 2.
3 Implementations
3.1 Hand Gesture Manager
The hand gesture manager consists of preprocessing data through Leap Motion API in the Orion version. The hand tracking applied the skeleton-based algorithm that provided the positions and orientations of hands and fingers. The Leap Motion device was mounted on the Microsoft HoloLens. The Leap Motion controller implemented the right-handed coordinate system shown in Fig. 3. The coordinate frame was set at the center of Leap Motion. In this proposed system, Unity3D used the left-handed system so its z-axis was the opposite to the Leap Motion controller’s. The Leap motion API helped to solve this problem. The hand gesture manager provided an ability of manipulating with 3D model in 3D space by six features as shown in Table 1.
3.2 Graphics Manager
The graphics manager was used to display or visualize the 3D models such as holographic object, virtual hands, and user interface on the Microsoft HoloLens. Unity3D game engine and Microsoft Visual Studio have supported the development of HoloLens application. Unity3D game engine allows to build any three-dimensional worlds but it is not a CAD software. In addition, the graphics manager was used to create and deform the structure of 3D model sent to the 3D printing manager.
Setting the Holographic Environment.
The graphic manager took care of displaying the holographic object in three dimensions. The visualization of holographic object in this system required the scene configurations of HoloLens camera, AR camera, and Leap Motion camera. The scene properties of HoloLens camera, the main camera, was configured based on the HoloLens guideline. The coordinate system of HoloLens applied a meter as the unit. The AR camera was the scene camera for Vuforia. The Vuforia library supported the registration of holographic object in real world. The main camera was combined with the AR camera by transformation of the target poses into the Microsoft HoloLens coordinate system. The properties of AR camera, such as app license key, datasets, and rendering behavior of the scene, was configured for providing the position of holographic object related with the AR-maker. Next, Leap Motion camera was added into the scene for displaying the virtual hands. Leap Motion camera was translated in Y axis of Microsoft HoloLens about 0.05 unit and rotated with declining angle about 25°. The pose of Microsoft HoloLens had to be determined relatively with the Leap Motion’s pose via Perspective-n-Point (PnP). In the holographic application, it displayed the virtual hand as shown in Fig. 4.
Mesh Deformation.
The proposed system implemented the vertices mesh deformation based on mathematical operations, combinatorics, and logical connections for mesh manipulation. This mesh modeling provided the cube shape and allowed the gesture commands to manipulate each vertex (Vx, Vy, Vz) of cube in 3D space. The modified cube shape can be translated, rotated, and scaled. The original and modified cubes are shown in Fig. 5.
3.3 3D Printing Manager
The 3D printing manager was used to generate a STL file for 3D Prototype. After the holographic cube modification, the user needed to save 3D model file. In general, Unity3D does not provide the feature for saving 3D prototypes file format. Most of 3D modeling methods for designing 3D prototypes apply the Stereo Lithography (STL) standard. The proposed system created the STL format file from a list of the triangular surfaces which describe a computer generated solid model. Usually, this kind of format has two types which are an ASCII (printable character) format and a binary format. In this research, ASCII STL format was used and its syntax structure can be shown in Fig. 6.
Figure 7 illustrates the process of 3D prototype making. The process starts from creating of the virtual object in CAD software and exporting it into the STL file format. Next, the user opens the slicer software, configurates the work space of 3D printer, opens STL file, slices the 3D model, and saves G-code for printing. In this proposed system, the 3D printing manager received the structure of 3D model from the graphics manager and converted that structure file into a STL file. Next, the STL file was uploaded to the web server as shown in Fig. 8. The 3D printer was connected to the computer via wireless network. That computer also run the backend application for retrieving the STL file from the database of web server. When the backend application found the new list as shown in Fig. 9, it uploaded the file to Cura slicer for obtaining the G-code and printing the 3D prototype on the 3D printer.
4 Experimental Setup and Evaluation
This section presents the effectiveness of the proposed holographic application in real environment. In experiment, the actual 3D printed cube was put on the table as an initial shape for the evaluation of system. The user wore the proposed device, Microsoft HoloLens attached with the Leap Motion, in Fig. 1. The initial user’s position was far from the table by 2 m when the holographic application began. Afterwards, the user could move to any positions. The optimal zone of operating the proposed system was in the range of 0.85 to 2 m from the center of the actual 3D printed cube to the user. The evaluations investigated the system performance and usability.
4.1 System Performances
The system performance was tested including frame rate, the interaction tasks, and visualization of hologram. The frame rate is the most important for visualization of hologram. It has the effects on a hologram display such as shaking, real time operating, and causing of eye fatigue. The Microsoft HoloLens guide recommended the frame rate should be at least 60 frames per second (fps).
In the proposed holographic application, the normal frame rate was equal to 65 frames per second (fps). The frame rate was 60 to 62 fps when the virtual hands and hologram were displayed. Further, it was 55 to 60 fps when the virtual hand interacted with the holographic object or virtual object. The results of averaged frame rate are shown in Fig. 10. These averaged frame rates were still acceptable for rendering the holographic object during various activities.
The interaction tasks were evaluated the performance of the system’s gesture recognition. The user did actual gesture 30 times/command and the detected gestures were counted. The gesture commands consist of six commands such as selecting, translating, rotation, scaling, operating for deformation, and displaying the hand panel. The results showed that the accuracy of gesture command detection in the proposed system was good as shown in Table 2.
4.2 Usability
The usability was evaluated using the proposed holographic application for creating some 3D prototypes. In this experiment, a physical object, an actual 3D printed cube, was put on the table as an initial shape and list of three targeted shapes was given to the participants for designing of 3D prototypes. The participants were eight people participated in the usability test. They were all volunteers aged between 19 and 28 at the Institute of Field Robotics. The participants were entry level or non-experience CAD designer. Two participants have used the Microsoft HoloLens while six persons never used it.
First, the participants designed holographic cube to fit with the actual 3D printed cube via HoloStudio application and proposed holographic application. The HoloStudio application provided hand gesture interaction with core hand gestures of HoloLens. The hologram was manipulated by gaze and commit interaction of HoloStudio application. The proposed holographic application provided the hand gesture interaction using both hands via the Leap Motion API. Its hologram was manipulated by gesture commands in Table 1. Designing times were limited to 15 min. The holographic object was saved in STL file format from both applications and printed as 3D prototype. The sizes of holographic objects were compared with the size of actual 3D printed cube for measuring of accuracy along with averaged time of designing as shown in Table 3. The result showed that some users spent times of designing less than the limited time. The shorter time implies that one application would be easier for designing of 3D model than another one.
Secondly, the participants were asked to design and deform the original holographic cubes into three targeted shapes in Table 4. Next, those holographic objects were saved in STL file format from the proposed applications and printed as 3D prototypes. It was found that the HoloStudio application could not perform this task due to lack of deformation capability. The outputs from this experiment were the real 3D printed prototypes obtained from the designed holographic object. In addition, the 3D prototype was assembled with the physical object as shown in Fig. 11.
Afterwards, the participants were asked to fill the questionnaire for evaluating the level of user’s satisfaction. The questionnaire rating started from 1 (strongly disagree) to 7 (strongly agree). The results from the questionnaires are shown in Table 5.
5 Conclusions
This research proposed the design of 3D model and build of 3D prototypes by merging technologies such as augmented reality (AR), hand gesture recognition, and 3D printing. The proposed system consisted of three main software parts which were hand gesture manager, graphics manager, and 3D printing manager involving with the Leap Motion, Microsoft HoloLens, and 3D Printer, respectively. Some experiments were conducted to evaluate the proposed system performance and usability of designing of 3D prototypes using augmented reality. The experimental results showed that the normal frame rate was equal to 65 fps which was more than Microsoft HoloLens guide recommended. Although the frame rate during interaction between virtual hands and holographic object was dropped to 55 fps, it was still acceptable to provide a good experience to the user. Moreover, after comparison of the uses of HoloStudio and the proposed application for designing of 3D model, it was found that the proposed system was better than the HoloStudio due to the shorter times of designing and more available features of hand gesture interaction. In addition, the size of 3D prototype built by the proposed system was nearly with the size of targeted object. The result from the questionnaire showed that the proposed system was able to reduce designing time, enhance user’s experience, and improve the usability.
In the future, several initial 3D shapes should be available in this proposed system so that the user can design and build different complex 3D prototypes. Furthermore, the system devices could be reconfigured to be a wearable system.
References
Nee, A.Y.C., Ong, S.K., Chryssolouris, G., Mourtzis, D.: Augmented reality applications in design and manufacturing. CIRP Ann. 61, 657–679 (2012)
Cheng, D., Wang, Y., Hua, H., Talha, M.M.: Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism. App. Opt. 48, 2655–2668 (2009)
Ong, S.K., Shen, Y.: A mixed reality environment for collaborative product design and development. CIRP Ann. 58, 139–142 (2009)
Ruiz, J., Li, Y.: DoubleFlip: a motion gesture delimiter for mobile interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2717–2720. ACM (2011)
Lin, P.-J., Chen, S.-C., Li, Y.-H., Wu, M.-S., Chen, S.-Y.: An implementation of augmented reality and location awareness services in mobile devices. In: Park, J., Adeli, H., Park, N., Woungang, I. (eds.) Mobile, Ubiquitous, and Intelligent Computing. LNEE, vol. 274, pp. 509–514. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-40675-1_76
Kitagawa, M., Yamamoto, T.: 3D puzzle guidance in augmented reality environment using a 3D desk surface projection. In: 2011 IEEE Symposium on 3D User Interfaces (3DUI), pp. 133–134. IEEE (2011)
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21, 34–47 (2001)
Garon, M., Boulet, P.O., Doironz, J.P., Beaulieu, L., Lalonde, J.F.: Real-time high resolution 3D data on the HoloLens. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 189–191 (2016)
Cui, J., Kuijper, A., Sourin, A.: Exploration of natural free-hand interaction for shape modeling using leap motion controller, pp. 41–48 (2016)
Hürst, W., van Wezel, C.: Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools Appl. 62, 233–258 (2013)
Ong, S., Wang, Z.: Augmented assembly technologies based on 3D bare-hand interaction. CIRP Ann.-Manuf. Technol. 60, 1–4 (2011)
Vinayak, K.R.: A gesture-free geometric approach for mid-air expression of design intent in 3D virtual pottery. Comput.-Aided Des. 69, 11–24 (2015)
Developer.leapmotion.com. https://developer.leapmotion.com/documentation/csharp/devguide/Leap_Coordinate_Mapping.html
Bae, S.-H., Balakrishnan, R., Singh, K.: ILoveSketch: as-natural-as-possible sketching system for creating 3D curve models. In: Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (2008)
Igarashi, T., Matsuoka, S., Tanaka, H.: Teddy: a sketching interface for 3D freeform design. In: ACM SIGGRAPH, article no. 21 (2017)
McDonnell, K.T., Qin, H., Wlodarczyk, R.A.: Virtual clay: a real-time sculpting system with haptic toolkits. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics, pp. 179–190 (2001)
Radkowski, R., Stritzke, C.: Interactive hand gesture-based assembly for augmented reality applications. In: The Fifth International Conference on Advances in Computer-Human Interactions (2012)
Nishino, H., Utsumiya, K., Korida, K.: 3D object modeling using spatial and pictographic gestures. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp. 51–58 (1998)
Fetter, M., Beckmann, C., Gross, T.: Towards interactive fabrication of 3D prints with FabAR—modifying and creating 3D models through gestures. In: 8th ACM International Conference on Tangible Interaction, pp. 205–212 (2014)
Chang, Y.S., Nuernberger, B., Luan, B., Höllerer, T., Donovan, J.O.: Gesture-based augmented reality annotation. In: IEEE Virtual Reality (VR), pp. 469–470 (2017)
Vuforia Developer Library: Working with the HoloLens sample in Unity. https://library.vuforia.com/articles/Training/Developing-Vuforia-Apps-for-HoloLens. Accessed 1 Feb 2018
Vuforia Developer Library: Developing Vuforia Apps for HoloLens. https://library.vuforia.com/articles/Solution/Working-with-the-HoloLens-sample-in-Unity. Accessed 1 Feb 2018
Bourke, P.: STL format. http://paulbourke.net/dataformats/stl/. Accessed 1 Feb 2018
Burkardt, J.: STLA Files. Department of Scientific Computing, Florida State University (FSU) (2014). http://people.sc.fsu.edu/~jburkardt/data/stla/stla.html. Accessed 1 Feb 2018
Hongwei, X., Weihua, J., Minjuan, L., Wei, L.: A slicing model algorithm based on STL model for additive manufacturing processes. In: 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pp. 1607–1610 (2016)
Henkel, K.: STL import/export for Unity. https://github.com/karl-/pb_Stl. Accessed 1 Feb 2018
Acknowledgment
This research work was financially supported by the Institute of Field Robotics (Thailand).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Jailungka, P., Charoenseang, S. (2018). Intuitive 3D Model Prototyping with Leap Motion and Microsoft HoloLens. In: Kurosu, M. (eds) Human-Computer Interaction. Interaction Technologies. HCI 2018. Lecture Notes in Computer Science(), vol 10903. Springer, Cham. https://doi.org/10.1007/978-3-319-91250-9_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-91250-9_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91249-3
Online ISBN: 978-3-319-91250-9
eBook Packages: Computer ScienceComputer Science (R0)