1 Introduction

As a representative of traditional Chinese art, traditional shadow play is a unique art ingeniously combining Chinese folk arts and crafts with opera. Traditional shadow play and its associated objects have high artistic and collection value in China and even the world because of their cultural details and production techniques. Chinese shadow play was selected for the list of items of Intangible Cultural Heritage in 2011.

However, at present, the inheritance and development of traditional shadow play, which has a history of more than 2000 years, is facing challenges in China. Traditional shadow play places an extreme demand on new practitioners due to its cumbersome production process and complicated operation skills. Nevertheless, shadow play scripts are mostly old-fashioned. Its obscure words and monotonous melodies are not very appealing to modern youth. These are the main problems restricting the inheritance and development of Chinese shadow play. In addition, the lack of channels to propagate traditional Chinese art is also responsible for the current plight of shadow play.

The development of digital media and human-computer interaction technology has provided new ideas for the inheritance and development of traditional shadow play art, and research on digital shadow play has increased. This new technology digitizes the production and performance of traditional shadow play while preserving its unique artistic aesthetic. The technology can overcome the limitations of traditional shadow play in production and dissemination and create conditions for a stronger inheritance and development of shadow play art.

Traditional shadow puppets are rich and diverse, and their vivid shapes are produced by laborious craftsmanship. Furthermore, depending on the type of puppets and performance needs, the positions and numbers of joysticks can be different. By the action of the joystick, relative movements occur between the various components (such as the human figure puppet’s head, chest, abdomen, limbs, and hands, collectively referred to as components in this paper) of the shadow puppet, resulting in a variety of vivid motions. Similarly, for digital shadow play, the creation of each shadow story requires designing different shadow puppets, and each puppet can be interactively controlled to perform. This requires the collaboration of artists and computer programmers.

In this paper, we study a method of creating digital shadow puppets based on parameterized templates. This method enables the rapid creation of digital shadow puppets, which can be controlled by hand gestures. We summarize the contributions of our paper as follows:

  1. (1)

    Based on an analysis and summary of traditional shadow play art, we classify shadow puppets and design the corresponding parameterized template. Using the template, one can quickly create a digital shadow puppet based on the materials one designs. The puppet created can then be controlled to produce a performance.

  2. (2)

    According to common human gestures and the needs of controlling shadow puppets, we design gestures based on the Leap Motion device for control. Then, the recognition of these control gestures and the mapping of hand movement to the motion of shadow puppets are achieved.

2 Related works

At present, there are two categories of work related to digital shadow play: interactive and non-interactive. Non-interactive digital shadow play is mainly used to display, propagate, and protect traditional shadow play culture. For example, the China Academy of Art has created a “digital museum of shadow” (http://shadow.caa.edu.cn), which digitally collects, manages, and permanently saves relevant data on the field of shadow play. Interactive digital shadow play adds elements of human-computer interaction based on non-interactive digital shadow play [4, 12, 13].

Zhu et al. created the digital shadow play animation Butterfly Love in 2003 [15], which uses subtle motion of a shadow puppet and light effects in a manner similar to a real shadow play performance. However, its algorithm is too complex to enable the real-time interactive performance of a shadow play. Lu et al. developed a digital shadow play system that allows users to create digital puppets and performance backgrounds [7]. However, users are restricted to creating a secondary design on the template provided by the system. Moreover, the creation tools cannot meet the needs of professional artists because the system is designed for children. Other related work [11] allows users to create digital shadow puppets by splicing the components of the shadow puppets built in the system. They can create characters based on fixed data only, which is not sufficiently flexible or extensible.

Most works of interactive digital shadow play mainly use interactive devices, such as somatosensory devices, handheld controllers and touch screens. Lu et al. developed a digital shadow story performance system with a handheld wireless sensor as input [7] to increase children’s creativity and level of cooperation. The work of Zhang et al. controls a digital shadow puppet by identifying multiple joysticks with colour marks [14]. Hsu and Li et al.’s animation system can generate digital shadow play animations according to high-level input from a user [3]. Shi et al. designed and developed a digital shadow play system using a Kinect [11] for shadow play performances. Liang et al. incorporated elements of Chinese shadow play into a digital narrative system based on gesture control [6]. However, it is simple in terms of hand gesture control and the display of characters’ motion. Characters in [6, 9, 11] can be controlled only to achieve the overall movement of the character and the playing of pre-set animations.

Techniques for sensing hand gestures have been developed, and commercial hand motion capture devices with affordable prices have begun to appear on the market. At present, research and application based on gestures focus mainly on the manipulation of virtual objects, such as playing the guitar through changes in gestures [2]. In the past few years, there have been many studies on interactively controlling objects based on hand movements. Masaki et al. proposed a hand-based motion control interface that allows users to interactively change the pose of a character using their hands and fingers [8]. Kan et al. proposed a method of using hand gestures to animate the bending and twisting motions of short plants such as grass and flowers [1]. This method can naturally use gestures to describe the movement of short plants. Helge et al. proposed a method to map different human motion inputs to different character motions, such as mapping arm motion to horse motion [10]. This method can provide fast and flexible interactive motion mapping and intuitively control characters in real time. In addition, some research on digital narrative systems has also designed and developed the function of storytelling using gestures to manipulate virtual puppets [5, 6]. These studies aim to improve children’s narrative skills, controlling the puppet through hand movements such as palm translation, making a fist, and letting go, in which, puppet animations were pre-set and triggered by gestures for performance. In our work, instead, puppet animation was generated in real-time and its movements were driven by the palm and fingers which is similar to the traditional Chinese art, shadow play. With Leap Motion’s 200 frames per second tracking speed and the action-gesture mapping set, the user can use hand movements to control the character in real time to make various actions instead of triggering pre-set animations.

The method of creating digital shadow puppets based on the parameterized template proposed in this paper can quickly transform static characters designed by artists into controllable shadow puppets. This makes the creation of digital shadow puppets more flexible and extensible. We consider the ease of use and the limited degrees of freedom the human finger joints usually have in the design of control gestures, which makes it convenient for non-professionals to master and control the characters smoothly.

3 Overview of the parameterized template

A parameterized template for digital shadow puppets is a tool to quickly transform static shadow characters designed by artists into controllable shadow puppets. This tool enables artists to focus on the design of shadow puppets. Since the human figure, the quadruped, and other types of characters in traditional shadow play have different structures and control methods, the template is designed and implemented based on the classification of shadow puppets. A parameterized template should have the following key features:

  1. (1)

    Parameterization of character appearance:

The traditional shadow play has spread widely in China and combines plastic art with local characteristics in the long-term evolution of different regions. In addition, the characters are very different in shape, and the proportions of each component are also different. The parameterized template should be able to quickly generate a digital character with the same appearance (including texture, the shape of each component, and the size ratio between components) according to the character materials designed freely by the artist.

  1. (2)

    The mapping relationship between hand movements and shadow puppet motions should meet the following criteria:

The constraints on the components of the puppet and the relative motion between the components caused by using the joysticks to control the control points determine the performability of the traditional shadow puppet. Controlling the control points set on the traditional shadow puppet with joysticks creates a correspondence between the control devices and the particular points on the components. The parameterized template should define the positions of control points on the components according to performance needs and establish the corresponding relationships between the control points and the control gestures to obtain a mapping from hand movements to shadow puppet motions.

As illustrated in Fig. 1, the parameterized template for digital shadow puppets is composed of the puppet geometry model and motion control module. The puppet geometry model is a parameterized model. It can adjust the size of each component to be consistent with the input character materials designed by the artist and then generate a puppet model with the same appearance by texture mapping. The motion control module defines the constraints between each component of the puppet, and the puppet obeys these constraints during the movement. The module defines the correspondence between hand gestures and control points on the puppet and achieves real-time control of the shadow puppet by hand gestures.

Fig. 1
figure 1

The composition of a parameterized template for a digital shadow puppet

Full size image

The basic process of creating an interactive digital shadow puppet based on the parameterized template includes three primary steps: the pre-processing of the component materials, the automatic generation of the character model and the interactive adjustment of the character model, as shown in Fig. 2. The input to this process is the component materials, which are two-dimensional texture data designed by the artist. The pre-processing is used mainly for the purpose of adjusting the orientation of the component materials and cropping the blank edges of textures. The automatic generation of the character model creates an interactive and controllable puppet with the same appearance as the character designed by the artist by mapping the pre-processed component material to the corresponding component model in the parameterized template. The interactive adjustment of the character model fine-tunes the relative position, scale, and rotation angle of each component, the constraints between components, and the gesture control settings of the original model. In the interactive adjustment step of the character model, the function of rehearsing the control of the puppet created based on the Leap Motion device is provided to assist in adjusting the model parameters.

Fig. 2
figure 2

Flow chart of the process of creating digital shadow puppets based on the parameterized template

Full size image

4 Digital shadow puppet model

4.1 Classification of digital shadow puppets

As mentioned in Section 3 of this paper, the parameterized template is designed and implemented based on the classification of shadow puppets. In traditional shadow play, the characters mainly include human beings, monsters, immortals, birds, beasts, and Chinese dragons. We consider the differences in the body structures and control methods of these characters, and we divide digital shadow puppets into four categories: human figures, winged bipeds, quadrupeds, and snake-like animals (such as Chinese dragons and snakes).

The human figure is the most frequently used character in traditional shadow stories. It is usually made of 11 components: the head, chest, abdomen, legs, upper arms, forearms, and hands. A character with a unified body component (the chest and abdomen are combined into one component) can also meet most of the performance needs, and the movement of the hand component relative to the forearm component has little impact on the performance. Therefore, this paper treats the character’s body (both chest and abdomen) as one component and does not define hand components separately. The character’s leg is treated as two components, the thigh and the shank, to show the details of leg movement. In summary, the human figure puppet model defined in this paper is made of 10 components: the head, body, upper arms, forearms, thighs, and shanks (as shown in Fig. 3a). Clearly, the components can be easily adjusted based on the work of this paper (the body can be treated as two components, the leg as a single component, the hand as an independent component, etc.).

Fig. 3
figure 3

The categories of shadow puppets and the components. a. A human figure. b. A winged biped (this image is from www.cnich.cn). c. A quadruped. d. A snake-like animal (this image is from kmsp.khcc.gov.tw). a and b are from the “digital museum of shadow” (http://shadow.caa.edu.cn)

Full size image

Regarding the winged biped and the quadruped, except for the different positions of their limbs, most traditional shadow puppets of these types are made of 7 components: head, body, limbs, and tail. Correspondingly, the winged biped puppet model and the quadruped puppet model defined in this paper both contain seven components (as shown in Fig. 3b and c). As shown in Fig. 3d, the snake-like animal puppet model defined in this paper includes not only the two components of head and tail but also N middle segments (N ≥ 0; the value can be set when creating the puppet as required).

4.2 Parameterized geometric model

The components of traditional shadow puppets are carved from leather and are connected by threads. When the shadow play performer performs, the connected components of the shadow puppet rotate relative to each other with the attachment point as the centre, which results in various motions.

Similar to the connection relationships between the components of traditional shadow puppets, this paper sets several joint points for digital shadow puppets. We restrict the relative movement between the two components associated with the joint point and define the components of the puppet as a tree structure according to the connection relationships between components and the movement laws governing the puppet. In the component tree, a parent node and child node are connected using the joint point. The parent node transfers its movement to the components of the child nodes. Figure 4a shows a component tree of a human figure puppet. The root node of the tree is the body component, and its child nodes are the following five components: the head, left/right upper arm, left/right thigh. The left/right forearm and the left/right shank are the child nodes of the left/right upper arm and the left/right thigh, respectively. Figure 4b–d are the component trees of the winged biped, the quadruped, and the snake-like animal, respectively. Similar to the component tree of the human figure, the winged biped and the quadruped also use the body component as the root node of the component tree. However, the component tree of the snake-like animal uses the head component as the root node. The geometric model of each character component is a square plane with uniform side length L (set as 30 cm in this paper).

Fig. 4
figure 4

The component trees of puppets. a. The human figure. b. The winged biped. c. The quadruped. d. The snake-like animal

Full size image

Figure 5 shows geometric models of the human figure, the winged biped, the quadruped, and the snake-like animal. There is a one-to-one match between each plane of the model and each component of the puppet. The initial geometric model of the snake-like animal (Fig. 5d) includes two component models (head and tail) and one joint point. The user specifies the number of middle segments to build the desired character model.

Fig. 5
figure 5

Digital shadow puppets created from parameterized templates. The left column shows the geometric models of the templates. The circles represent the joint points, and the stars represent the control points. The right column shows the generated digital puppet geometries

Full size image

This paper also defines the control points for the parameterized geometric models with reference to the performance methods of traditional shadow play (as shown in Fig. 5). The movement of these control points can cause the shadow puppet to generate corresponding motions (see Section 5).

4.3 Creating a shadow puppet geometry based on the parameterized geometric model

Component materials of characters provided by users can not exactly be the same in styles, such as blank space and orientation. We pre-process materials, including defining the smallest rectangular bounding box enclosing the component, cropping the material with the bounding box (remove the extra white space), and adjusting the orientation of the texture on the basis of principal components analysis.

Adjust the size of the model of the parameterized puppet template automatically according to the pre-processed materials of each component. And then, map the pre-processed 2D textures of the character components to the corresponding component plane of the puppet model to obtain the puppet geometry with the same appearance as the character designed by the user (The shape of each component, the size ratio between the components, and the textures are consistent with the artist’s design.). Then, do more modifications to the model according to the user’s interactive adjustments. The main steps of the template parameter adjustment process are as follows:

  1. Step 1:

    Transformation of the reference component plane: This paper defines the root node of the component tree as the reference component, which serves as the reference for the adjustment of the size of the other components. First, we adjust the aspect ratio of the reference component plane to make it consistent with the aspect ratio Sr of the pre-processed texture of the reference component. We keep the height l of the reference component plane unchanged and adjust its width to Srl.

  2. Step 2:

    Transformation of other component planes: Similarly, for each component i, we adjust the plane to make its aspect ratio consistent with the aspect ratio Si of the pre-processed texture. In addition, the size αi relative to the reference component (the texture height of component i / the reference component texture height) is kept constant. Therefore, we adjust the height of the component plane to αil and the width to Siαil.

  3. Step 3:

    Map the pre-processed textures of the character components to the corresponding component plane and generate the puppet geometry with the same appearance as the character designed by the user.

  4. Step 4:

    Interactive adjustment: The user specifies scale β to adjust the overall size of the puppet and manually adjusts other parameters of the puppet geometry, such as relative position, scale, and rotation angle of each component, and the rotatable angle range of the child component relative to the parent component.

Figure 5 shows the digital shadow puppets created based on the geometric models of the templates. These characters are generated by different character materials, with the same appearance as the users’ designs. In particular, Fig. 5d shows three “snake” shadow puppets with 5, 8 and 11 middle segments. These were created based on the snake-like animal template.

5 Motion control of the digital shadow puppet template

In traditional shadow play, the human figure puppet is generally controlled by three joysticks. Figure 6 shows a shadow play performer controlling one such puppet. The joystick fixed on the puppet’s neck is mainly used to control the movement of the puppet. The other two joysticks are fixed on two wrists to control the upper limbs. There are no joysticks on the head and legs. Their motions rely on the movement of the body and inertia. In addition, traditional shadow play usually has a baffle at the bottom of the screen, which is used to place scene props and support the legs of the puppet (Fig. 6). In the case of three-joystick control, although joysticks are not fixed on the legs, the performer can still control the puppet to perform motions such as kneeling and sitting by using the baffle. Therefore, this paper divides the motions of the digital shadow puppet into three categories: motions based on the control point, motions based on inertia, and motions based on the baffle.

Fig. 6
figure 6

A shadow play performer performs in Yuanjia village, Shaanxi Province, China. The image is from www.cnwest.com

Full size image

5.1 Motions based on the control point

Shadow play, as one of the traditional Chinese art, is performed with hand by the player, and the shadow puppet’s movement is driven by fingers interactively. In order to emulate the player’s performance, the presented digital shadow puppet is designed to control by hand gesture.

Kinect and Leap Motion are both somatosensory interaction devices which can capture the hand gesture information in real time. Kinect is more inclined to recognize human limbs, and the capture of specific details such as the hand is relatively rough. Leap Motion can finely detect hand and finger movements, which is more suitable for manipulating the digital shadow puppet than Kinect.

5.1.1 Descriptions of leap motion

Leap Motion, released in 2013, is a somatosensory device that uses infrared sensing technology for hand tracking and gesture recognition. It supports touchless interaction using palms and gestures. It tracks gestures at more than 200 frames per second and detects fine hand movements and finger movements with much lower latencies, which Microsoft Kinect cannot provide. In addition, it has the advantages of low cost, small size, and ease of use. Leap Motion not only provides users with an intuitive and natural manipulation experience but also can expand the interactive method of digital shadow art and are suitable for large-scale promotion. Its potential application fields are wide, such as virtual reality, games, medical rehabilitation, and cultural education.

In this paper, the Leap Motion device is used to capture hand movements in real time and provide natural gesture input to control the digital shadow puppets, to realize a digital shadow story performance based on gesture control. This paper represents the raw data obtained from Leap Motion as 〈PD, DD〉. PD is the palm data, which includes the palm ID, position, rotation (including pitch, yaw, and roll angles), and other data. DD is the finger data, which contains information about the five fingers of the hand, such as finger ID, the position of each finger bone, and rotation.

5.1.2 Analysis of motion driven by control points

In accordance with the traditional control method of using the joystick, we design a puppet control method based on the control points. This paper defines four control points for the human figure puppet, which are on the head component, the body component, and two forearm components. Three control points are defined for the winged biped and the quadruped, which are on the head component, the body component, and the tail component. For these three types of puppets, the control point on the body component is used to control the overall movement. This control point changes the position of the body component during the performance. According to the forward kinematics, we traverse the component tree to solve the positions of each joint-point and its connected subcomponents. For the snake-like animal, this paper defines only one control point, which is on the head component, to control the movement of the puppet.

Similar to the process of controlling shadow puppets in a traditional shadow play performance, the human figure puppet’s upper limbs are controlled by the control points on the forearm components, and the control process follows the inverse kinematics.

Figure 7a shows a simplified upper limb of a human figure puppet. Point A(xA, yA) is the shoulder joint point, point B(xB, yB) is the elbow joint point, and point C(xC, yC) is the control point on the forearm component. The length of the upper arm is L1, and the distance between the elbow joint point and the control point is L2. Clearly, since L1 and L2 are constant values, the vector \( \overset{\rightharpoonup }{\mathrm{AC}} \) defined by the shoulder joint point A and control point C satisfies the expression \( \left|\mathrm{L}1-\mathrm{L}2\right|\le \left|\overset{\rightharpoonup }{\mathrm{AC}}\right|\le \left(\mathrm{L}1+\mathrm{L}2\right) \).

Fig. 7
figure 7

Simplified diagram of the human figure puppet’s upper limb

Full size image

When the control point C moves to a new position, the puppet’s upper limb may move to one of four postures, as shown in Fig. 7b-d. We define the direction of \( \overset{\rightharpoonup }{\mathrm{AC}}\times \overset{\rightharpoonup }{\mathrm{AB}} \) in posture 2 as positive and the direction of \( \overset{\rightharpoonup }{\mathrm{AC}}\times \overset{\rightharpoonup }{\mathrm{AB}} \) in posture 3 as negative. In each frame, the posture of the upper limb is determined by the control point C in the current frame and the shoulder joint point A′ and elbow joint point B′ in the previous frame.

$$ \mathrm{posture}=\left\{\begin{array}{c}\mathrm{posture}\ 0,\kern0.5em \left|\overset{\rightharpoonup }{{\mathrm{A}}^{\prime}\mathrm{C}}\right|=\left|\mathrm{L}1-\mathrm{L}2\right|\\ {}\mathrm{posture}\ 1,\kern0.5em \left|\overset{\rightharpoonup }{{\mathrm{A}}^{\prime}\mathrm{C}}\right|=\left(\mathrm{L}1+\mathrm{L}2\right)\\ {}\mathrm{posture}\ 2,\kern0.5em {\mathrm{Direction}}_{\overset{\rightharpoonup }{{\mathrm{A}}^{\prime}\mathrm{C}}\times \overset{\rightharpoonup }{{\mathrm{A}}^{\prime }{\mathrm{B}}^{\prime }}}\ \mathrm{is}\ \mathrm{positive}\ \mathrm{and}\ \left|\mathrm{L}1-\mathrm{L}2\right|<\left|\overset{\rightharpoonup }{{\mathrm{A}}^{\prime}\mathrm{C}}\right|<\left(\mathrm{L}1+\mathrm{L}2\right)\\ {}\mathrm{posture}\ 3,\kern0.5em {\mathrm{Direction}}_{\overset{\rightharpoonup }{{\mathrm{A}}^{\prime}\mathrm{C}}\times \overset{\rightharpoonup }{{\mathrm{A}}^{\prime }{\mathrm{B}}^{\prime }}}\ \mathrm{is}\ \mathrm{negative}\ \mathrm{and}\ \left|\mathrm{L}1-\mathrm{L}2\right|<\left|\overset{\rightharpoonup }{{\mathrm{A}}^{\prime}\mathrm{C}}\right|<\left(\mathrm{L}1+\mathrm{L}2\right)\end{array}\right. $$
(1)

The angle formed by rotating the vector \( \overset{\rightharpoonup }{\mathrm{AC}} \) from point A counterclockwise to the downward direction (the blue dotted line in Fig. 7) is calculated as follows:

$$ \theta ={\theta}_1+{\theta}_2=\pi \pm {\cos}^{-1}\left(\frac{y_C-{y}_A}{\left|\overset{\rightharpoonup }{\mathrm{AC}}\right|}\right) $$
(2)

When xC ≥ xA, the sign ± is positive; otherwise, it is negative. When the upper limb is in posture 0, the counter-clockwise angles θU and θF from the upper arm and the forearm to the downward direction are θ and θ + π, respectively. For posture 1, the counter-clockwise angles θU and θF are both θ.

In the case where the upper limb is in posture 2 or 3, θ2 and θ3 can be obtained according to the cosine theorem. For posture 2, the counter-clockwise angles θU and θF are calculated as follows:

$$ {\displaystyle \begin{array}{c}{\theta}_U={\theta}_1\\ {}{\theta}_F={\theta}_U+\pi -{\theta}_3\end{array}} $$
(3)

When the upper limb is in posture 3, the counter-clockwise angles θU and θF are calculated as follows:

$$ {\displaystyle \begin{array}{c}{\theta}_U={\theta}_1+2{\theta}_2\\ {}{\theta}_F={\theta}_U-\pi +{\theta}_3\end{array}} $$
(4)

The motions of the head (for the human figure, the winged biped, and the quadruped) and tail (for the winged biped and the quadruped) are performed by controlling the component to rotate around the joint point using the control point located on the corresponding component. In this process, the distance between the control point and the joint point does not change. The motion track of the control point is a circle around the joint point.

5.1.3 Design of control gestures and the control of control points by gestures

During a digital shadow play show, Leap Motion provides a variety of information about the palms, fingers, etc., of the performer. After being processed, these data are used to control the shadow puppet. To ensure that the character’s movements can be controlled, we use the movement of palms and hand gestures (see Table 1 for the definitions of the gestures) to manipulate the control points.

Table 1 Motion-gesture map
Full size table

Moving the shadow puppet is the most frequent demand in traditional shadow play shows. For the human figure puppet, it is also necessary to finely control the joystick fixed on the wrist, which can give the puppet lifelike upper limb motions. Therefore, this paper uses the movement of the palm to control the overall movement and the upper limb motion. We transform the movement of the palm on the control plane (which can be defined by any two of the three coordinate axes x, y, and z in Leap Motion) into the displacement of the control points on the two-dimensional plane in the digital scene. The displacement of the control point ∆p is calculated as follows:

$$ \Delta \boldsymbol{p}=m\cdotp \Delta \boldsymbol{r} $$
(5)

where ∆r is the displacement of the palm and m is the scaling factor.

In addition, we assign the control of the control points located on the head and tail components to the fingers. Human fingers have limited freedom of movement and can only easily bend towards the palm or straighten. Therefore, in this paper, the angle at which the finger bends to the palm is mapped to the position of the control point.

5.2 Motions based on inertia and the baffle

In traditional shadow play art, the players often take advantage of inertia to generate the motion of the puppet’s components, such as the head, upper limbs, and lower limbs. We simulate this kind of motion caused by inertia with the flexible and data-driven physical constraint system provided by Unreal Engine 4. Such, when the puppet is driven to move by the control point on the body component, components connected to the body could be driven to move by inertia. A child component rotates relative to its parent component according to the direction and speed during the movement of its parent component. Figure 8 shows some motions caused by inertia when moving digital puppets.

Fig. 8
figure 8

Examples of motions of the four types of puppets under inertia

Full size image

The angle range of the child component rotating relative to the parent component changes the effect of inertia during a shadow play performance, and this range can be set by the user according to the performance needs. Figure 9 shows some motions of human figure puppets caused by inertia when the angle range of the upper arm and the forearm component rotating relative to the parent component is set to zero. By adjusting the angle range, special performance needs can be satisfied.

Fig. 9
figure 9

Motions of human figure puppets when the angle range of the upper arm and the forearm component rotating relative to the parent component is set to zero

Full size image

In addition, similar to traditional shadow play, we set a baffle at the bottom of the digital performance scene. During the downward movement of the puppet, the feet (end of lower limbs) contact the baffle. The thigh and shank components rotate relative to the parent component due to the support of the baffle. Similar to the upper limb motion based on the control point, when the feet contact the baffle, the lower limbs also have the same four possible postures as shown in Fig. 7b-e. Point A is the foot, point B is the knee joint point, and point C is the hip joint point. The calculation of the motions is similar to that described in Section 5.1.2. Figure 10 shows that the puppet can be controlled to make motions such as kneeling and sitting by using the support of the baffle.

Fig. 10
figure 10

Examples of motions using the support of a baffle; the baffle is shown in blue for clarity

Full size image

6 Experimental results and evaluations

Using this paper’s study results, we design and implement an interactive platform for creating and performing digital shadow stories based on Unreal Engine 4 (combined with the blueprint and programmed in C++). With the platform, users can create digital shadow stories that can be performed by hand gestures by inputting shadow scenes and character materials designed by artists. The platform includes functions such as creating puppets, setting performance scenes, and choosing background music and storylines. The platform runs on a personal computer with an Intel (R) Core (TM) i7–4790 CPU, 16 GB RAM, and an NVIDIA GTX 660 GPU and SSD; Leap Motion is the input device to control the digital puppet. Figure 11 shows a user making hand gestures to control puppets on our platform.

Fig. 11
figure 11

A user performs with gestures

Full size image

6.1 User study and evaluation

Participants and procedure

Considering the large age span between shadow play audiences, we invited 12 participants (6 women) to participate in this evaluation, ranging in age from 14 to 64 years old. These participants are non-professionals in shadow play and are not familiar with 3D gesture interfaces before, such as Leap Motion. The character materials required for the evaluation were designed by artists taking Chinese masterpieces, fables, and traditional shadow plays as references, such as Three Times Defeating the White Bone Demon, Monkey Makes Three Attempts to Borrow the Plantain Fan, Daiyu Burying the Flowers, The Battle Between the Monkey King and the God Erlang, Mr Dong Guo and the Wolf, and The Cowherd and the Weaving Maid. In our evaluation process, participants performed the tasks of digital shadow puppet creation based on the parameterized template and manipulating digital characters for shadow play performance on a personal computer with Leap Motion.

The study mainly included four sections: training, puppet creation, motion control, and user feedback.

  1. 1.

    Training: We first introduced to 12 participants the methods we proposed and the platform designed and developed accordingly. Then, we gave them time to learn and become familiar with how to create digital shadow puppets based on the parameterized template and how to use their hands to control the puppet.

  2. 2.

    Puppet creation: After training, participants were required to create four different categories of digital shadow puppets based on the given component materials of characters. They could also manually adjust parameters of the automatically generated character model according to the character concept and personal preferences, such as relative position, scale, and rotation angle of each component, the overall size of the puppet, and the rotatable angle range of the child component relative to the parent component, which is not required.

  3. 3.

    Motion control: In this part, participants were required to use gestures to control the generated digital puppets with reference to the motion-gesture mapping table.

  4. 4.

    User feedback: We recorded the participants’ performance and problems during tasks of creating puppets based on the parameterized template and manipulating digital characters for shadow play performance. After the end of the task, we sent out a questionnaire about evaluating the functions of puppet creation and motion control to each participant. In terms of puppet creation, we provided five criteria for evaluation: overall satisfaction, template classification, quick transformation, appearance of the original model and adjusting the model interactively. Similarly, the evaluation of motion control also involved five criteria: overall experience, easy to learn, easy to control, trigger accuracy and visual effects of character motions. Participants were asked to rate each criterion, with scores ranging from 1 to 5 points, indicating poor to excellent. In the questionnaire, we also gave an open-ended question for suggestions. By collecting and analysing the questionnaire, we collected user feedback to improve our experimental design.

Puppet creation evaluation results

Figure 12a shows the participants’ feedback on our platform’s function of creating digital shadow puppets based on the parameterized template. Overall, all participants expressed satisfaction (all criteria received positive feedback, and the average rating of each criterion exceeded 4.40). They reported that our template is helpful and can easily transform the static character designed by the artist into a controllable digital shadow puppet. They affirmed that the four categories of templates essentially cover the character categories common in traditional shadow play (the average rating was 4.47) and that the parameterized template makes the restrictions on character materials very small. They further agreed that our platform can quickly transform character materials into character geometric models (the average rating was 4.80) and that the aspect ratio of the components and the size ratio between the components are broadly consistent with the existing character design (the average rating was 4.72). They also affirmed that the option of interactive adjustment of the generated character model is very helpful (the average rating was 4.57), because they can perform secondary creation on the generated initial character model according to their own ideas.

Fig. 12
figure 12

The average rating of each criterion in the two tasks. a. Puppet creation. b. Motion control

Full size image

Time cost of puppet creation

We timed the procedure of puppet creation, which starts when raw component materials are inputted and ends when the puppet geometry is generated. In this evaluation, the raw component materials of 5 characters were used, and the puppet geometry of each character was created 300 times. We gave the average, maximum and minimum time of creating a puppet in Table 2. As shown in Table 2, it only takes around 0.40s to generate a human figure puppet based on the given raw component materials, and takes less time for other categories of puppets, indicating that our method can transform the component materials into the puppet at a fast speed.

Table 2 The time cost of the puppet creation
Full size table

We also timed the material pre-processing procedure during the puppet creation, and the detailed data was given in Table 3. The results show that the component material pre-processing takes up a large part of the time during the puppet creation. Therefore, we can further speed up the puppet creation by improving the efficiency of the material pre-processing in future work.

Table 3 The time cost of the material pre-processing during the puppet creation
Full size table

Motion control evaluation results

As shown in Fig. 12b, the participants reported that the experience of using gestures to control the digital characters they generate is easy and pleasant (the average rating of the overall experience was 4.59). The gestures we designed were easy for them to learn and become familiar with in a short time (the average rating was 4.80). Participants agreed that they could control the puppet to make motions similar to those in traditional shadow play. That is, they confirmed that the motion state generated by using gestures to control digital shadow puppet is credible in visual effects and can reasonably and naturally realize the motions of shadow puppet characters in traditional shadow play performances, especially those based on joysticks (the average rating was 4.60).

Latency

The latency for the manipulation is the time from when the hand data is captured by Leap Motion to the time the motion of a puppet is performed, which includes the system latency and the mapping latency. The system latency depends on the tracking camera frame rate, the USB data transfer rates, tracking algorithm, display refresh rate, etc. Using the computer with a USB3.0 port and a 120hz monitor with a low response time, the system latency can be reduced to below 30 ms, which is imperceptible for humans. Our platform processes the hand data received from Leap Motion and implements the corresponding motion of the puppet. The time cost of this process is the mapping latency. We used gestures to control the human figure puppet to perform a digital shadow play, and measured the mapping latency during the performance. The maximum and minimum mapping latencies measured were 1.21 ms and 0.25 ms respectively, while the average mapping latency was 0.42 ms. The evaluation results show that the mapping latency is very low, and our platform can work on real-time interaction using Leap Motion on a computer with good configuration, which is important for users to control the puppet during the performance.

6.2 Discussion

The method proposed in this paper is an effective method to obtain and control digital shadow puppets. Furthermore, it can also be used to transform other types of static characters, not just shadow characters. Our method can provide services for traditional animation production. For example, in the storyboard stage, compared with charts, our method can quickly obtain intuitive animation images to explain how a scene is composed. The advantage of our method is that it is fast and minimally restrictive and provides tools for interactive adjustment of the generated character model, so users can efficiently control the digital characters generated to design the storyboard.

Our method may not allow users to experience the feeling of using joysticks to control a puppet like a traditional shadow play performer. However, based on the feedback from the participants, they are happy to use the gestures we designed to control the digital characters. They reported that our gestures are simple and easy to learn. Simplifying the control method in traditional shadow play, which is complicated and difficult to master. Non-professionals such as the participants can thus experience shadow play.

Although the participants gave relatively good positive feedback on the trigger accuracy (the average rating was 4.35), we noticed some problems during the evaluation process. For example, when one hand overlaps with another during the participant’s manipulation, it may cause erroneous detection and affect the accuracy of hand tracking. Another limitation of Leap Motion is that it can detect hands and fingers in a limited tracking area only. If the participant’s movement amplitude is too large, it causes the position of the hand to easily exceed the scope of the Leap Motion capture field, thereby losing tracking. In this case, to maintain the consistency of the shadow puppet’s motion, the puppet is kept in its original position until the hand is detected by Leap Motion again. In addition, some participants said that although there is no limit to standing or sitting in the process of control, the long-term suspension of upper limbs and hand gesture control can cause upper limb fatigue. For all applications based on hand gesture control, this is a relatively common problem—one we need to solve urgently.

The evaluation results show that the method of digital shadow puppet creation based on the parameterized template proposed in this paper can quickly transform the materials designed by an artist into a controllable puppet. This approach can meet users’ needs of interactive shadow play animation creation in both appearance and controllability.

7 Conclusion and future work

This paper proposes a method of creating digital shadow puppets based on parameterized templates. Based on an analysis and summary of traditional shadow puppetry art, we classify digital shadow puppets and design corresponding parameterized templates. A template can quickly transform static character materials designed by artists into controllable shadow puppets. In accordance with common human hand gestures and the needs of digital shadow play performance, we design and implement a method of using hand gestures to control a digital character. This technique is easy for non-professionals to master and control.

We can further improve the creation and use of digital shadow puppets by modifying our parameterized templates. A necessary modification is to work with professionals to make the template setting, performance process, and control rules closer to those of traditional shadow play art. We are considering adding an option that allows users to freely combine components to set character templates to meet different creative needs. The aria also plays an important role in traditional shadow play art, and we are considering recording and playing an aria and a digital shadow play performance simultaneously. To improve the experience of performing a digital shadow play, we might further improve the motion control module of the template so that the puppet can be controlled by more diverse hand gestures to make more lifelike motions. In addition, we are considering adding a function allowing users to pre-record the motions of puppets and use hand gestures to trigger them during the show, which is expected to be very convenient and useful.