Design and evaluation of user’s physical experience in an Ambient Interactive Storybook and full body interaction games

Abstract

The paper presents the design and evaluation of an original Ambient Interactive Storybook (AIS) for children, including its platform, the background story, and 10 full body interactive games. The evaluation, which focused on the user’s physical experience and elements important to the designer, has been methodologically derived from the Kroflič’s and Laban’s framework Body, Space, Time and Relationship, and additionally supported by sport science measurements. An experiment with 8 participants playing 10 games for 20 min was conducted and recorded to a digital video. Participants’ physical experience was evaluated through the analysis of postures, the quality of the movement, the body parts used in the interaction, the playing area, the direction of movement, direction of gaze, tempo, dynamics and the quantity of motion (QoM). Results of the experiment are discussed in relation and with implications for game design. Conclusions are drawn with the summary of main findings, to better understand the mechanisms and principles involved the design of user’s physical activity in full body interactive games for children. The theoretical work of Laban and Kroflič proved to be useful for interaction and games design in the transition from desktop to full body interactive games.

1 Introduction

1.1 Movement and storytelling

Game playing and storytelling have been always connected with all our senses: sound, taste, smell, touch, and seeing. Games have been played through the body, and body movement has been used to tell, as well as to feel stories. With the introduction of radio and TV, and especially information communication technology “we became post human as information lost its body” [11].

This lack of “meaningful engagement of our body” [11] meant that humans worked, learnt, listened and watched stories and also played games in static postures. At the time when computer games became the chosen form of play for many children [23], such a lifestyle not just poses risks of upper body musculoskeletal disorders, but it can contribute to the obesity. There is a general perception that the current obesity epidemic is directly linked to the amount of time children spend playing computer games and similar activities. The National Health Service [24] found that 40% of UK’s children are overweight and that they have problems moving backwards and/or standing on one leg.

The speed of computing, which was one of the main obstacles in computer vision applications, is increasing rapidly. Accessibility of powerful processors, mega pixels video cameras in mobile devices, and even projectors in digital cameras (e.g. Nikon Coolpix S1000pj) offer new opportunities in full body interaction games, not only as a part of a particular game console. The popularity of today’s flat-panel liquid-crystal and plasma TV displays shows increasing trends in favour of large-screen displays. Interactive whiteboards are becoming a norm in an average classroom. This offers an opportunity to introduce more movement into infotainment and edutainment, so that they become a part of everyday life, entertainment and learning, either at school or at home.

The following paper is structured into five sections, and presents: (1) introduction and related work in the field of the design of full body interaction games, (2) an original Ambient Interactive Storybook (AIS), including its platform, the background story, and 10 games for children, (3) evaluation of the games, using the Body-Space-Time/Effort-Relationship analyses, (4) results of analyses with discussion of findings and implications for game design, and (5) conclusions with the summary of main findings.

1.2 Related work

Although the beginning of full body interaction games goes back to the 1970s, the design of full body interactive games is a new research and design area. In 1970’s, Myron Krueger was dissatisfied with “restricted dialogue where man-machine interaction is usually limited to a seated man poking at a machine with his fingers or perhaps waving a wand over a data tablet” and started to work on his project VideoPlace [19]. Full body interaction was investigated in the area of new media art in a form of interactive video installations with projects such as Text Rain [32]. In 1996, Mandala Gesture Xtreme (GX) System [22] became the first commercial arcade application using computer vision. Instead of a projector, the system used a large CRT display, which changed the player’s experience as the space was lighter, and therefore a player could move with more confidence. In 2001, Intel® Play ^TM Me2Cam Virtual Game System [15] the first home product was released. At the time, when an average PC screen was a 15″ CRT display, it was difficult to play, as due to a small size it was hard to see the action on the screen. The recent commercial success of Sony PlayStation Eyetoy [8] and Nintendo Wii [25], which is not a “proper”¹ full body interactive game, introduced the idea of a computer body interaction, full body interaction, and physically interactive games, i.e. games where the main interaction device is a user’s body. The user’s action was finally moved to take place in front of a large screen (i.e. TV display), where there was more space available for the physical gameplay.

Early design projects in full body interaction games established and presented game design principles [5]. Warren [33] introduced gamelets (i.e. sub elements of the games) which incorporated important advantages of using a computer vision over more traditional interfaces in video games. Labanotation has been used in the interaction design context to evaluate Eyetoy Sony PlayStation games [21] and it provided a valuable foundation for the design of movement-based interaction. Work of QuiQui Giant Bounce [28] and Kick Ass Kung-Fu [16] focused around transforming the user movement into the gameplay. Projects such as Wristbands as Interaction Devices [20], QuiQui’s Giant Bounce [28], Kick Ass Kung-Fu [16] and most recently experiments with Harmony Space [12] investigated the educational potential of connecting the motor and intellectual activities in full body interactive games.

The purpose and objectives of our work were: (1) to design and produce an original Ambient Interactive Storybook (AIS) for children, including its platform, the background story, and 10 full body interactive games, (2) to select, adapt and apply methodological approaches and tools for the evaluation of the user’s physical experience in full body interactive games, (3) to evaluate the games and the user’s physical experience with a group of children, with commonly available webcams, screens and PCs, (4) based on the results of the experiment, to discuss the findings in relation and with implications for game design.

2 Ambient interactive storybook (AIS)

2.1 The AIS platform

The ambient interactive storybook for an immersive environment is equipped with a webcam as the motion sensor, a large visual display, and a PC using low level computer vision algorithms, such as detecting mass centre and motion detection. AIS design goals were that games and activities should target all children, including those who don’t exercise regularly or don’t like sports, as well as those who are full of energy and can’t be quiet and still. AIS targets primarily younger children (age 4–9).

The AIS encourages interest for nature, sports, dance and healthy lifestyle. Edutainment games and activities cover different learning areas, and are designed to use the body to solve problems through play and active learning. Users spontaneously discover a story and characters, play games, solve problems, and express and reflect about their aesthetic experience with their body. The user experience is designed as a creative journey using body movements through fictional worlds, meeting characters, visiting spaces, playing games and solving puzzles.

Games and activities in the AIS are based on popular children’s edutainment genres, such as interactive storytelling, racing, matching, memory and catching, but are obviously adapted to the situation where the user sees himself/herself on the screen. Music sets the mood and influences the gameplay, as a higher music tempo is more physically demanding. Games vary in tempo; some have a high tempo, while others require patience for successful playing.

2.1.1 Technical details

AIS platform was developed in Adobe Flash 9. It consists of a live video, provided by a webcam of a size 320 x 240 pixels, and mass centre and motion detection algorithms written in ActionScript 3. Contrary to Me2Cam [5], Krueger’s VideoPlace [19] and Kick Ass Kung-Fu [16], the AIS platform does not use background subtraction. The video of the user is embedded inside computer graphics as shown in Figs. 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12, which provides accurate visual feedback of the user’s movements.

When a user moves a whole body (Fig. 1—user actions), the centre mass algorithm first calculates the difference in the current and previous frame of the video (Fig. 1—computer vision algorithms), which is stored as a temporary bitmap image. In the next step, a new colour pixel value in the bitmap image is set to those pixels that have values above a specified threshold. Then, a rectangular region that fully encloses all pixels of a specified colour within the bitmap image is determined (e. g. red rectangles in images centre left and right in Fig. 2). Centre points of this rectangle region (e.g. white lines in Fig. 2) are calculated (Fig. 1—x_y_position), after additional algorithms compensate possible changes of values due to changes of lighting conditions, drops of frames and the quality of a webcam’s image (Fig. 1—compensations algorithms). X and y values are then mapped to an avatar (Fig. 1—games elements) or other game elements that are controlled by the user’s whole body movement.

Fig. 1

Schematic representation of the whole body movement detection process in the case of AIS platform

Fig. 2

Two different body postures with similar mass centre x and y values. Left: original input—posture 1. Centre left: the value of the difference between the current and previous frame—posture 1. White lines show the mass centre of the user. Centre right: original input—posture 2. Right: the value of the difference between the current and previous frame—posture 2. White lines show the mass centre

When a user moves with a whole body (Fig. 1—user actions), Adobe Flash function ActivityLevel (Fig. 1—computer vision algorithms), specifies the amount of motion the camera is detecting. Compensated values (Fig. 1—compensations algorithms) are then passed to games elements (Fig. 1—games elements). Such structure of the platform allows quick authoring and scripting of games elements (Fig. 1—games elements) with collision detection and other games algorithms. The games are authored in Adobe Flash document in full-screen mode. Additional blur filters are applied to improve jagged edges in the video, which occurs in Flash player.

AIS games (Figs. 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12) use mostly tracking of a user left and right (x value), while the tracking up and down (y values) is used only in game 9 (Fig. 11). The use of mass centre algorithms allows that an avatar can be controlled not only by moving left and right (Fig. 2 right), but also by bending (Fig. 1 left) or any other posture, which results in similar x and y position. Such use of computer vision algorithms, together with randomness of the game’s elements, makes the movement in AIS’s full body interactive games freer and less predictable, as it is based on open skills in an unpredictable environment. In this respect, the movement becomes similar to the movement in sports games.

Fig. 3

Racing the city

Fig. 4

Symmetry game

Fig. 5

Don’t wake up Mr Smith

Fig. 6

Hula hoop game

Fig. 7

Flying on the cloud

Fig. 8

Spinning river

Fig. 9

Racing game

Fig. 10

Deep dark woods

Fig. 11

Wisdom tree

Fig. 12

Fallen stars

AIS games have been successfully tested on different displays (i.e. 32″ and 40″ plasma screens and projectors), in rooms with different lighting conditions (i.e. day light and tungsten), with commonly available webcams and PCs.

2.2 The background story

“The Adventures of Spinning Top” is an interactive story that promotes a healthy lifestyle and interest in physical activities. It’s an ambient interactive storybook that gets children up in front of their favourite story and characters, and keeps them physically active.

The story begins:

Spinning Top is a funny, little, and well, very spinny person. She loves going to new places, loves doing new things and she most of all loves to spin. On sunny days she spins away for a walk with her dear friend Hula. It makes her brain work harder.

She says:

‘If your mind goes blank at times, Try to do some exercise.

Spinning, dancing, jumping high. New ideas spring into life.’

Before we begin our story, you should know that this is no ordinary story. Things only happen in Spinning Top’s world if you keep moving with her.

The characters are favourite children’s toys, adults and children. The story is about Stella & Tom, who live in a Big City, in a Big House, in a flat where they cannot run and jump. Their neighbour, Mr Smith, is constantly complaining. Spinning Top appears and takes Stella and Tom to “Spinning World”, where children can be children and are free to jump, crawl, dance, and run as much as they want … but on the way to the “Spinning Island” something happens….

2.3 Games and activities

In the game 1 Racing the City (Fig. 3 and as shown in the video (Online Resource 1)) children learn about the full body interaction and warm up. In order to successfully play the game, the user needs to move his/her whole body left and right to control the avatar (i.e. the main character). He/she catches a character (i.e. collectable items) and mimics it with movements. The user’s physical activity determines the speed of the game. Children in the game take a role of different characters, match shapes and learn to listen, watch and respond with their body. Computer vision algorithm of mass centre determines the user’s x position. This value is then mapped to the avatar. Motion detection determines the user’s physical activity, which then defines the speed of games elements (i.e. collectable items).

In the game 2 Symmetry (Fig. 4 and as shown in the video (Online Resource 2)) children make connections between their own image on the screen and two halves of the objects. In this slow tempo game, the player needs to patiently control, with the whole body, the upper halves of the object by moving left and right to find the corresponding halves (i.e. matching shapes and colours). Computer vision algorithm for mass centre is used to determine the user’s x position. This value is then mapped to a digital object (e.g. upper half of the object). Motion detection is used to determine whether the user spent enough time comparing and putting together the two halves. If the corresponding halves have been joined correctly, the game continues with a new digital object.

In the game 3 Don’t wake up Mr. Smith (Fig. 5 and as shown in the video (Online Resource 3)) the user controls a falling ball with a seesaw to hit a cloud (the set of clouds bellow the see saw) and to avoid hitting the drums (the set of drums bellow the clouds). Mr Smith has his usual afternoon nap and must not be disturbed. The game suggests what is right, what is wrong, and why. Computer vision algorithm for mass centre is used to determine the user’s x position. This value is then mapped to an algorithm that defines the angle of the see saw. For example, when a user is positioned left, the left side of the see saw is down.

Some games also encourage children to express themselves with their body. Rules of the games do not prescribe a particular movement, as well as they don’t require a constant gaze on the screen. In the game 4 HulaHoop game (Fig. 6 and as shown in the video (Online Resource 4)) the user spins a digital Hula Hoop in order to move all three main characters from the bottom to the top of the screen (i.e. to spin characters on the cloud). Computer vision algorithm for mass centre is used to determine the user’s x position. If this position is within values for a character, then the character will start to move up. Motion detection is used to determine user’s physical activity. If this is within a certain range, the virtual HulaHoop will spin around the user, and a character will move up. If this activity is too low or too high, the hula hoop will drop down. The game is finished when all three characters are on the top of the screen.

In the game 5 Flying the cloud (Fig. 7 and as shown in the video (Online Resource 5)) the user’s physical activity transports the characters to the Spinning Island. Motion detection is used to determine the user’s physical activity, which defines the speed of the two characters on the cloud. The game is finished when characters on the cloud move from left to the right side of the screen.

In the game 6 a racing game Spinning River (Fig. 8 and as shown in the video (Online Resource 6)), the user moves dynamically left and right to control the avatar in order to avoid objects on the river. The user’s physical activity determines the speed of the game. Children learn to distinct shapes and colours. Computer vision algorithm for mass centre is used to determine the user’s x position. This value is then mapped to the avatar (i.e. the main character on the boat). Motion detection is used to determine the user’s physical activity, which defines the speed of game elements on the river.

Game 7, Racing game (Fig. 9 and as shown in the video (Online Resource 7)) is similar to the game above. The user moves left and right to control the avatar and avoid objects on the road. Children learn to distinguish shapes and colours. Computer vision algorithm for mass centre is used to determine the user’s x position. This value is then mapped to the avatar (i.e. the main character on the road). Motion detection is used to determine the user’s physical activity, which defines the speed of the avoidable game elements on the road.

In the game 8, memory game Deep Dark Woods (Fig. 10) the user controls falling objects left and right to put them on the corresponding place. The user explores what is behind the dark shadow, which helps him/her to understand that shadows and noises in the dark are just everyday things. Computer vision algorithm for mass centre is used to determine the user’s x position. This value is then mapped to the falling object. The user stirs the falling object towards the corresponding object. If collision between the two corresponding objects is detected, then the hidden object comes out of the dark.

In the game 9, catching game Wisdom Tree (Fig. 11 and as shown in the video (Online Resource 8)) the user moves the basket left and right to catch or avoid objects according to their shapes, colours and content. Computer vision algorithm for mass centre is used to determine the user’s x position. This value is then mapped to the basket at the bottom of the screen.

The shooting game 10 Fallen stars concludes the whole experience (Fig. 12 and as shown in the video (Online Resource 9)). The user needs to catch two stars at the time. An algorythm detects if a collision took place bewteen the position of the stars and the rectangular region (Fig. 2) of the mass centre algorithm. New stars randomly appear, each one on each side of the screen, after 4 seconds or before, if the collision took place.

3 Evaluation

3.1 Evaluation background

Evaluation of full body interactive games is complex, as the coordination of mental and physical skills is required for successful playing [13]. Different methods, based mainly on the observation and written output have been used in the past, while visualisations of human movement (ie. tracking) have been used in the analysis of sports and dance.

One of the first evaluation frameworks of full body interactive games [33] tried to answer how successfully the gamelets (i.e. sub elements of the games) incorporated the five important advantages of using computer vision over more traditional interfaces in video games; these were (1) Athleticism, (2) Expressivity, (3) Whole Body, (4) Vocabulary of Action and (5) Playability. Labanotation has been used in the interaction design context to evaluate Eyetoy Sony PlayStation games ([8, 21]), and it provided a valuable foundation for the design of movement-based interactions. According to Höysniemi & Hämäläinen [13], a successful full-body interaction playing of Kick Ass Kung-Fu required (1) locomotor skills, (2) nonlocomotor skills, (3) manipulative skills, (4) movement awareness, (5) body awareness, (6) spatial awareness, (7) focusing attention, (8) observing visual feedback, and (9) ability to remap movements. The above mentioned evaluations have used different approaches, and their pioneering work was extremely valuable, as it addressed issues of interaction and games design, such as the relation between the user movement and the games element on the screen. But as it was acknowledged that full body interactive games, such as Kick Ass Kung-Fu, are in fact as digital sports [10], because challenges are not only mental, but also physical, we felt that it was necessary to investigate also the physical aspects of the games.

Sports and health sciences and dance, on the other hand, have a great tradition of analysing human physical activity. Sports science analyses the performance of an individual player and a team. In professional sports, this includes analysis of biomechanics, physiology, motor skills, game tactics, techniques and behaviour. Software such as Simi Motion [29] used computer vision algorithms that performed motion analysis. Software Simi Scout [30] tracked the distance each player covered during a game. Speed and directions of movement, strokes and/or hits could also be measured. LucentVision system [26] tracked the motion trajectory of a tennis player. Data were visualized for further analysis with heat and coverage maps, speed charts, 3D images and animation, still and sequential imaging, and more conventional graphs.

Health experts have shown a great interest in the full body and other physically interactive games, especially Wii Nintendo. The research was dealing with the effects such games had in respect of spiroergometry, heart rate and the whole body kinematics [3]. Research comparing the energy consumption showed that playing Wii Sports used significantly more energy than playing sedentary computer games, but not as much energy as playing the sport itself [9], while the research comparing Wii Tennis and EyeToy Tennis showed that the energy consumption was greater in the later [3].

In the EyesWeb project [7], two expressive motion cues were considered [4]: the Quantity of Motion (QoM), and the Contraction Index (CI). Both cues were global indicators of human movement; QoM was correlated with the user’s energy, and CI with the space occupied by the user.

In dance, Labanotation is a system of recording movement, and Laban Movement Analysis (LMA) has been developed to analyse the movement. Originally devised by Rudolf Laban in the 1920’s [18], the analysis continues to be used in fields traditionally associated with the physical body, such as dance choreography, physical therapy and drama.

In the LMA, the Body category describes structural and physical characteristics of the human body while moving. This category is responsible for describing which body parts are moving, which parts are connected, which parts are influenced by others, and general statements about the body organisation. This area in sports science covers biomechanics with motion analysis and to some extent also the motor skills tests and physiology.

According to the LMA, the Space category identifies (1) kinesphere, which is the area that the body is moving within, and how the mover is paying attention to it; (2) Spatial Intention, the directions or points in space that the mover is identifying or using; (3) Geometrical observations of where the movement is being done, in terms of emphasis of directions, places in space, planar movement, etc. In the sports science research, one would conduct an analysis of the player position during the game, area covered, direction and speed of the movement, etc.

The category Time covers Effort, or what Laban sometimes describes as dynamics, and what Labanotation calls “Length of time it takes to do the movement” [18]. Sports and health science covers energy consumption through an endurance test using spiroergometry and monitoring the heart rate.

Kroflič [18] based her own framework of Time, Body, Space and Relationship on Laban’s work. The category of Relationship indicates that all categories are closely linked, and does not include any measurements.

3.2 Materials and methods

3.2.1 Participants, space and equipment

The experiment was conducted in Museum of Recent History, Celje, Slovenia in January 2008 with 8 participants aged between 5 and 13 years, and body height between 116–178 cm (Table 1). Each participant played 10 games (from Figs. 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12) in a continuous session of 20 min. AIS is targeting younger children (age 4–9), but older children were included for better distribution of body height and age, as well as to investigate whether AIS games could also be interesting to older children. No participant had any prior experience with Sony PlayStation EyeToy, Wii Nintendo or similar games.

Table 1 Participants and their characteristics

The experiment was conducted in a bright room without windows, with tungsten lightning. The playing area was limited to 4.50 m in width and 5.00 m in length. For safety reasons, several sports mats were placed on critical places to minimize the impact of a child hitting solid or firm objects, in order to minimize the risk of injuries.

Toshiba PC laptop with Intel Pentium 1,70 GHz processor and 1 GB RAM was attached to 42″ Panasonic widescreen plasma display. Namco web cam was used [8]. The experiment was recorded with 2 digital camcorders. Camcorder 1 (Fig. 13) was positioned in front of the participant and captured almost identical view as a web cam used as a sensor for the games. Camcorder 2 (Fig. 14) was positioned on the side and captured most of the playing area and the TV screen.

Fig. 13

View of camcorder 1

Fig. 14

View of camcorder 2

3.2.2 Evaluation methods and design

Before starting the evaluation, it was not clear what type of movement could be expected, therefore an open framework seemed to be the most appropriate.

The goal of the evaluation was to address the user’s physical experience. Ermi and Mäyrä [6] put the immersion in the main focus of the games experience. Immersion, in desktop games, means becoming physically or virtually a part of the experience itself, while players are constantly faced with both mental and physical challenges that keep them playing [6]. The elements related to pleasurable gameplay experience in desktop games are found in three immersion categories: audiovisuality, fantasy and level of challenge [6]. There are some similarities to sports games, such as winning, advancement, fun and humour, that are also elements of pleasurable experiences. However, in full body interaction games, such as Kick Ass Kung-Fu, which are considered as digital sports [10], challenges are not only mental, but also physical. Therefore, the evaluation was designed to establish and understand which elements influence the movement and physical experience of the user, in order to provide a feedback to the game designer. As no directly applicable research framework was found neither in previous works on sports and health sciences, or in interaction design, Laban’s and Kroflič’s framework was used as a basis for the evaluation, with additional research methods and strategies that have been used in games and interaction design (e.g. observations), as well as in sports science, dance (trajectory tracking, motion analysis, energy consumption, etc.) and computer games (quantity of motion—QoM). Using Laban’s Kroflič’s framework helped a designer to see a broader picture of what needed to be analysed in the games.

The category Body

In the category Body, the technology [29] for whole body kinematics or biomechanical measurements was not selected for our experiment. This technology is usually used in controllable environments, and requires reflective markers [29], which might cause problems to computer vision. A heat map, which shows areas of different intensity of movement with a scale of colours, was initially considered and tested for the category body parts. An EyesWeb’s [7] patch was created using Dydic arithmetic operation block. However, from the final image it was not possible to conclude which body parts were used, so the heat maps were abandoned. The following three parameters were finally applied:

• 1. The number of postures used in a game. Each video was analysed, and the number of postures participants used in each game (i.e. standing, bending, moving, lying, kneeling, running, crouching) was recorded. This sub-category was focused on general statements of the human body while moving.

• 2. Quality of the movement. Each video was analysed, and the observer gave a qualitative appraisal on the basis of how many different types of movement (i.e. jumping, bending, waving, swinging, drumming, gesticulating...) a participant used during the game. This sub-category was focused on smaller structural and physical characteristics of the human body while moving.

• 3. Body parts used in a game. In the analysis of each video, the use of the participant’s body parts was recorded in interaction with the game (i.e. trunk, legs, hips, arms and head).

Sub-categories “The number of postures” and “Quality of the movement” use similar elements, but they differ in focus and scale. For instance, if the user walks and waves, then walking gives a general statement of the moving human body, while waving is a smaller characteristic. But when waving is replaced by swinging with both arms while the user still walks, then swinging becomes a general statement, and walking a smaller characteristic.

The category Space

In the category Space, a coverage map [26] was chosen instead of the Contraction Index (CI) [4], because it provided more details, not only about the size of a playing area, but also for the directions of the movement. Direction of the movement corresponded to what Laban called the spatial intentions and geometrical observations, and was a simplified version of the visualisation of routes [30]. Gaze complemented the direction of movement, and was included in the analysis to find out whether a game required a constant gaze toward the screen, or whether the sound and storytelling could be used as a base for the gameplay. The following three parameters were applied for the evaluation of Space:

• 1. Size of playing area. Participant’s left leg from the video (Fig. 15) was tracked in each game to determine the size of the playing area, using the video from camera 2 (Fig. 15). Video was imported in Adobe Flash and participant’s left leg was traced with a pencil tool in each frame of the video.

  Fig. 15

  

  Tracking participant’s left leg and determining the playing area

• 2. Direction of the movement. Video and visualisations of tracking participant’s left leg (Fig. 15) were analysed to determine how many spatial lines (i.e. left and right, forward and backwards, circular, diagonal and standing) a participant used.

• 3. Gaze. Video was analysed to determine the ratio between the time a participant looked toward the screen, or when his/her gaze was turned away from it.

The category Time

In the category Time, the tempo was defined as a speed of movement, and the dynamics as a rate of change in the speed of the movement. For example, the sprint would be a high tempo game, and the long distance run a low tempo game. Although the sprint has a high tempo, it also has a low level of dynamics, as the speed does not change greatly during the race. Basketball, on the other hand, is a game with a high level of dynamics, as the speed of playing tempo varies a lot. To measure a player’s effort, equipment for spiroergometry [3] was not available, therefore the QoM [4] was used instead. The following three parameters were applied for the evaluation of Time:

• 1. Tempo. Each video was analysed, and the observer gave a qualitative appraisal of the participant’s tempo in each game.

• 2. Dynamics. Based on the video analysis, the observer gave a qualitative appraisal of the participant’s dynamics in each game.

• 3. QoM. Video 1 (Fig. 13) was analysed using the Quantity of Motion (QoM) patch in the EyesWeb software [4]. A value for each frame was stored and a mean for QoM for each participant’s game was calculated.

4 Results of the experiment with discussion

4.1 General observations from the experiment

For successful playing, coordination of the child’s physical and mental skills was required. For example in game 9 (Fig. 11), a participant needed not only to understand if a number was odd or even, but also to react quickly with his/her body to catch or avoid it. The challenge for the participants was to coordinate their movement with mental skills such as educational content, strategy, rules, digital objects etc. Similar to the creative movement [18] the participants had to think productively to resolve the problem (i.e. situation in the game) with a movement. It was noted before ([10, 13]) that successful playing of the full body interaction in Kung Fu Kick-ass required several motor and cognitive skills. Whole-body tasks in Harmony Space were both physically and mentally challenging—a physical and mental workout combined [12].

All children were excited by the experience, and had put a lot of physical and mental effort into the experience of playing 10 games; each child played continuously for a session of totally 20 min. The games were sequenced in a wave, beginning with a slower warming up, continuing with a moderate to faster playing tempo with more demanding physical movements, and slowing down towards the end of the playing session, concluding by a tranquil game of catching the stars.

Children were deeply immersed into the story, into physical activities and the game environment, despite being watched and observed in a public space. Visual immersion was important, as it was exciting for the participants to see themselves on the screen while playing. Only the youngest participant (Participant M1—Table 1) had a slow start and he needed more than one game to warm up, fully understand and enjoy the experience. He had no previous experience of attending a sports or dance club, and only had a limited experience of computers, unlike the majority of participants. In developing a strategy for successful playing, participants used their previous knowledge and experiences of playing desktop computer games, and physical experience of exercising, sport, and/or dance. Children, who were members of an athletics club, would prefer to run; those who were basketball players, would run forward and backward facing the screen all the time, while the dancers used more expressive movement.

4.2 Body, space, time/effort and relationship analysis

4.2.1 Body (analysis of the postures, movement and parts of the body)

Analysis of postures showed that the participant’s starting posture was always standing. According to the tempo and the dynamics of the game, participants used other postures, such as running, lying, kneeling, crouching, and others. Participants used more postures in games with a higher level of dynamics (games 3, 9 and 7). Less postures were used in the game 4.

Analysis of the movement showed that in order to successfully interact with digital objects, participants used movements such as walking, running, jumping, bending, dancing, mimicking, drumming, rollerblading, hitting, swinging with hands and standing still. More diverse movement was noted in games with a higher level of dynamics (games 1, 9 and 6).

Using analysis by Höysniemi & Hämäläinen [13] the following physical skills were identified: (1) locomotor skills (i.e. moving left and right to control avatars or objects, moving forward and backwards to reach the objects, moving forward to approach the camera or to readjust balance), (2) non-locomotor skills, such as speed (i.e. moving quickly to reach digital objects), balance (i.e. keeping balance after moving quickly), flexibility (i.e. bending to control an object), precision (i.e. patiently controlling objects), (3) manipulative skills (i.e. using limbs to control objects), (4) movement awareness (i.e. responding to the change of a digital object on the screen with a movement), (5) body awareness (i.e. knowledge of the position of the body and the digital objects), (6) focusing attention (i.e. distinguishing own image from digital objects), (7) observing visual and audio feedback (i.e. planning the next step).

Analysis of body parts used for interaction showed that most of the interaction was done with the body, less with arms, legs and head. This was expected in the games where the mass centre algorithms were used. Participants used more parts of the body in games with a higher level of dynamics. Specific interaction was noted when the story suggested a certain movement (e.g. to mimic a character, or to catch a star with the hand). A similar motion was noted by D'Hooge & Goldsmith [5], who stated that ‘When you see bubbles floating around you, your normal reaction is to pop the bubbles with your hands’. However, a sophisticated computer vision system would be needed for discerning body parts, as it was pointed out in Kick Ass Kung-Fu evaluation [10].

4.2.2 Space (analysis of the playing area, directions of movement and gaze)

Analysis of the playing area showed that games with a higher level of tempo and dynamics and a higher level of QoM required more space. Figure 16 shows combined playing area of all participants in game 2 (lowest level of QoM and tempo) and game 5 (highest level of QoM).

Fig. 16

Playing area in game 2 (left) and game 5 (right)

Participants also used more space in games in which they did not look at the screen (i.e. gaze) for most of the playing time, as this allowed them to play out of the display boundaries.

Analysis of the direction of the movement showed that most of the movement was done in a left–right direction. Some intentional movement forward—backward was also noticed (i.e. in mimicking characters) and some unintentional movement forward—backward (i.e. when readjusting position after a participant lost balance in a game with a high level of dynamics). Some participants realized that if they moved closer to the camera, it was easier to play. Participants used more directions in games with a higher tempo and in games where the rules did not require them to watch the screen most of time.

Analysis of the gaze pointed out that the participants observed the action on the screen most of the time. During the games with less strict rules and which did not require constant focusing on the screen, participants left the screen, used more space and moved in different directions, including in circles and diagonally. The importance of gaze was first emphasized in Kick Ass Kung-Fu [13], when participants felt confused when switching the gaze between one screen to another.

Further analysis of the playing area also informed the designer about the required size of the playing area. In Fig. 17 all visualisations from game 9 were combined. An average distance for each participant in the game was established, as presented in Table 2.

Fig. 17

Comparing participants’ movement and body height in game 9

Table 2 An average distance from the screen in game 9 with participants’ characteristics

Fig. 18

Comparing participants’ height and distance from the camera

Figure 18 illustrates the relationship between the average distance from the screen and the body height of participants and the camera. The trendline connects the Y position of the webcam and the position of the eyes of most of the participants, except the participants F3 (the tallest), and M4 (the smallest). Further analysis of the video of participant F3 showed that it was hard to see the details on the screen from a distance greater than 4.50 cm, which became the maximum distance. This result is in accordance with previous works by Warren [33], who suggested the same maximum distance for his full body interactive games. Participant M4 was physically very active, and found out that it was better for him to play the game not too close to the screen.

Left–right was the dominant direction of the movement in the experiment. The same was observed in most of the previous full body interactive games with low level computer vision algorithms, such as Me2Cam [5], Sony Eyetoy [21] and QuiQui’s Giant Bounce [14]. Left–right dominant movement was used by the authors in the design of educational full body interactive games, where players learned by making connections between the position of their body and the position of the digital objects in the direction left–right on the screen [17].

4.2.3 Time (tempo, dynamics and QoM)

Analysis of the dynamics showed that a higher level of dynamics was noticed in the racing and catching genres, while lower levels of dynamics occurred in the matching and memory games and in games that required repetitive movements. In games with a higher level of dynamics, a participant used more postures, movement was more diverse, and more space was used. Sometimes a participant moved quickly, waited a couple of seconds, and then moved again.

Analysis of the tempo showed a higher tempo in games where a motion detection was used to define the speed of the game. A high level of tempo was also found in the catching game. In the high tempo games participants used more of the playing area.

Analysis of the QoM showed a high value of QoM in games where the tempo of the game required a lot of movement with the whole body, and where the participants used more space, moving in different directions. Low value of QoM was found in games where the participants did not observe the action on the screen all the time. In such games many had left the screen for some seconds and in that time the value of QoM was zero. Low value of QoM was found in games that took longer to finish.

In the experiment, participants had 15 seconds breaks between each game. The designers were interested to know if this period of resting was sufficient for the regeneration. Further analysis of the QoM was conducted to detect periods of lower physical activity, which could indicate tiredness. An additional analysis of each video was required to check if a resting period was not a result of problems in the gameplay. First signs of tiredness were found in game 5, a game with a very high QoM, with younger participants (F1, F2, M1, M2, M3 and M4). In Fig. 19, a period of low QoM was found with participant M4 between 25th and 43rd second, when the participant took some active break (i.e. still exercising, but with less intensity). Similar break was found with participants F2 between 67th and 85th second and M3 between 110th and 130th second, respectively.

Fig. 19

QoM values of the participant M4 for the game 5

Another type of tiredness was found with participants F1, M1 and M2 (Fig. 20), where a gradual drop of QoM values started in 53rd second and continued to the end of the game.

Fig. 20

QoM values of the participant M2 in the game 5

No signs of tiredness were detected with two oldest participants (F3 and F4), who did not show any major drop of QoM value during game 5 (Fig. 21) or any later game.

Fig. 21

QoM values of the participant F4 in the game 5

The results of QoM analyses suggest that the games were physically more demanding for younger children. However, the games required substantial physical effort from all participants. At the end of the session, they were short winded, had red cheeks, and were thirsty.

4.3 Findings and implications on the game design

Not only the experiment itself, but also studying Laban’s work proved to be useful for informing game designers about issues important to the human body and human motion, and for making the transition towards the full body interactive games smoother.

In the category of the Body, not only the dynamics of the game influenced the movement, but also the rules of the game (e.g. to swim the river while avoiding all objects, or to catch certain objects), the interaction between the body and digital objects (e.g. to control the avatar left and right, slower or faster), the narrative component of the user interface (e.g. story and instructions that asked a participant to move in a certain way), the rhythm of the music, as a higher tempo required from participants to move faster, the visual feedback on the screen (i.e. a participant could see a projected image of himself/herself only within a limited distance on the left and right), and the physical space (i.e. size, lightning conditions and quality). Results of the evaluation helped the designer to understand how a particular full body interactive game, game genre or interaction offers an opportunity to induce a certain movement, such as smaller movements using genres that require patience, or expressive movements using storytelling.

Results of the Space category showed that the physical space is an important part of physical activity, as the size of playing area, objects (e.g. furniture), quality of floors (e.g. it is easier and safer to move on wooden floors) and lighting conditions in the space (e.g. light improves playing space) influenced the user’s experience. These results also informed the designer about the required size of the playing area. As most of the computer games require constant focus on the screen, one of the interesting findings of this experiment was that games with less strict rules did not require the participants to watch the screen all the time. When they turned away, they could explore the playing area outside the screen, and therefore used more space, making the playing area bigger. In the category Space, the main challenge for the designer is to create the environment, and to maximize the space as much as possible, to allow the physical activities that require more space (e.g. running, jumping).

The category Time informed the designer about how to use the length, tempo and dynamics of the game, with game elements such as music, gameplay, genres, to manipulate the participant’s effort. Many early commercial full body interactive games for Sony Eyetoy had a high tempo. Players had a high energy consumption, which resulted in exhaustion and a negative user experience. Most game genres trigger a competitive side in the player; if the player wants to achieve a better score, he/she needs to make a greater physical effort. An interesting approach was used in ‘Ere Be Dragons [2], where a heart beat monitor was introduced, which allowed playing the game only when the player’s rate was inside the healthy heart rates. In our experiment, slowing down was achieved by introducing a physical activity which required accuracy, smaller and finer motor skills; by manipulating motion detection algorithms to use their values for the speed of the elements in the game; by using music with lower tempo; by using genres such as memory or matching; and by making digital objects smaller or less visible.

Results of different measurements suggest that many parameters are closely connected. For example, the more postures participants used in a game, the bigger was the playing area. Therefore, the number of the postures can on its own indicate the player’s physical effort, as standing still in front of the screen would result in a lower physical effort than when moving left and right to control the avatar. The correlations recognised between some evaluation parameters give a designer an option to conduct a quick evaluation of the game in an early stage of development just with one or few measurements.

However, the designer needs to be aware that full body interactive games are not only about the physical effort and fitness. In sports games, dance and other physical activities, performer’s technique, tactics, variation and aesthetics of movement also play an important role, counterbalance the physical effort, and contribute to the final user’s satisfaction.

4.4 Discussion of evaluation methods

The work described in this paper, analyses and discusses the elements of a user’s physical experience within the Laban and Kroflič’s framework. Compared to the methods mentioned in earlier works (chapter 3.1 of this paper ([8, 13, 21, 33]), this evaluation framework focuses on how a user moves his/her body in a space in a certain time with an effort, rather than on the relation between the user and the screen ([8, 21]) or movement studies ([10, 14]).

While first evaluation frameworks [33] touched issues of physical activity (e.g. Athleticism and Expressivity), later works were more focused on a relationship between an image on the screen and the user ([8, 21]), somehow neglecting the fact that such games (e. g. Kick Ass Kung-Fu) were in fact digital sports [10]. Although some research analysed hart rates [10], and movement [13], physicality was treated partially. It did not address important elements of sports games and dance, such as duration, size of the playing field, the quality of flooring and other issues. Some research, however, addressed issues of energy consumption ([3, 9]), but not in order to help the game design (e.g. to define the optimum playing time), but rather to compare energy consumption of such games and real sports, while other studies mentioned an average [33] and optimum distance from the screen ([5, 21]).

Labanotation was found a potentially useful tool to support the design of movement-based interaction ([8, 21]). While it is useful, as it describes movement in details, the authors of this paper agree with [14] that the original Laban method is too detailed, difficult and laborious for the evaluation of computer games, as it was developed to notate dance, which is different movement than playing an AIS full body interactive game. In AIS games environment is constantly changing, and movements have to be continually adapted and changed. This might result in many more physical actions to notate, especially in a game with a high tempo. In this respect, the understanding of Laban and Kroflič’s framework plays a different role, as it gives a designer a framework to understand the main elements of movement and its correlations in full body interaction.

In the experiment, where each of the 8 participants played 10 games in a continuous session of 20 min, the distribution of age, sex and body height proved useful. Especially the inclusion of two participants, who were initially out of a target age for AIS, namely an 11 years old girl and a 14 years old girl with a body height of 178 cm, helped to understand the importance of body height and age in the experiment, and, although on a small number of participants, indicated that games could also be interesting to older audiences.

The experiment was conducted in a single session, therefore research question which require several sessions over a longer time period could not be addressed. Another evaluation with AIS games was conducted [17], which investigated the progress of the player over a period of two weeks.

The experiment showed that user’s characteristics such as body height, age, motor skills and abilities, previous experience of similar desktop games, experience in physical activities and sports, influence the playing in full body interaction, and should be considered as variables for future research and design of physical games.

4.5 Future work on AIS platform and games

It is not easy to compare AIS games with Me2Cam [15], Eyetoy [8], QuiQui Giant Bounce [28], Kick Ass Kung-Fu [16] and other projects mentioned in the chapter 2 of this paper, as although they might share the target audience, they used different equipment and algorithms, and were designed and tested for different screens. Me2Cam was designed for 15″ PC screen, while Kick Ass Kung-Fu had two screens with the camera not in front, but on a side [10]. Krueger’s VideoPlace [19] was an installation for a gallery, played in a dark environment, as older projectors were not powerful enough to display an image in daylight. How different conditions influence the playing experience, shows the comparison of AIS results with the Me2Cam [5]. Although both projects used similar games genres and principles, the results of this paper confront several of their findings [5]. The findings of this paper show that although the dominant direction is left and right, other movements can be used; an average distance from the camera is correlated to the player’s body height and gaze in games; where rules are less strict, the gaze does not have to be directed toward the screen all of the playing time. Such discrepancy of results suggests that playing in a bigger space in front of a large TV screen changes the user’s experience, compared to the image on a 15″ PC screen, which is too small to provide an efficient and relaxed interaction.

Kick Ass Kung-Fu [13] has been described as a digital sport. The authors of this paper consider AIS games similar to digital sports games. However, the rules in sports were carefully developed over a period of time, to make optimum experience, not only for players, but also for viewers, while the area of full body interaction is still in early stages.

Further research and development is needed to better understand how to develop a system for monitoring tiredness (i.e. fatigue) with QoM. In the experiment first signs of tiredness were found in game 5 only with younger participants, while no signs of tiredness were detected with two oldest participants. Such system would need to take into the consideration not only the age, motor skills and abilities, but also the order of the games, as a player might play only one game or choose different path of the story. Such system would identify the signs of tiredness and manipulate the games elements in order to lower the physical effort of a player or suggested a break.

Although the main focus of this paper was not a technical evaluation, its findings proved that interesting games can be designed with low level computer vision algorithms. In AIS platform the user without any problems controlled an avatar (or similar digital object) with whole body movement left and right, while tracking whole body movement up and down was used only in one game (game 10—Fig. 12). In the future we will investigate hands and head tracking to add this dimension into the games. Movement in all three dimensions is vital in sports and dance; one of the drawbacks of Kick Ass Kung-Fu [14] was that a player could not move threedimesionally. In AIS games this was less noticeable, as games were not requiring a specific three-dimensional movement. This paper also pointed out that movement was not always and only defined by the technology. Other games elements, such as story, narration and others mentioned in section 4.3 of this paper, also played an important role. However, new commercial products ([1, 27, 31]) already offer robust tracking systems, which give an accurate third-dimension measurement (depth) of a user’s position, which adds the missing dimension of the movement, and offers new possibilities for gaming.

5 Conclusions

The research was aimed to better understand mechanisms and principles in the design of user’s physical activity in full body interactive games for children. In the evaluation of an ambient interactive storybook with 10 games, elements important for the designer have been derived from an adapted Kroflič’s and Laban’s framework Body, Space, Time and Relationship, supported by sport science measurements. Participants’ physical experience was evaluated through the analysis of postures, the quality of the movement, the body parts used in the interaction, the playing area, the direction of movement, direction of gaze, tempo, dynamics and QoM. The experiment also informed the designer about correlations between the measurements, game elements, user characteristics and the physical space (Fig. 22).

Fig. 22

Main elements of the design and evaluation of user’s physical activity in full body interactive games

The results of the study can be summarised into the following main findings:

• 1) For successful playing of full body interactive games, coordination of the child’s physical and mental skills is required.

• 2) Visual immersion (i.e. seeing mirror image on the screen) is an important part of games, and is exiting for children.

• 3) Children used their previous knowledge and experiences of playing similar desktop computer games, and physical experience of exercising, sport, and/or dance.

• 4) The diversity of the movement is defined by several factors: the dynamics and rules of the game, interaction between the body and digital objects, narrative component, rhythm of the music, visual feedback on the screen, and the physical space.

• 5) Physical space is an important part of physical activity, as the size of playing area, objects, quality of floors and lighting conditions in the space influence the user’s experience.

• 6) The suggested surface for the playing area is 4.50 m (depth from the screen) by 5.50 m (width).

• 7) The playing area can be bigger if games do not require a constant gaze directed to the screen. In such games audio interfaces can play an important role.

• 8) Tempo and dynamics can manipulate user’s physical effort by introducing movement that requires smaller and finer motor skills; by manipulating motion detection algorithms, by using music with lower tempo; by using suitable genres and by making digital objects less visible.

• 9) Original Labanotation, although potentially useful, is too complex and laborious. Laban’s Kroflič’s framework helps a designer to see a broader picture of what is important in the full body interactive games, and to evaluate the games in order to recognise the main parameters and their correlations.

• 10) The EyesWeb’s QoM patch can be used for measuring and analysing participant’s physical effort, and to identify tiredness through periods of low user activity.

• 11) Comparing to desktop games, user’s characteristics such as body height, age, motor skills and abilities, previous experience of similar desktop games, experience in physical activities and sports influence the playing, and should be considered as variables for further research and design of full body interactive games.

Design and evaluation of full body interactive games has opened the communication between experts in computer science and the fields of drama, dance and sports. The results can be used in further theoretical and practical work of full body interactive games, and in further developments of educational and edutainment full body interactive games and stories.