Development and Validation of a Mixed Reality Exergaming Platform for Fitness Training of Older Adults

Abstract

Populations are becoming older in developed countries because of low birth rates and increased life expectancy. At the same time, sedentary lifestyles are the 4th mortality factor worldwide. Exergames have been shown to motivate players to get physically active by promoting fun and enjoyment while exercising. However, most exergames are not designed to produce recommended levels of exercise that elicit adequate physical responses in the aged population. Designing meaningful and enjoyable exergames for fitness training in older adults pose critical challenges in matching user’s needs and motivators with game elements and typically do not consider the usability and cost-effectiveness constraints of target end-users and institutions. Here, we present the conception and field validation of PEPE—a gaming platform with mixed reality components whose purpose is to fight a sedentary lifestyle by promoting active aging in elderly-care centers. We show that PEPE’s custom-made exergames can be successfully used by trainers for delivering sustained long-term training, with benefits in terms of efficiency, elicited physical activity, and perceived effort. Also, PEPE improved the overall perception of the quality of life and social relations in institutionalized older adults.

1 Introduction

The world population is aging at a fast pace. According to the United Nations, by 2050, 17% of the population will be 65 years old and over (United Nations, Department of Economic and Social Affairs, and Population Division 2020). This is a projected increase of 84% in Australia and New Zealand and 48% in Europe and Northern America. In Portugal, predictions indicate that the aging ratio will almost double by 2080, to 300 elderly people for every 100 young people, and the most aged region will be the Região Autónoma da Madeira, with 429 elderly people for every 100 young people (“Resident Population Projections 2018–2080” n.d.). These demographic changes parallel an increase in the life span that challenges our societies and the sustainability of health systems as we know them today. In addition, globally, more than 1.4 billion adults are at risk of disease from not doing enough physical activity (Guthold et al. 2018). In contrast, physically active elderly (≥60 years old) are at a decreased risk of cardiovascular mortality, breast and prostate cancer, falls, cognitive decline, and dementia and have a better quality of life and cognitive working (Cunningham et al. 2020). Dealing with the social and economic burden resulting from the increasing number of age-related disabilities represents a significant challenge for modern societies, particularly during the subsequent decades when an important aging of the world population is expected. Therefore, there is the need to develop solutions to promote active aging and prevent sedentarism, as well as to find new tools to support the large populations of patients that suffer chronic conditions because of aging.

In this context, the advances in information and communication technologies (ICT) and assistive technologies have the potential to increase quality of life (Chaumon et al. 2014) and change healthcare delivery models, reducing costs, and improving monitorization (Andreassen, Kjekshus, and Tjora 2015). These technological advances have a strong potential toward novel and low-cost treatments, health promotion, and disease monitoring and prevention. At a community level, such eHealth applications support the development of personalized and person-centric care methods and services that easily transfer from clinic-based training to at-home applications for telerehabilitation, creating a continuum of diagnostics and training possibilities, making real the new healthcare paradigm of ‘home as care environment’ (Datta et al. 2015; Korzun et al. 2017).

One possibility is the use of gaming technologies for physical exercising, known as exergames. Exergames have been shown to motivate players to get physically active by promoting fun and enjoyment while exercising. However, most exergames are not designed to produce recommended levels of exercise that elicit adequate physical responses for optimal training in the aged population. The design of meaningful and enjoyable exergames for fitness training in older adults poses critical challenges in matching user’s needs and motivators with game elements. These challenges are often due to the lack of knowledge of game preferences of older adults, their little, or no technology literacy and reduced involvement of the target population in the design process. Furthermore, exergame platforms design typically does not take into consideration the usability and cost-effectiveness constraints of target end-users and institutions. Aspects like the simplicity of use for healthcare professionals, easy configuration for users with different motor abilities, easy maintenance, and setup are often disregarded and compromise the frequent and long-term use of the resources.

In the last 20 years, many mixed reality gaming platforms have been proposed engaging users through physical activity and some gained popularity in commercial devices. Two main components characterize the different solutions: the display type and the command interface. Most current solutions use portable display types carried by the player, such as headsets, mobile consoles, smartphones, and tablets. The games are commanded by moving the device around in the surrounding space and through interaction with tactile displays, buttons, or motion capture sensors on the device. A successful case is Pokémon Go, where players carry their mobile phones outdoors to collect items, with reported benefits in the reduction of psychological distress in workers (Watanabe et al. 2017). Although mixed reality games are mainly targeted at young adults, the aging population can also benefit from such an approach as it is prone to have a sedentary lifestyle and lack of motivation for exercise. However, decaying physical, perceptual, and cognitive capabilities in this population set specific constraints in the games and gaming platforms. Smartphone or tablet screens are often too small for older adults’ visual acuity, and headsets are hard to set up and uncomfortable. There are, however, some intergenerational approaches such as the Age Invaders game (Khoo et al. 2008) to put together in play children, their parents, and grandparents in a mixed reality environment (Khoo et al. 2009).

Another approach is to use human body signals to interface games and interactive applications. This has been widely promoted among the game user research community (Hughes and Jorda 2021). Measuring players’ responses when interacting with games by using physiological sensors provides the possibility of knowing seamlessly players’ inner states that can be beneficial to adapt the experience. The most popular body signals used to create physiologically adaptive games are (i) electrocardiography (ECG) and other cardiovascular sensing (e.g., photoplethysmography), (ii) electroencephalography (EEG), (iii) electrodermal activity (EDA), (iv) electromyography (EMG), and (v) eye-tracking (Hughes and Jorda 2021). Physiological signatures of human states such as stress, cognitive load, cardiorespiratory performance, mood, and even emotions are constantly used to create adaptive experiences in interactive systems (Cowley et al. 2016). The concept of feeding back the measured body signals in real-time has also been popularized when using games for training self-regulation skills, known as biofeedback, and it is considered the grandparent of the more modern physiological computing approaches (Pope et al. 2014). A more sophisticated way of using body signals is to empower the system with algorithms capable of inferring human states (e.g., stress, engagement) by using physiological signals and react accordingly. This concept, known as the biocybernetics loop, uses knowledge from control theory to close the loop, allowing real-time automatic modulations to take place during the interaction for enhancing user experience (Pope et al. 2014). Biocybernetic loops to create physiologically adaptive applications can enhance the experience by assisting players in adjusting the game challenge via modifying the difficulty (e.g., dynamic difficulty adaptation). Although it is a fascinating and promising area in the field of affective and adaptive gaming, a recent review pointed out the lack of studies documenting and evaluating physiologically adaptive systems that can work outside research laboratories (Loewe and Nadj 2020). This aspect could be due to many variables associated with (i) the difficulties when interfacing physiological sensors to game engines, (ii) the challenges in deploying physiologically adaptive games in non-controlled environments (e.g., connectivity, data transmission), and (iii) the lack of integrated solutions (both hardware and software) that are user-friendly and do not require technologists for operation. Most of the applications that have used automatic adaptation driven by body signals are in the aviation and military applications, where the conditions are strictly monitored (e.g., body movements), and the activities are reduced to highly controlled working tasks (Loewe and Nadj 2020). In exergames, the automatic adaptation using fitness variables (e.g., heart rate) that can maximize the benefits of gaming while exercising has been previously implemented (Robinson et al. 2020). This concept, called the dual flow model, uses the conventional flow model (human state of full involvement) proposed by Csikszentmihalyi and adds an extra dimension of physiological correctness (Sinclair et al. 2009). The design of exergames considering elements of the dual flow model has been proposed as a way to create experiences that can be enjoyable and effective in eliciting the desired physiological effects (Martin-Niedecken and Götz 2017; Muñoz et al. 2018). Interfacing custom-made exergames with commercially available wearable devices have been proposed as one of the most feasible solutions with the potential to be adopted by players outside research laboratories (Pope et al. 2014; John Edison Muñoz et al. 2016a, b, c). However, commercially available solutions that combine body signals and exergaming are scarce, and they can be mainly found as experimental prototypes in research laboratories.¹

Here, we present the conceptualization and field validation of PEPE—a gaming platform with mixed reality components whose purpose is to fight a sedentary lifestyle by promoting active aging in retirement homes. Its conceptualization followed several user-centered design cycles with end-users, therapists, gerontologists, physicians, sports scientists, and elderly institution managers, to provide easy to use, enjoyable, and effective experiences for both the exercise practitioners and trainers. The end result—PEPE—is a mixed reality platform that can be easily transported, configured, and deployed in elderly institutions to promote exercise. PEPE incorporates feedback from seniors from its physical implementation to its games, allowing the informed design and gamification of fitness training routines. The system uses floor-projected games and depth sensing, where the users can interact through body movements, either standing or sitting. By playing exergames, older adults can work on their physical fitness while training cognitive function through a rich diversification of stimuli. PEPE allows users to engage in personalized task-oriented activities, engaging motivational factors, a key aspect for successful training (Rizzo et al. 2011). Additionally, PEPE integrates physiological computing for online adaptation of exercise regimes. Further, the inclusion of multi-user gameplay to enable social interactions among users makes the approach unique. Bringing in social aspects into exercising can increase adherence as well as resistance to age-related cognitive decline (Dause and Kirby 2019). PEPE has been shown to be effective in stimulating elderly to practice physical exercise with the addition of fun and social interaction in multiple settings. Finally, the proposed approach has a high generalization potential as it can be applied to many other domains where goal-oriented repetition is needed to learn a skill (e.g., professional sports, dance, martial arts, etc.).

2 Developing PEPE: Portable Exergames Platform for the Elderly

PEPE is a novel, integrative, and cross-disciplinary approach that combines innovation and fundamental research in the areas of human–computer interaction, serious games, and physiological computing. Our goal was to develop a new generation of ICT-based solutions that can transform healthcare by optimizing resource allocation, reducing costs, improving, and enabling novel therapies, thus increasing quality of life. To that end, we developed an adaptive mixed reality physical training tool that can deliver online feedback on performance to prevent sedentarism, support active aging, and provide personalized tools for function (re-)training in the elderly population, which can be achieved thanks to its monitoring capacity by employing biosensors, computer vision systems, and exercise performance data (Fig. 1).

Fig. 1

PEPE consists of a wheeled mobile platform with a depth sensing camera and floor projection capabilities. It delivers a customizable exercise program through gamified activities and physiological adaptation that can be used through full-body interaction

2.1 User-Centered Design Targeting End-Users and Institutions

As our work targets institutionalized older adults, the perspectives of the caretakers and institution managers must be considered. In group activities, occupational therapists have the challenge of preparing sessions of physical or cognitive stimulation exercises. The preparation should be simple to prevent the drop of motivation in both the participants and the therapists. Games based on devices like headsets or smartphones often require a significant overhead in preparing the devices and the gaming space. Furthermore, maintenance and economic aspects are involved in the decisions of the institution managers to acquire those devices. In this section, we describe a study on the design of a mixed reality platform considering the expectations and needs of the users, therapists, and directors of elderly-care services. This study is based on user-centered design techniques to generate a customized solution that attempts to satisfy the desires and constraints of all interested parts. We departed from a technological basis and logistic requirement imposed by the nature of our research project augmented human assistance (AHA)² (Gouveia et al. 2018) associated with this work: The infrastructure required for the mixed reality platform should be easily transported and installed in multiple end-user sites to test and deploy the proposed solution in several contexts.

From that initial requirement, we performed an exploratory study in three elderly-care institutions: one private institution more focused on daily occupations, and two senior residences (one public and one private) with occupational and care services. The users and professionals of the institutions were consulted on the desirable characteristics of a platform for mixed reality games. Three sessions of about 60 min each were run in each institution, orchestrated by an investigator who watched, took notes, and recorded audio, supported by a professional of the visited institution. Participants were from multidisciplinary areas of the geriatric sector: 2 psychologists, one gerontologist, one occupational therapist, one psychotherapist, and one technical director. A total of 24 older adults, ranging in age from 66 to 94 years, were consulted during the three sessions. Seven participants were male, and 17 were female. Eleven participants had some type of motor disability. During the first session, we presented very basic concepts of possible instantiations of portable mixed reality platforms. In the second session, some drawings and sketches based on the outcomes of the first session were presented. Finally, participants could interact directly with a prototype developed following the user-centered design process in the last session. More details on the study can be found elsewhere (Simão and Bernardino 2017).

The results of the study pointed out in three main directions. The first issue pointed out by professionals related to the cognitive and motor limitations of some older adults, for example, dementia or reduced mobility. This implies that solutions requiring non-natural interfaces to control the games, like smartphones or joysticks, should be excluded, and games should be controlled by simple body gestures. This suggested non-intrusive motion captures devices, like the Kinect, to be the most adequate. Also, due to the motor limitations, participants should be able to play while standing, or sitting by people using wheelchairs or crutches, and controlled by any body part that is functional. So, game control should be easily customized for each particular player. As limitations in perceptual acuity demand for large displays of information, this also excludes smartphones, tablets, or mobile console platforms, instead suggesting projection-based mixed reality solutions.

The second issue pointed out by the professionals was related to the quick and easy setup of the exercise sessions. Due to the shortage of staff, time is an essential resource. It should be dedicated to effective exercise and not spent on complex setup procedures like installing wearable devices or external motion capture systems. Space management is also an issue since some rooms in institutions may be used for multiple purposes. Thus, the installation and uninstallation of the setup should be optimized. Also, the exercise sessions may have to be run in different spaces and sometimes in individual rooms. These constraints led to the concept of a compact platform that could integrate all the necessary components and be easily transportable between rooms.

The third issue was related to ergonomics and esthetics aspects. Professionals stated that the platform design should have an empathic design to increase the levels of acceptance and participation. Several forms, colors, decorations, and materials were proposed and iterated during the study. Esthetic lines from the robot Vizzy (Moreno et al. 2016), also used in the research project AHA, were exploited to reinforce a technology “brand” associated with this research. Combining the different user constraints with the initial requirements of the research team (easy transport, deploy, and test in different institutions), this resulted in a wheeled and compact system containing all necessary system components (motion sensor, projector, computer, configuration interfaces), able to fit in a normal car boot. Figure 2 presents the three main concepts developed during the user-centered design sessions. Both professionals and older users voted on the first design at the end of the sessions due to more empathic colors and rounded forms.

Fig. 2

Different design concepts worked out in the user-centered design sessions that illustrate the shape and color of the PEPE mobile platform when parked. The size is approximately of 80 × 60 × 60 cm. All designs consist of a wheeled platform that contains all hardware components, with openings for the floor projection and depth sensor, and a lid on top to uncover the access to the monitor and computer system

2.2 The Mobile Platform

During the user-centered co-design sessions, the concept of a compact and standalone prototype of the mixed reality platform for exergames was developed, integrating all the necessary components for the interactions and easy and quick transport and setup. The main components of the platform are the game interfaces for the users (display, command) and configuration interfaces for the professionals (touch screen). All components are managed by a PC computer.

The design of PEPE is presented in Fig. 3. Figure 3a shows the front view. Inside an enclosure of approximately 77 × 62 × 68 cm (length × 6 width × 6 height) are a computer and a projector. The projector beam passes through an opening in the enclosure to project the game elements on the floor. On the outside, fixed to an articulated arm, there is a Kinect sensor to sense the body pose and movement of players. The monitor in the front displays additional information of interest to the player but, in some cases, can be distracting to the player; thus, it can be removed. The monitor in the back displays information to the health professional and has a touch screen to facilitate the input of configuration commands. The articulated arm can be folded to store the external elements inside the enclosure (see Fig. 3b). Figure 3c shows the rear part with the detail of the opening on the back to fold the articulated arm, the rear touch screen panel for configuration, a folder with a wireless keyboard/mouse for advanced configuration and maintenance, and the mains plug. A mobile base with four freewheels supports the platform. To transport the platform in non-flat surfaces or over obstacles, there are two handles on the sides.

Fig. 3

PEPE prototype. Top row—with cover: a front view, b folded for car transportation, c rear view. Bottom row—folded views without cover: d lateral view, e front view, f top view. Dimensions are indicated in cm

Fig. 4

Biocybernetic loop engine is a physiological computing tool created to facilitate integrating body signals into games and interactive applications. Top: acquisition panel. Bottom: adaptive rule console

A critical decision had to be made regarding the projection surface. From the point of view of ease of configuration, two alternatives were considered: projection on the floor or a wall. Both technical and practical issues favored the decision for floor projection. First, image focus and sharpness exclusively depend on the distance between the projector and the projection’s surface; therefore, a floor projection requires calibration once. Instead, a wall projection requires a re-calibration every time it is moved. Second, a wall projection configuration is hard to prevent occluding the projection (if the player is between the projector and the wall) or occluding the player’s view (if the platform is between the player and the wall). Third, a projection on the floor ensures that the game area is free of obstacles and that the user actively observes the floor, reducing the chances of locomotion hazards.

However, there are two main issues in floor projections requiring attention. First, textured, very reflective, or dark floor surfaces can significantly change the graphical aspect of the game projections. In these cases, a white, non-reflective mat, or carpet can be used in front of the platform. If external illumination is kept at controlled levels, even dark and textured pavements like wood can be used comfortably (see Fig. 1). Second, the angle of players looking to the floor should not be too steep. This can be prevented by letting players play in a sitting position or at a larger distance from the projection.

2.2.1 Biocybernetic Loop Engine

The creation of software tools that streamline the integration of physiological signals in interactive games has been investigated, and multiple systems have been proposed. Neuromore³ and NeuroPype⁴ are two of the most interesting software tools that have included visual language programming and compatibility with popular physiological sensors (both research-grade and wearables) and communication protocols (e.g., lab streaming layer—LSL). A behavioral and affective rule-based tool was also created using one of the most popular game engines, Unity (Unity Technologies, San Francisco, USA) (Benlamine et al. 2021). Nevertheless, these software tools are mainly focused on algorithms and applications using neurophysiological signals (e.g., EEG), and their use to create physiologically adaptive exergames has not been reported. To tackle the previously reported limitations when integrating physiological signals into adaptive games, we have created a freely available integrative software tool that can be used to create and validate physiologically responsive exergames, the biocybernetic loop engine (BL engine)⁵ (Muñoz et al. 2017).

The BL engine utilizes the closed-loop construct to create rule-based adaptations using physiological signals as inputs to modify game variables in real-time. It integrates multiple sensing technologies and a rule-creator environment that allows physiological computing designers to architect the logic behind the intended physiological adaptation to be integrated into the exergames. The BL engine is divided into three modules:

• 1. Signal Acquisition (Fig. 4, top panel): The BL engine uses external software clients that access services (e.g., raw data, processed features) and allows streaming them through user datagram protocol (UDP), a popular message-oriented layer that facilitates data structuring and transmission. The BL engine uses a graphical user interface to allow capturing information from cardiovascular sensing devices (e.g., Polar Chest Strap Sensor, BioPlux, Bitalino), facial expression through the EmoCam (Freitas et al. 2017), which uses the Affectiva SDK,⁶ EEG data using the Muse EEG wearable sensor and others using the Reh@Panel (Vourvopoulos et al. 2013). The BL engine is able to capture information from multiple devices simultaneously.

• 2. Physiologically adaptive rule creator (Fig. 4, bottom panel): To create rule-based adaptations, the BL engine includes a console that runs in real-time and allows creating IF/THEN rules using both physiological features and game variables. The environment has multiple blocks to receive features from multiple protocols (e.g., UDP, open sound control—OSC, LSL) and create rules using comparisons, mathematical, and/or logical operations, as well as game output blocks to modulate game variables in real-time. Visual scripting allows the creation of adaptive rules via dragging/dropping those boxes and connecting them to create the physiological pipeline (e.g., IF heart rate is more than 100 BPMs, THEN increase game difficulty).

• 3. Communication with exergames: Games developed in Unity can be transformed into physiologically adaptive games using the BL engine.⁷ A game connector (prefabricated package in Unity) facilitates the integration of custom-made exergames with the adaptive rule creator; therefore, the modulation in the game can be programmed in the BL engine. The communication between the BL engine and a game is bidirectional, meaning that, the game informs the BL engine which variables are susceptible to be adapted, and in turn, the BL engine streams back the value changes that should be applied.

The BL engine has been validated as an integrative and agile tool for integrating physiological adaptation in exergames (Muñoz et al. 2017) and virtual reality simulators (Muñoz et al. 2016a, b, c).

2.2.2 Exergames

A set of 5 different exergames were developed to address the main dimensions of fitness training according to the recommendations of the ACSM (American College of Sports Medicine and Bushman 2017). These games were created in a user-centered design process involving researchers, sports professionals, and older adults as described in (Muñoz et al. 2019) and reproduce a cultural journey of Portuguese traditions. These are described in the following:

• 1. Grape Stomping (Fig. 5a): It replicates the traditional methods of wine production, where people step on grapes to extract the juice to produce wine. The game aims to produce as much grape juice as possible, which elicits a stepping-in-place exercise, typically used in aerobic training. This is combined with arm pulling motion for extra variety. This game presents three barrels, and grapes are brought into the play area with a conveyor belt. The grape bunches can come in three kinds, green, red, and rotten (distractors). The players stand on the projected barrels and, by flexion–extension of the arms, catch the grapes from the conveyor. Once in the barrels, the player can tread the grapes, which are converted into a rising level of juice contained on the barrel. Each grape bunch needs some steps to be successfully processed. As soon as the grape juice hits the top that barrel starts emptying its contents through a channel and becomes unavailable to play, forcing players to move laterally into another barrel. The game can be configured to have an extra cognitive difficulty layer by having each barrel requiring specific amounts of red and green grape bunches. Pulling rotten grapes into any barrel freezes the barrel for some time. The height that the feet must be raised from the ground to stomp a grape successfully can be configured at startup, together with the percentage of rotten grapes (distractors), the grape type requirements per barrel, and the time between new grape bunches on the belt. This allows an adjustment of the exercise difficulty.

  Fig. 5

  

  Set of exergames developed. Grape stomping (a) and Exerpong (e) train aerobic fitness. Rabelos (b) trains upper and lower limb strength while the Exerfado (c) and Toboggan Ride (d) train motor ability

• 2. Rabelos (Fig. 5b): For centuries, the Douro river valley has been a wine-producing region of Portugal, famous for the port wine. In the past, to transport the wine barrels downstream, from the vineyards to the city’s cellars, wooden cargo boats, named Rabelos, were used. This game replicates those voyages. The goal of the game is to collect as many barrels as possible while avoiding the river rocks. The player is in charge of navigating a Rabelo boat downriver, avoiding obstacles, and docking on the margins to collect barrels. The game aims at exercising upper limbs and takes a third-person perspective from behind the boat, where the riverbanks are aligned with the projection’s lateral edges. The boat position on the river is controlled through the player’s waist position, directly mapped to the projection or by leaning the trunk sideways. To move the boat forward, an arm rotation gesture that replicates the rowing activity is required. As the players row the boat downriver, they encounter rocks they have to avoid through lateral movement and barrel-filled docks at the river’s margins. These docks must be approached, and their barrels collected via an elbow extension-flexion motion. The difficulty is set by adjusting the rowing mode (light or hard), the distance between docks, rocks, and their probability of appearing during gameplay.

• 3. Exerfado (Fig. 5c): This game reproduces the environment of a typical Fado house from Lisbon, Portugal, where people go at night to eat, drink, and listen to live music. Inspired by the Guitar Hero video game, Exerfado resorts to music’s potential as a physical activity stimulator. The projection renders a keyboard on the floor with seven keys aligned with the projection’s bottom. The player stands at the bottom of the projection, and both feet control which key they want to activate as musical notes travel downwards. The goal is to have the correct key activated when a musical note hits it. Therefore, the player must play the piano with their feet in synchrony with the visual cues. Over which track each note appears depends on the pitch of the music being played, with low pitch-making notes spawn at the keyboard’s left keys and high pitch at the right. Some special notes can be activated by an arm “swiping” movement. This activity is intended to train agility in both upper and lower limbs. This swiping gesture can be set to short or wide arm extension, enabling it to be personalized to users with different capabilities. Missed musical notes lead to distortion of the song being played, producing negative audio feedback. The music to be played can be chosen from an extensive list of midi files, each with different durations. By setting, at startup, the falling notes’ speed, the percentage of bonuses, and the time between consecutive notes, the difficulty is controlled.

• 4. Toboggan (Fig. 5d): In Madeira, Portugal, a unique way of transportation was used in the past. Wicker toboggans, driven by two people, would carry passengers downhill, from the hills to the city center. In the game, this activity is recreated virtually. The toboggan and player are presented in the center of the screen from a third-person perspective. Lateral movement is controlled just like Rabelos, by moving sideways along the bottom edge of the projection. Alternatively, the lateral movement can be also controlled by trunk rotation. The speed is adjusted through trunk inclination; leaning the trunk forward accelerates the toboggan while leaning backward deaccelerates it. The trunk inclination angles can be adjusted to each user for both acceleration and deacceleration options. Over the path, there are pedestrian crossings and car intersections to force the player to slow down. While the game’s goal is to drive as far as possible in the allotted time, there are also obstacles to avoid, bonuses to collect, and speed limits to keep. Difficulty is set by changing the distance between bonuses, and the percentage of bonuses and obstacles during game play.

• 5. Exerpong (Fig. 5e): This is an exergame adaptation of the classic games of Pong or Breakout, used to provide a fast-paced game experience aimed at training aerobic endurance (Muñoz et al. 2016a). In this game, the player controls a virtual paddle through lateral movements while a ball bounces around the walls. The player, who stands along the bottom of the projection, has his or her waist tracked, and the game matches the paddle location on the screen with it so that both player and paddle are always aligned. The ball bounces around the other three edges of the screen covered by walls; the player must then use the paddle by moving laterally along the bottom of the projection to prevent the ball from going through the lowest edge. A pattern of colorful bricks is represented at the center of the screen; these bricks get destroyed whenever the ball passes over them twice. The game’s goal is to clear these bricks without letting balls pass through the bottom of the screen. The game difficulty can be adjusted by varying the paddle’s width and the ball’s size and velocity. Alternatively, the game can adjust its difficulty according to the player’s physiological responses when controlled by the BL engine.

All exergames implement the same rewarding system which consists of offering a medal whenever one of the three ranks is reached, which can be bronze, silver, or gold. Those ranks are set proportionally to the difficulty setting and gameplay duration. Moreover, all games are able to store user data and display the score evolution over time.

3 Field Studies

This section reports on three different studies performed with PEPE, to validate (1) its effectiveness as a tool for physical activity training in a longitudinal randomized controlled study, (2) assess the feasibility of the biocybernetic adaptation, and (3) to assess its acceptance by end-users in elderly-care institutions.

3.1 Impact of Exergames in Physical Activity in the Senior Gym

To evaluate the effectiveness of the five exergames presented previously (Gonçalves et al. 2017) in eliciting moderate–to-vigorous physical activity (MVPA) levels in the elderly, we conducted a three-month-long randomized controlled trial (Gonçalves et al. 2021). Two conditions were tested, each with its corresponding participants’ group: a combined exercise (exergames and conventional) training group and a conventional exercise training group, acting as a control. Both groups underwent two weekly physical training sessions, equivalent in frequency, intensity, time, and type (FITT) (Heyward and Gibson 2014). The ACSM recommendations for multidimensional training for older adults (American College of Sports Medicine and Bushman 2017) were followed to structure the training, while the difference between the groups was in the exercise modality practiced at the sessions.

• Exergames group—Combination of exergames and conventional training. Engaging once a week in an individual exergames session and also on a conventional group exercise session.

• Control group—Conventional training. Engaging in conventional exercise group sessions two times per week.

All sessions were performed in an exercise room of a local senior gymnasium. Conventional training sessions were based on the gym’s exercise patterns. Odd-numbered sessions (1–23) of both groups were always conventional exercise, while the even sessions (2–24) were conventional for the control group and exergames for the exergames group.

We relied on both objective (measured) and subjective [rate of perceived exertion (RPE)] data to quantify physical activity (PA) levels. The ActiGraph WGT3X-BT (Actigraph, Florida, USA) accelerometer was used to quantify PA: time people spent in MVPA (in minutes), the EE (metabolic equivalent—METs). This sensor has been widely used and is considered a gold-standard tool to quantify PA in different populations (Chu et al. 2017). Subjective data on the levels of physical exertion after each exercise routine was collected using a pictorial version of the 0–10 rating of RPE scale OMNI (Chodzko-Zajko et al. 2009).

A total of 31 active community-dwelling older adults were recruited where the study took place, with inclusion criteria of: 50–75 years old, able to read and write, members of the gymnasium for more than three months, able to understand the procedure, game rules and goals, no severe visual impairments, no impediment to exercise practice, no severe or unstable heart diseases, and no falls over the past six months. 16 were allocated to the control group (12 females, age avg. 69.1 SD 4.4) and 15 to the exergames group (10 females, age avg. 67.6 SD 5). All the participants gave their informed written consent.

Data was divided into two, the odd sessions consisting of the conventional sessions of both conditions and the even sessions of both conventional and exergames. This allowed us to analyze: (1) the differences between conventional exercise and exergames’ sessions and (2) the users’ response to the exercise training program. For analysis of these data, a two-way mixed MANOVA was used. The between-subjects factor was the training program, each participant was allocated to (2 levels), and the within-subjects factor was program progression (session number). The dependent variables were RPE on the OMNI scale, METs spent, and minutes of MVPA in a session. Separate ANOVAs were run for each dependent variable to ascertain which ones were genuinely affected by the training program, where the degrees of freedom were corrected using the Huynh–Feldt estimate of sphericity. There was incomplete data from accelerometry on the 1st, 2nd, 6th, 7th, 9th, and 11th weeks of the study. Thus, as the two-way mixed MANOVA requires complete data, we removed those weeks from the analysis. Additionally, data from the 5th and 8th weeks was also excluded for failing Levene’s test of equal variance. The remaining weeks, 3rd, 4th, 10th, and 12th were analyzed using the MANOVA with four levels of within-subjects factor. The significance level used was α = 0.05, and Bonferroni’s correction was used to correct for multiple comparisons. All analysis was done using IBM SPSS Statistics 22 (IBM, New York, USA).

Comparing the conventional exercise sessions of the control group with the corresponding weekly exergame sessions of the exergames program group revealed a statistically significant effect of the type of training program, Wilks’ Λ = 0.244, F(3, 27) = 27.958, p < 0.05. Univariate ANOVAs exposed significant differences on all three outcomes, with more METs spent on control sessions (M = 2.976, SD = 0.106) than exergames (M = 2.046, SD = 0.110), F(1, 29) = 37.138, p < 0.05 (Fig. 6); but, on the other hand, MVPA, F(1, 29) = 11.044, p < 0.05, and OMNI, F(1, 29) = 7.977, p < 0.05, had higher marginal means for the exergames program (M = 36.183, SF = 0.545; M = 3.767, SD = 0.271) than control (M = 33.664, SD = 0.527; M = 2.703, SD = 0.262) (Fig. 6).

Fig. 6

Total METs spent (top left) and minutes of MVPA (top right) during conventional sessions by participants in the control program and exergame sessions by the subjects in the exergames exercise program, at weeks 3, 4, 10, and 12. Self-reported exertion, on the OMNI scale, weeks 3, 4, 10, and 12 at the end of the conventional exercise sessions by subjects in the control program and the end of the exergames sessions by the subjects in the exergames program (bottom left) and at the end of the conventional exercise sessions by participants of both the conventional and combined exercise program (bottom right)

The interaction effect between the type of training program and time, Wilks’ Λ = 0.526, F(9, 21) = 2.106, p = 0.077, was not statistically significant. The univariate ANOVAs of the outcome variables presented no significant differences for OMNI, F(2.517) = 1.253, p = 0.296, METs, F(3.000) = 1.733, p = 0.166, and MVPA, F(2.911) = 1.198, p = 0.315.

Comparing conventional exercise sessions of participants in the exergames exercise program with the equivalent (conventional) session by the control group revealed a statistically significant effect of the type of training program on the dependent variables, Wilks’ Λ = 0.723, F(3, 27) = 3.444, p < 0.05. Separate univariate ANOVAs on the outcomes did not show significant differences in both spent METs, F(1, 29) = 2.280, p = 0.142, and MVPA, F(1, 29) = 0.041, p = 0.841. However, significant differences were observed in the OMNI scale, F(1, 29) = 6.119, p < 0.05, with higher score of RPE (M = 4.117, SD = 0.302) for the exergames exercise program than control group (M = 3.078 SD = 0.292) (Fig. 6). There was not a statistically significant interaction effect between the type of training program and time, Wilks’ Λ = 0.563, F(9, 21) = 1.810, p = 0.126. Univariate analysis of the outcomes also failed to find significant difference in METs, F(2.220) = 2.905, p = 0.057, MVPA, F(2.260) = 0.805, p = 0.465, and OMNI, F(2.596) = 0.830, p = 0.467.

Our results show that exergaming sessions performed by older adults can meet the international recommendations of MVPA. Exergame sessions were able to meet the MVPA goals and surpassed the minutes of MVPA spent during conventional exercise, which is a benefit for the growth and preservation of functional aptitude. Having more time spent in MVPA and lower METs during exergaming might be interpreted as participants in the conventional workout having exercised with higher intensities but spending less time within the recommended levels when compared with exergaming. Therefore, participants during exergaming were able to exercise with lower intensity levels but, at the same time exercising within the recommended levels for longer, being more efficient in their training as more energy expenditure does not necessarily mean greater health benefits in the older population (Heyward and Gibson 2014). One possible explanation of why the participants spent more time in MVPA during exergames sessions than traditional exercise is that the games can keep players engaged with the activity, as the participants get absorbed by the individual stimulation of a game that reacts to them, which in conventional training would equate to personal training. This might have meaningful impacts on the long-term adoption of exergaming technology in the older population, producing a firm notion of a safe environment for exercising (Skjæret et al. 2016). Subjective data from the OMNI never exceeded the hard intensity (score = 8), which successfully meets the ACSM guidelines (Jones and Rose 2005).

3.2 Assessing the Effect of Physiological Adaptation

A cardio-adaptive, floor-projected version of the classic game pong, Exerpong, was used to investigate the effectiveness of using the BL engine to create physiological adaptation in real-time. A Motorola 360 smartwatch was used to stream to the BL engine heart rate (HR) data from the photoplethysmography (PPG) sensor at a frequency of 1 Hz. The goal of the physiological adaptation was to drive players to their target HR zone, for which we used 55% of the HR reserve, which is within the ACSM guidelines (40–70%) for older adults (Heyward and Gibson 2014). An experiment was conducted to explore how physiological adaptation could modulate the cardiorespiratory responses of older adults to Exerpong, in two different versions of the game, with and without physiological adaptation. This stage is called psychophysiological modeling (Muñoz et al. 2016a). Results from this stage revealed that the ball speed was a good candidate to modulate older adults’ HR in real-time. Hence, Exerpong was interfaced with the BL engine, and the ball speed was used as a variable to close the loop using HR data. That is, based on the average difference every 30 s between the current and the target HR, the ball speed was altered through a proportional controller with the following equation being Kp = 0.06:

$$ \Delta {\text{Speed}}_{{{\text{Ball}}}} = K_{p} \Delta \left( {\text{HR}_{{{\text{target}}}} - \text{HR}_{{{30}{\text{s\_average }}}} } \right) $$

Subsequently, an experiment involving 15 community-dwelling older adults (11 females, ages 66 ± 7 years, height 1.60 ± 0.08 m, weight 73.7 ± 14.8 kg) was conducted in a local senior gymnasium, comparing the cardio-adaptive version of the Exerpong (adaptive Exerpong) versus conventional cardiorespiratory training (control) sessions for seniors. Results demonstrated that participants spent around 40% more time in their individual target HR zone (state of desired cardiorespiratory performance) when using the adaptive Exerpong compared to conventional exercise (Fig. 7). In addition, participants showed lower RMSE values in the adaptive Exerpong (M = 15.2, SD = 8.3) when compared with the control condition (M = 24.3, SD = 6.4). Statistical analysis revealed that the difference was significant, F(1.0, 14.0) = 12.3, p < 0.05, r = 0.44. (Muñoz et al. 2018). Finally, we can conclude that the biocybernetic adaptation delivered a more controlled, safe, and effective cardiovascular training avoiding risky situations while maintaining good levels of enjoyment.

Fig. 7

HR responses during the control (blue) and the adaptive Exerpong (black) conditions over the 20 min of exercise. The solid line indicates the mean participant response ± standard deviation (solid area). The resting and maximum HR and target HR values used for the physiological adaptation are also indicated. Adapted from (Muñoz et al. 2018)

Fig. 8

Schematics of the experimental settings (top) and illustrative picture during the trials (bottom). a Main activities room, where Group 1 participants play in standing mode. b Physiotherapy gym where Groups 2 and 3 play in sitting mode

3.3 Group Sessions in Senior Facilities

The benefits of mixed reality games in physical, cognitive, and psychological dimensions have been demonstrated before, but not many works have addressed the social dimensions, which are quite important for the overall quality of life (QoL). To bridge this gap, we have studied how our mixed reality exergaming approach contributes to the social dimensions of QoL of institutionalized older adults. We have run a series of groups sessions with residents of a nursing home in Lisbon, Portugal, where participants played exergames with cultural motifs to trigger memories and promote relatedness and engagement.

A sample of n = 18 participants (85.28 ± 6.02 year-old, gender F = 12, M = 6) was selected among the resident of a nursing home and divided in 3 groups, suggested by the therapist of the residence and based on Barthel index score. Group 1 (G1: n = 6, F = 5, M = 1) includes participants with high levels of autonomy and functionality that perform the exergames while standing. Group 2 (G2: n = 5, F = 0, M = 5) and Group 3 (G3: n = 7, F = 7, M = 0) have lower levels of autonomy and perform the exergames while seated. For a detailed description of the selection and exclusion criteria, please refer to (Cardoso et al. 2019).

The study ran for three months with weekly sessions. It started with one assessment session to assess the baseline condition of the users. Then, there were 11 group training sessions and a final assessment session at the end of the study. Each group training session lasted for about 90–120 min. The group sessions were run in two different settings. G1 performed activities in a large room with enough space for the playing area and chairs set in a semicircle around the playing area where the participants (or other residents not participating in the study) could observe the player executing its exercise. Groups 2 and 3, which had reduced mobility and need of careful supervision, performed the exercises in a private room where only the participants and therapists were present.

During the sessions, participants played each game for three minutes. After a player finished the game, there was a period to visualize and communicate the score to stimulate the competitiveness among players. Then, the player returned to his/her seat, and the next player in the round started the exergame. Overall, G1 participants play 5 of the exergames in our library during a session. G2 and G3, due to limited mobility, played only 4 of the 5 exergames, which were adapted to the sitting position. Other exergames configurations (speed, difficulty, distracting elements) were also carefully configured and adapted considering the disabilities of the participants of each group (Cardoso et al. 2019).

For assessment, besides demographic data such as age, gender, schooling, and number of attended sessions, were used the satisfaction with social support scale (SSSS) (Wethington and Kessler 1986), which analyzes the person’s perceived satisfaction with friends, family, and social activities they do together. SSSS has 15 items evaluated in a 1–5-point Likert scale, where a higher score corresponds to better social support. The 15 items are organized into 4 sub-dimensions: “satisfaction with friends”, to measure the satisfaction with friends; “intimacy”, which measures the perception of the existence of intimate social support; “family satisfaction”, which measures satisfaction with existing family social support; and “social activities”, which measures the satisfaction with the performed social activities.

We also used the WHOQOL-BREF scale (Skevington et al. 2004), a 5 Likert points scale composed of 26 items, with four assessment domains, physical health domain (sleep, energy, mobility, medical treatment to function in daily life, level of satisfaction with their capacity for work), psychological domain (concentrate, self-esteem, body image, spirituality, frequency of positive or negative feelings), social relationships (satisfaction with personal relationships, social support systems, and sexual satisfaction), and environment domain (safety and security, home and physical environment satisfaction, finance, information, leisure activity, accessibility, and transportation satisfaction). This scale is often used under the elderly population with good accuracy in quality of life perception evaluation.

Statistical analysis was performed with SPSS v.25 software. Despite the reduced sample size (n = 18), the Shapiro–Wilk test showed that data was normally distributed. Two-tailed paired samples T-tests were executed to determine the significance of differences between the study’s beginning and end. ANOVA was used to find statistical differences between groups, and Pearson correlation test analyzed the correlation between the number of attendances to the group training sessions with any other variable. In G3, 3 participants attended less than 50% of the group sessions and were removed from the post-intervention analysis. However, they were still considered for the correlation analysis.

From the statistical analysis, we observed some interesting results such as score improvements between pre (1) and post (2) intervention regarding the perception of social relations quality domain (from WHOQOL-BREF scale) and in the satisfaction with friends domain (from SSSS) at the end of the study (Table 1).

Table 1 Paired sample tests

Regarding differences between groups, the ANOVA analysis revealed that G1 had a higher and significantly different mean score in the satisfaction with social activities domain (from SSSS) when compared to the others groups (see Table 2).

Table 2 Significant differences between groups in ANOVA test

Concerning correlations, the attendance of group sessions (number of attended sessions by the participants) showed a positive and significant moderate correlation with the social relations domain (r = 0.491, p = 0.038), indicating that attendance to the group sessions was an important element to establish social relations and improve their quality over the time.

4 Conclusions

Here, we presented the portable exergames platform for the elderly (PEPE). This is an integrated mixed reality platform including custom-designed hardware and software to address the needs of the older adult population and healthcare institutions. Our main goal was to develop a system capable of promoting physical activity following validated guidelines and a user-centered approach involving all stakeholders. The participatory approach informed us on the challenges of cognitive and motor constrains in institutionalized populations and strategies to overcome them through customizable natural user interfaces. Also, it informed us about the deployment requirements and setup limitations in shared spaces in healthcare institutions, and of esthetic preferences. From a hardware point of view, a quick and easy to set up mobile platform was developed, which addresses cognitive and motor limitations of our target population with the integration of depth sensing cameras. A short-through floor projection system combined with the on-board depth sensing makes this system calibration free and deployable in a couple of minutes, while allowing standing and seated interaction with its content. Also the ergonomics and esthetics were chosen to facilitate its acceptance. The final system, PEPE, consists of a mobile mixed reality platform that delivers floor-projected exergames controlled through markerless full-body tracking. PEPE includes five highly customizable mixed reality games developed in Unity that address the main dimensions of fitness training according to the recommendations of the ACSM. Additionally, PEPE utilizes a closed-loop construct that exploits rule-based adaptations through physiological signals to intelligently modify game requirements in real-time. This is achieved through the biocybernetic loop engine, which enables the easy creation of adaptation rules through a visual language that communicates in real-time with both physiological measurement devices and the exergames, using standard and open protocols.

The system has been deployed in multiple sites aiming to address and validate its multiple facets in the context of different studies and has been shown to be effective in stimulating elderly to practice physical exercise with the addition of fun and social interaction. First as a feasible social platform to engage elderly in physical activity in elderly-care institutions. Then, addressed its efficacy in a randomized controlled trial compared to training sessions delivered by certified personal trainers. Finally, its adaptation capabilities compared to a non-adaptive approach. When applied regularly in group sessions at nursing homes, PEPE improved the overall perception of the quality of life and social relations in institutionalized older adults. We also show that the developed set of custom-made exergames can be successfully used by trainers to set up personalized training sessions and can be used in combination with regular exercise for sustained long-term training, exposing differences between traditional physical training and exergaming in terms of efficiency, elicited physical activity, and perceived effort. Data revealed that while participants spent more time in moderate-to-vigorous physical activity during exergaming, they also spent less energy, thus working out at lower intensities but for a more sustained amount of time. We also showed that physiologically augmented exergames with PEPE lead players to exert around 40% more time in the recommended effort levels than conventional training, avoiding over-exercising, and maintaining good enjoyment levels. Hence, the use of the combination of the multiple facets of PEPE—gaming, physiological adaptation, and its deployment in elderly institutions—has a high potential for the higher efficacy of computer mediated training and additional extrinsic motivational factors of gaming. However, it is very difficult to disentangle the actual contribution of each of those components and the particular software and hardware design decisions of PEPE in the presented studies. Thus, it is possible that other means of VR other than floor projection or set of games could deliver different results. Factors such as, for instance, the size of the area of the play as well and the details on the full-body interaction modalities used during gaming have a decisive effect on exertion.

There are, however, some limitations in the work here presented. First, the PEPE design has been developed through a participatory design with day care institutions and end-users that may not necessarily capture the specific needs of all potential end-user populations as well as healthcare institutions. The technology available at this time limited the size of the floor projection, being it now possible to have short-throw projectors with larger aperture and more contrast. In addition, none of the presented studies were performed exploiting all the features of PEPE. That is, the 3-month longitudinal study did not include the biocybernetic loop construct. Hence, the efficacy of a PEPE exploiting all its features is likely to differ from the one reported here. Similarly, the population tested on our randomized controlled trial and biocybernetic loop construct is a healthy sample and does not represent the average institutionalized end-user. Hence, further studies are needed to quantify PEPE’s efficacy on different patient populations.

Overall, the work presented here shows the potential of using entertainment technologies to develop new training paradigms that are easy to use, engaging, effective, and well-received by end-users and institutions. The results show that exergames can be used by older adults to perform exercise sessions that meet the international recommendations of MVPA and that are a feasible complement to current practices in senior gymnasiums and elderly-care institutions.