Keywords

1 Introduction

As the world’s population continues to age, nearly every country has experienced a steady growth in the proportion of its population aged 65 or higher [1]. This is particularly the case in countries such as Japan, wherein the elderly are projected to account for over a third of the population [2]. Consequently, global demand for physical therapy and healthcare to combat the physical disabilities caused by musculoskeletal degeneration and other detrimental health effects of aging has seen a sharp increase, with demand rapidly outweighing availability of therapists and physicians in many cases [3].

A promising solution in recent research has been the usage of exercise-based serious games, or exergames, as a tool for self-driven exercise to benefit elderly physical health and healthy lifespans. A wide variety of exergames have been developed for this purpose, with many recent approaches including adaptation strategies to match the difficulty of gameplay with performance and physical limitations of the player. These exergames are designed to provide an engaging and meaningful method to motivate regular exercise, and utilize physical motions directly related to those prescribed within rehabilitation programs and therapy in order to relate gameplay to physical health. However, the designs of many such exergame systems share a glaring issue: while it is often shown in resultant data that users of these exergames improve in physical function, due to the lack of a direct link or mapping between the properties and outcomes of the exercise task and the corresponding properties and outcomes of game objectives, it can be very challenging to conclude that the evidence for these improvements in health outcomes are based within gameplay, and therefore the effectiveness of these solutions in practice cannot be effectively determined.

To address this issue, it is proposed that the design of these exergames can utilize a mapping strategy wherein the physical and temporal properties of the motor task, as well as the individual goals, successes and errors within these domains, are linked directly to corresponding gameplay objectives and outcomes, in such a way that evidence for successful and erroneous performance within gameplay serves also as evidence for performance in a motion task. Furthermore, adaptation of the gameplay objectives should utilize a learning backend that learns to link an individual’s performance at the exercise task to officially recognized clinical metrics for health and physical function.

These are the key principles in the design of a novel framework for exergame development, the Stealth-Adaptive Exergame Design (SAED) Framework, presented in this work. Related work on exergames for elderly and rehabilitative users and the principles for adaptation and exergame design that serve as the basis for the proposed framework are presented in Sect. 2. Section 3 provides an overview of the SAED Framework and details of the evidence-centered mapping and adaptation strategies described above. In Sect. 4, two cases are presented wherein the SAED framework is utilized in the design of exergames to support rehabilitation, along with details of their respective strategies for individually focused stealth adaptation. Conclusions and directions for future work including implementations of the framework across a variety of domains of exergaming are provided in Sect. 5.

2 Related Work

2.1 Exergames for Elderly and Rehabilitation

The potential for exergames, or exercise-based videogames, to promote the healthy lifespan, rehabilitative recovery, mobility, and physical fitness among the elderly is well-documented in research. Zheng et al. demonstrated the ability of these games to improve multiple health outcomes including strength, balance and other physical characteristics among the elderly population [4]. Liao et al. have further indicated that exergames can assist elderly subjects in recovering from effects of frailty [5]. The primary motivation for utilizing exergames over conventional exercise is also well-noted in research, and is verified by Huang et al. [6]: exergames can create a positive effect on an individual’s enthusiasm and perception toward exercise, and may thus assist in improving compliance with regular physical activity requirements over time.

Based on these motivations, and the need for individually adaptive rehabilitative exercise, exergames have been developed and evaluated extensively within this population and have seen a variety of approaches toward adaptation and adjustment to match skill level. As a notable recent example, Garcia et al. developed an asynchronous solitaire exergame wherein gameplay elements requiring cognition were separated from those requiring physical exercise in order to reduce cognitive load in consideration of the preferences of the population [7]. Earlier approaches [8] recognized the need for these exergames to respect the physical and cognitive capabilities of their target population, giving rise to adaptive systems. In some cases, adaptation was facilitated through intervention by a physiotherapist [9]; however, automating the adaptation process becomes increasingly desirable in modern work as the demand for physical therapist time and resources begins to exceed availability.

Adaptive exergames often adopt a default approach to design wherein a predefined game concept or game task is fitted to a motion task and then distributed to a series of users, treating “the elderly” as a group rather than considering interpersonal and intrapersonal variation in design. The Person-Centered Multimedia Computing [10] paradigm argues that even among populations who share common attributes, such as the geriatric population, individuals can be vastly varied from the perspective of technology and human-computer interfaces, especially in the case of physical motion capabilities and limitations. In this regard, adaptation mechanisms such as that of Paliyawan et al. [11] which utilize AI to learn about the player, gradually improving the precision of their adaptation to match their knowledge of that player’s attributes, are preferable to more static implementations. The recently-proposed fuzzy logic model by Zhang et al. [12] for assessing adaptation of various exergaming environments using individual characteristics also demonstrates the effectiveness of AI-based dynamic adaptation.

Individual variation also leads to differences in preferred game types, with individuals often preferring games with subject matter they are more familiar with, particularly when these players are elderly users [13]. The choice of subject and mechanics within exergame design should account for these factors in addition to considering the motions themselves. Finding the right game abstraction for an exercise is no trivial task; the chosen game scenario should be the most natural abstraction possible for the motion task. This concept was suggested early in the development of exergame research by Sinclair et al. [14] in their separation of effectiveness and attractiveness of exergames, but was not explored in detail. This places several fundamental constraints on the exergame implementation, which many existing exergame approaches often fail to adequately account for.

Furthermore, the requirements of the trainers, therapists and standard assessments need to be considered and integrated into the AI’s assessment of performance in these systems, so that they can be more easily integrated into existing training and exercise programs. Earlier studies relating exercise to physical health in the elderly utilized evidence in physical performance characteristics to support claims about health outcomes [15]. Modern exergame studies utilize validated, trainer-approved evidence of health outcomes to assess the effectiveness of their implementations, but these are often done outside of gameplay, since these instruments of assessment are not mapped into the game’s design [16].

2.2 Evidence-Centered Adaptation

When game mechanics are designed merely to accommodate physical activity, rather than to embody it precisely, it is difficult to validate the effectiveness of exergame AI at evaluating and adapting to individual skill level [17, 18]. For example, often it is the case that performance in one of the three primary categories of motion assessment (posture, progression, or pacing) [19] is significantly different than the others, and only the elements of gameplay mapped to that category require adjustment. Recently derived requirements from the elderly population for the design of exergames reflect these concerns, prompting the need for a more flexible adaptation strategy [20].

Studies of adaptation in exergames for the elderly have favored more individually driven and motion-centric design of gameplay. Velazquez et al. [21], for instance, used an action research study to derive several recommendations for the implementation of adaptation within exergames supporting the elderly. Among these is the need to accurately and seamlessly classify motion capabilities of the player concurrently within gameplay. For this classification to relate directly to game performance, it follows that gameplay outcomes should serve as verifiable outcomes for performance results at the motion task. Without this relationship, the claim that adaptation of exergame elements and game mechanics leads to suitable motor learning, and thus improved health outcomes, cannot be easily verified.

Hence, the design of exergames should center itself around the characteristics which serve as evidence for exercise performance, and the physical and temporal characteristics of these mechanics can be mapped directly to the corresponding characteristics of physical performance such that in order to improve their proficiency, users can rely directly on game mechanics, rather than focusing on their own body movements, to guide them [22, 23]. This principle of the application of Evidence-Centered Design [24] toward “stealth” assessment forms the foundation for the “stealth assessment” concept in educational serious games proposed by Shute et al. [25] and is applied in this framework toward adaptive exergames for the elderly.

3 The Stealth-Adaptive Exergame Design (SAED) Framework

3.1 Overview

The Stealth-Adaptive Exergame Design (SAED) Framework, as shown in Fig. 1, is a novel framework for exergame selection and AI design which utilizes Evidence-Centered Design and Stealth Assessment principles [26] to facilitate individually focused machine learning and assessment of motion task performance. In this framework, a specific individual, trainer requirements (when applicable), and motion task properties are utilized as filters to select the best-matching game concept on a case-by-case basis.

Fig. 1.
figure 1

Overview of the stealth-adaptive exergame design framework.

Full size image

Once the exergame implementation is selected, an evidence mapping approach similar to [26] is adopted to map the spatial and temporal attributes of a motor task repetition to a repeating gameplay element, including a relationship between positive and negative task performance and in-game outcomes. Finally, an instrument for performance assessment is chosen from expert standards and a model for AI-based learning is chosen based on the relationships between the chosen assessment instrument and the exercise task properties. This model is used to adapt the game to the player’s performance by directly modifying game objectives at a chosen interval and tolerance range for error, resulting in seamless stealth adaptation.

3.2 Exergame Selection

The first task posed by the SAED framework is the selection of an appropriate exergame by beginning with the motion task. This motion task is provided by a therapist, trainer or physician, and is often tuned to the needs, capabilities and limitations of an individual; however, some common properties exist among rehabilitative or health-improving exercises. Generally, these exercises can be decomposed into three characteristics [19]: posture, which represents the requirements of the subject’s body configuration while performing the exercise, pacing, which represents the rhythm and rate of motion requirements, and progression, which represents the degree or precision of motion required to successfully complete the exercise. Furthermore, the exercises themselves are typically repetitive and rhythmic in nature, with each attempt comprising one repetition of the task and multiple repetitions forming a set.

As discussed in Sect. 2.1, not all game concepts are effective as choices of abstraction for these exercises. When considering the above requirement, the chosen exergame implementation must utilize an interactive game element which is comprised of a uniformly repeating task (for example, a racing track being comprised of a series of turns or curves), with continuously adjustable parameters, and clear, immediate evidence for performance [27]. The game task must also match the motion task in complexity by sharing similar spatial characteristics (posture, progression) and temporal characteristics (pacing) for adaptation to be effective.

In alignment with the requirements of person-centered design, and in respect of the interpersonal and intrapersonal variation within the population, the chosen game objective must match any additional requirements or constraints placed by the trainer or therapist involved in assigning the motion task (such as the use or lack of exercise equipment, and duration of a single session), should respect preferences of the individual toward certain game contexts or subject matter, and should choose a game task which serves as the most natural abstraction for the motion task under assignment. It would be difficult, for example, to associate the game task of flipping a coin to a knee flexion/extension task as input, as coins are typically flipped using the hand.

3.3 Evidence Mapping

Once an exergame implementation has been selected, a single repetition of the game task corresponds to a single attempt or repetition of the exercise task. At this point, it is necessary to determine the factors which serve as evidence for successful performance of this game task and relate one such factor to each of the three requirement domains for performance of the motor task. This task is greatly alleviated in difficulty when the game context is a natural abstraction of the motion task as previously stated. For posture, the game’s design should consider what consequences could arise from poor or proper posture during completion of the game task. In the racing track example mentioned earlier, poor posture when driving a vehicle can result in a loss of control at critical moments, as one illustrative example.

Once factors for evidence of game performance are mapped with related factors of exercise performance, a dependency relationship is created wherein the degree of success or failure in each of these mechanics of gameplay is also directly related to the degree of success or failure in performance of a particular attribute. This mapping allows the player to focus entirely on learning how to play the game as effectively as possible, since doing so requires improvement in performance at the motor task. The distinction between these three characteristics of exercise task performance also allows performance in each attribute to be distinguished from the others during gameplay, allowing an individual to learn which areas require improvement more effectively. This matches the type of guidance provided in rehabilitation or exercise with trainers, who utilize their observation and expert knowledge of the individual to give targeted advice for improvement. If the individual’s posture is correct, but the pacing is slow, a trainer can point out this distinction so that the individual knows to maintain posture while modifying pacing. Evidence mapping allows an exergame to intuitively embed this guidance into gameplay outcomes, allowing for more effective self-assessment with an external focus [22].

3.4 Stealth Adaptation Using AI

Finally, requirements for the degree of error allowed in each parameter or motion (also referred to as “tolerance range of error”), frequency of adaptation, and other factors such as physiological or affective data, are combined with the real-time performance measures of gameplay as features into the AI component, which can then learn to accurately relate them to motion performance for a specific individual over time using either standard performance measures or trainer-specific measures. This results in real-time adaptation that is argued to more closely reflect the individually variant nature of individuals within the elderly population, as well as the exercise programs and rehabilitative programs most relevant to each.

To achieve this type of adaptation, it is first necessary to select an appropriate metric for performance assessment. Generally, experts such as therapists, trainers, physicians or medical associations provide the instruments of assessment that relate performance at a particular task to predicted health outcomes or functional ability. A range of standard metrics exist for motor function assessment in rehabilitation, for example, such as the Wolf Motor Function Test [28], Barthel Index [29], Fugl-Meyer Assessment [30], and others. However, it is often the case that customized or trainer-specific instruments of assessment are adopted for a particular exercise program. It may also be the case that the relationship between the task performance characteristics and health outcome assessments is not entirely known, in which case it is also possible to learn and characterize automatically using AI. The selected performance assessment metric is used as a calibration or pre-assessment tool to characterize the “actual performance” of the subject in practice (this, of course, can also be provided directly by a trainer or therapist who performs the assessment with the individual externally). This is then used as training data for the AI model to “learn” about the subject by identifying the errors in its prediction of the subject’s performance.

Once the metric of assessment is in place, the appropriate learning model can then be selected. The model should account not only for the assessment method and the relationship between that method and the performance parameters of the task, but also for the range of error tolerance for performance of the task, as well as the frequency of adaptation or learning and any other factors present in the implementation, such as the processing power of the exergame platform. When implemented, the model learns about the subject by using the assessment instrument and motion characteristics to predict performance, determining where performance is above or below expectation, and then referring to the mapping to determine what gameplay parameters need to be adjusted to reduce or increase difficulty, and to what degree they should be adjusted based on the severity of error.

4 SAED Case Examples

To illustrate the utilization of the SAED framework, two cases of its implementation are presented in this work. Following the procedure outlined in the framework, each implementation begins by identifying a motion task for improvement of health outcomes, followed by the selection of an exergame implementation accounting for both the task and other influencing factors of the individual, trainer or population. The decomposition of each task into their postural, progression and pacing components is performed, and then evidence mapping with corresponding game mechanics is determined. Finally, instruments for assessment in each case are used to select a learning model for real-time stealth adaptation, and the characteristics of this adaptation are discussed.

4.1 The Autonomous Training Assistant: Fruit Slicing Exergame

The first case of SAED implementation is a fruit slicing exergame from [31], shown in Fig. 2. In this case, the role of the exergame was to autonomously monitor, assess and improve function of the paretic arm in a hemiparetic individual whose trainer’s rehabilitative exercise program involved the use of martial arts stick techniques. The assigned motion task by the trainer was an arc swing motion of the stick. The exergame environment for this program, entitled the Autonomous Training Assistant, included a customized training stick device (the Intelligent Stick) equipped with motion sensing and haptic feedback capabilities, a Kinect depth camera for subject body and joint tracking and the exergame interface played on a television screen at the subject’s home (without presence of the trainer). The subject was assigned to swing this device in a horizontal arc at a specific pace while holding it with both hands, in such a way that the functioning arm guided the paretic arm during motion.

Fig. 2.
figure 2

Screenshot of the Fruit Slicing exergame from the Autonomous Training Assistant.

Full size image

The design process for the exergame is detailed in Fig. 3. The trainer’s recommendation that the subject swing at targets with a sword or stick-like object in gameplay, along with the individual’s interest in fruit slicing games such as the commercially-available Fruit Ninja by Halfbrick Studios [32] resulted in the selection of a game task wherein the subject must slice fruit objects along a horizontal path as they fall from the sky. These fruit objects fall together repeatedly in groups; a single repetition of the swing motion thus results in a single swing of the sword at a group of fruit, the objective being to slice as many of the fruit in a single swing as possible.

Fig. 3.
figure 3

SAED framework applied in the design of the Fruit Slicing exergame.

Full size image

Postural characteristics of the motion required that both arms should remain on the stick when swinging. This was directly mapped to the requirement that the subject hold the virtual sword with both hands to swing it properly. If the paretic arm loses contact with the stick, the control of the sword is lost and it cannot be swung. The difficulty of this parameter related to the percentage of time during a swing that two-handed contact was maintained.

Pacing requirements included that the subject must match a specific pace of motion determined by the trainer and adjusted on a regular basis. Since the objective is to slice all of the falling fruit objects, their speed of motion directly corresponds to this pacing requirement. The more rapidly the fruit fall, the less tolerance for error when attempting to slice them, and the faster the motion required to slice all of the fruit in one group.

To maintain optimum progression, the trainer recommended that the subject contact critical points along an arc trajectory when swinging the stick. In the exergame implementation, the fruit objects were selected as the mechanic which represents the critical point concept. In other words, the timing of the fruit’s descent is configured in such a way that the subject must contact each object at various points along a horizontal arc trajectory to slice them. Furthermore, the size of the fruit or type of fruit represent the difficulty or tolerance for error in progression, with smaller fruit requiring more precise trajectories.

Having performed the evidence mapping required to directly relate game performance to task performance, the instrument for assessment was then chosen. In this case, the trainer did not utilize a single, combined “mastery” score or assessment but instead assessed each category independently by observation during in-person training sessions with the subject. Thus, the trainer recorded a “correct” template for execution of the motion, and the critical points of this template along with its pacing and postural details were used as the basis for evaluating the subject’s task performance. As the three categories needed to be assessed independently, and the relationship between exercise parameters and successful health outcomes was already known and provided by the trainer, a Bayes Net implementation, as illustrated in Fig. 4, was used to learn about the individual and adapt the game difficulty parameters. When the error in a particular category was above a certain trainer-provided-and-adjusted threshold, the gameplay mechanics mapped to that category were automatically adjusted in real-time. The system learned by being supplied information about actual performance in relation to its predictions by regularly provided training session data from the subject and trainer.

Fig. 4.
figure 4

Bayes Net diagram guiding real-time stealth adaptation of the Fruit Slicing exergame.

Full size image

The system begins by assuming equal probability that the subject is below, above, and at expectation for performance in each category. For example, it assumes in the case of posture that the subject is equally likely to hold the stick device with both hands for 90%, greater than 90%, and less than 90% of the swing’s duration, until data on each swing attempt is used to update the model. As these probabilistic predictions are maintained independently for the three categories, the system makes a decision between each swing attempt (or slice) to adjust difficulty separately for the game mechanics relating to each category. If postural performance is above expectation but pacing and progression are less than predicted, the subsequent slice attempt will require a longer period of time holding the stick with both hands, while using larger fruit that fall more slowly to allow for greater deviations in the motion’s pace and trajectory. Facial expression data was used to estimate the subject’s emotional response to gameplay, for the system to self-evaluate based on how long the subject maintained the state of optimal engagement, or flow state [31].

4.2 Ski Exergame

The second case of SAED implementation is a ski squat exergame [33] as shown in Fig. 5. This implementation was designed for elderly users to maintain healthy locomotion, mobility, and consequently, improved healthy lifespan by regularly performing the two-leg standing squat in an interactive virtual environment. In this case, the Japanese Orthopaedic Association (JOA) was selected as the expert source for assessment of health outcomes, as they have published extensive work on the characteristics and prevention of Locomotive Syndrome [34]. The exergame environment consists of a RealSense depth camera for tracking motion, and a virtual reality interface for presentation of the ski exergame. In future work, this implementation will be converted to a non-VR format in consideration of the safety of elderly subjects when playing.

Fig. 5.
figure 5

Screenshot of the Ski Squat exergame.

Full size image

The SAED design process for the ski squat exergame is shown in Fig. 6. The squat motion was selected due to its popularity in Japan as a daily exercise to improve lower extremity strength and balance and to prevent locomotive degeneration. Furthermore, popularity of skiing in Japan and the natural fit of the Ski squat motion, a standard exercise to improve skiing performance, provided the means to select a skiing game which utilizes squats as an obstacle avoidance strategy. In this case, the player is asked to ski through a straight course wherein no turns are required (or they are performed automatically without player intervention). A series of gates appear which can only be cleared by squatting down below a certain height, where each gate along the course requires a single squat motion. The requirements for the postural, progression and pacing aspects of the motion were derived in consultation with an expert health/sports science researcher.

Fig. 6.
figure 6

SAED framework applied in the design of the Ski Squat exergame.

Full size image

Postural requirements included that the subject maintain stability during the squat (that is, the knees remain firm with as little shakiness as possible) and that the subject’s center of mass (CoM) moves in a consistent and smooth motion. Within the exergame implementation, these are naturally fit to the requirement that the subject maintain an average horizontal knee deviation and CoM deviation less than a certain (adjustable) threshold to maintain control during the ski task. The spacing between gates represents this requirement within gameplay.

Pacing of the squat task was simply represented as the number of squats expected over a fixed time interval (such as 30 s or one minute). This time interval was used to determine the length of the ski course, and the number of gates generated in the course represented the number of expected squats to perform. The higher the number of gates over the same length of course, and the shorter the spacing between these gates, the more rapidly the subject is expected to squat to clear all of them.

Progression in this case related to the degree of bending performed during the squat task. A more difficult squat requires the subject to move lower (up to a limit for safe squat completion, beyond which the exercise is considered harmful). This relates directly to the height of each gate obstacle, the lower the gate, the lower the subject is required to squat to clear it when passing through. While an error in progression would normally be considered as a collision with a gate, this collision is not represented realistically in gameplay, as it would be disruptive to the steady pacing requirement of the game. Instead, the subject simply receives points for gates that are successfully cleared, and no points when colliding with a gate obstacle, which the subject simply passes through.

The JOA has derived a standard assessment strategy for determining locomotive risk in elderly subjects, entitled the short test battery for locomotive syndrome (STBLS) [35]. It consists of three assessments: one in which the subject stands up from a seated position at various heights and receives a score based on the lowest height from which he or she could stand without losing balance (sit-stand test), another in which the subject takes two strides as far as possible without falling, and measures the length of the two strides normalized by his or her height (two-stride test), and a third assessment in which the subject answers 25 questions about pain or difficulty with mobility over a time period (25-question assessment). These three tests result in three scores for a subject, which are then combined into a single locomotive risk score (0, 1 or 2) using clinical decision boundaries derived by the JOA and presented in [35]. However, the exact relationship between the above attributes of squat performance and performance in the STBLS is unknown, and needs to be characterized over time by the learning model.

As such, a neural network was deemed appropriate in this case, as it could learn the association between squat input parameters and STBLS scores through training and backpropagation. The structure of this network is shown in Fig. 7. It is initially configured through a set of training data to perform moderately accurate estimations for individuals within the target population (pre-training). The structure of the network is then tuned in real-time according to the error generated between the network’s output and the actual STBLS performance of the subject in a pre-assessment. The difficulty of the game is adjusted on a course-by-course basis using the risk value generated by this network as a combination of its outputs.

Fig. 7.
figure 7

Diagram of feedforward Neural Network for stealth adaptation in the Ski Squat exergame.

Full size image

5 Conclusions and Future Work

Initial prototypes of the above case examples in [18] and [33] indicated high accuracy of adaptive capabilities as expressed by real exercise performance and subjective feedback. Quite importantly, these results serve as validation of the effectiveness of adaptation in each case as it relates to physical performance. The Autonomous Training Assistant was adopted for long-term use by the subject and trainer as well, further validating the approach. This serves as a proof of concept that the SAED framework can facilitative effective and verifiable adaptive exergame design centered in evidence and standards of performance and health outcome assessment. Future work should further validate the generalizability of the SAED approach by include game outcome evaluations across a variety of cases and validation through comparison with health outcome measures. One particularly useful effort would be the creation of an exergame concept matching table which provides the most naturally fitting exergame implementation strategies and context across the gamut of rehabilitative motions, individual capabilities and interests.