Introduction

Knee disorders are significant public health problems, and knee rehabilitation training is an important treatment modality for knee disorders [1]. The knee has a complex structure and is highly susceptible to injury in daily life. There are many types of disorders, such as knee osteoarthritis (KOA), meniscal tears (MT), and anterior cruciate ligament (ACL) injuries [2,3,4]. The frequency of knee diseases increases simultaneously with age and joint degeneration [5]. According to epidemiological statistics, the prevalence of KOA is as high as 42.8% among people over the age of 60 in China, and many older people suffer from varying degrees of knee disease [6]. Knee disorder is a significant cause of disability and reduced quality of life [7]. The rehabilitation exercise is the most effective form of prevention and treatment of knee disorders according to the effectiveness of the intervention or the duration and energy consumption [8]. OA treatment guidelines consistently recommend rehabilitation exercise as a core component of managing knee disorders, and there is solid evidence of its effectiveness [9,10,11]. The effect of pain and functional improvement after an exercise-based KOA program is comparable to the observed effect of medication for OA [12]. Disease prevention and postoperative recovery require proper rehabilitation exercise to restore knee function. Therefore, an adequate and effective knee rehabilitation training can effectively promote knee rehabilitation.

The exergame is a promising rehabilitation training solution [13]. Exergame integrates gaming enjoyment, physical movement, and training functions to improve rehabilitation training interventions' safety, adherence, and affordability [14]. The effectiveness of Kinect-based exergame has been widely studied, and its application in rehabilitation training has been well validated [15, 16]. Su C H designed a Kinect-based exergame for total knee replacement rehabilitation and conducted clinical trials to fully demonstrate the effectiveness of gesture interaction in facilitating postoperative knee training. However, such gesture interactions require specialized depth cameras, which sometimes have problems with the sensor availability and correspondingly demanding computer configurations [17]. Besides, image-based human pose estimation has been rapidly developed in recent years. A variety of technical solutions have gradually emerged, such as OpenPose [18], AlphaPose [19], and Mediapipe [20], for the human body's joints., They are also improving the accuracy of the position measurement [21]. Blazepose technology is developed based on the machine learning toolkit Mediapipe, and it is gradually being used in pose interaction because of its fast and accurate 3D human joint position pose estimation [22]. The feasibility of applying the image-based pose estimation algorithm to knee rehabilitation training guidance needs to be further explored.

User experience analysis refers to assessing the experience of the interaction between the user and the rehabilitation training system and is used to evaluate and improve the system's suitability [23]. Special user experience characteristics exist for the elderly and people with joint diseases. The elderly often has a cognitive impairment and slow reaction time, which require the exergame system to be designed with a user-friendly interface and multiple forms of assistance in mind. Patients with joint diseases tend to fear training because of pain, leading the suitability and safety should be considered in the user experience.

In summary, we designed a gesture-based interaction exergame to aid the rehabilitation of patients with knee disorders. The primary purpose of this paper is to present the system's design, evaluate the system's interactivity and analyze the effect of user experience. The structure of this paper is organized as follows. The hardware setup and software design are described in “Exergame system design and setup” section. “Materials and methods” section designs the clinical experiment to analyze the interactivity and user experience of the system. Sections “Results” and “Discussion” sections show the results of the clinical trials and provide a systematic discussion. “Conclusion” section concludes our system.

Exergame system design and setup

System hardware requirements and setup

The hardware for the system is straightforward, requiring only a computer with a camera or an intelligent mobile terminal. The computer or mobile terminal installs the system software. The system invokes the camera to acquire the full-body image of the user and calculates the user's primary joint position information in the image in real-time by the 3D pose estimation algorithm. Then the motion recognition algorithm is designed for human pose interaction according to the functional needs of the system. In addition, for a better experience, the user could use an additional large screen and sound system to improve the display and audio of the exergame system. Figure 1 shows the hardware structure of the exergame for knee rehabilitation.

Fig. 1
figure 1

The exergame system's hardware architecture

Full size image

The exergame proposed in this study mainly uses BlazePose real-time body pose tracking technology to realize gesture interaction of humans [24]. BlazePose is a 3D body pose estimation tracking system developed and open-sourced by Google Research. BlazePose can infer 33 key bone location points of the human body from a single frame, including bilateral shoulder joints, hip knee, and ankle joints [25]. Figure 2 shows the joint structure obtained by BlazePose. If the user's body is within the camera's field of view, the system can quickly estimate the user's body joint points in real-time and map them into a simple human skeleton map. BlazePose simplifies the interaction and allows for more natural mapping, making the game more accessible to older adults who lack gaming experience [26]. Compared to high-precision inertial and optical motion capture systems and Microsoft's Kinect system, BlazePose requires only a common camera and does not require wearing any additional sensors, is easier to use, and has better affordability. Moreover, it makes future mobile applications possible and promotes use easier.

Fig. 2
figure 2

33-joint human skeleton model of 3D Human Pose Estimation based on Images

Full size image

The system functional design

The system includes three functional components: knee exercise plan formulation, exergame, and exercise feedback. Firstly, the knee patient makes a functional evaluation, and the physicians give an exercise plan based on the knee functional evaluation information, converted into exercise plan parameters. Then the system generates an exergame for each training task according to the rehabilitation training parameters and guides the patient to exercise. Finally, the system records the patient's exercise data for the patient and the physicians to view. Figure 3 shows the flowchart of the user using the exergame.

Fig. 3
figure 3

The flowchart of the user using the exergame

Full size image

Exercise plan formulation

The exercise plan formulation module identifies the exercise tasks for each patient with knee disorders. Different rehabilitation training tasks exist for different knee disease types and disease levels; therefore, knee exercise plans need to be evaluated and formulated by physicians based on the patient's knee disease status and functional status. This module provides basic knee information collection and upload, exercise plan viewing, and confirmation. It also offers a basic exercise plan for patients with common knee diseases like KOA. The basic knee information collection includes diagnosis of knee disease, functional tests, and common knee function assessment. The patient fills in the relevant information and sends it to the designated physician, who makes an exercise plan based on it and feeds it back to the system for the user to view and confirm. The system can automatically recommend an exercise plan for early KOA patients based on the patient's basic information, combined with exercise guidelines.

The exercise plan includes parameters such as the type of exercises, the intensity of exercises, the duration of exercises, and the frequency of exercises given by physicians [27]. The exercise types mainly include common motions such as walking, squatting, straight leg raises, hip abduction, up and down the step, and knee extension in a sitting position [28]. These motions cover a variety of training contents, such as joint muscle strength training, endurance training, and balance training, which meet the knee exercise plan's needs [29].

Exergame

Exergame is a game that arranges training tasks into game levels and their parameters according to the exercise plan, guiding patients to execute rehabilitation training precisely and ensuring the scientific and safety of rehabilitation training. The game levels are designed for different types of motions. For example, the game "Squatting to avoid obstacles" is designed for squatting exercises to strengthen the quadriceps. The game "Attack the Monster" is designed for straight leg raise training to train patients' muscle strength and balance ability. The game "Grabbing and catching food" is designed for sitting muscle strength training. Table 1 lists some of the designs of the exergame.

Table 1 The design of the exergame
Full size table

The user is incarnated as a game character and is mapped to the game character by the user pose to control the game character to execute the game. Using the camera to capture the user's image in real-time, BlazePose is used to calculate the joint position of the user's body. Then the sagittal and coronal angles of the knee and hip are calculated. Finally, KNN and other machine learning algorithms are used to continue the action recognition, to achieve action guidance and game control. During the game, the user is in a suitable position in front of the camera to ensure that the patient's whole body is in the camera's field of view. The user wears tight clothes when exercising as much as possible to improve recognition efficiency.

The exergame is designed using user-centered design and goes through multiple stages of design and experimentation. We finally chose the most suitable game design through consultations and experiments with physicians and patients. User characteristics analysis is the basis of user experience analysis; this exergame mainly includes five aspects: (1) UI design. The interface layout is reasonable, with fonts, buttons, etc., as large as possible to facilitate understanding by older users. (2) Prompt design. The interactive steps in the game are designed with simultaneous voice prompts to prevent users from not being able to understand the UI prompts. On the other hand, the remainder of exercise tasks can be done through SMS, alarm clock, and other forms to remind patients to complete the exercise tasks in time. (3) Multimedia exercise action demonstration and teaching. Use pictures, real-life videos or animations, and other multimedia forms to demonstrate the exercise action and combine the user's motion capture data to guide the user to complete the exercise. (4) Inclusive design of action execution. Older patients usually have reduced responsiveness, and there is a delay in executing the action. Therefore, the action recognition and interaction design give a certain delay time. (5) Incentive design. The reward and punishment design of the game can be combined with the actual needs of the elderly in addition to the traditional scoring. For example, giving physical rewards can increase the participation of elderly users. The effects of the exergame are shown in Figure 4.

Fig. 4
figure 4

The effects of the exergame

Full size image

Exercise feedback

Exercise feedback refers to the user's exercise data recorded by the system, which the user and the physicians can view. The exercise data include training execution data such as the completion rate of game tasks and joint angle change curves. The subjective perception, including pain and fatigue levels, is also recorded. The patient can view these exercise data for self-check. These exercise data can be fed back to the physicians for evaluation.

The exercise feedback is supported by a web-based system, which enables the feedback of patient exercise data and viewing and guidance by the physician [30]. The exercise data from the patient's exergame system can be transmitted to the physician through the Internet, and the physician can view the patient's exercise data. The physician can keep abreast of the amount and quality of completion of the patient's exercise plan and evaluate and guide the patient's rehabilitation training. The working contents are as follows.

First, based on the exercise data during the patient's exercise, understand whether the patient is following the exercise plan. For patients with poor compliance, a team of nurses will contact and remind them.

Second, based on the quality of the patient's completed movements, assess whether the exercise program is reasonable and reduce the difficulty of movements that are difficult for the patient to complete.

Third, as rehabilitation continues and the patient's knee function gradually improves, the exercise prescription needs to be adjusted dynamically accordingly, such as increasing joint mobility, increasing the number of sets and times of training, increasing weight bearing, etc.

This method allows for evaluation of the knee exercise process, remote supervision, and guidance by the physician to ensure the safety and effectiveness of the rehabilitation training.

Materials and methods

Design, recruitment process, and participants

The feasibility and user experience are basic for evaluating the exergame system. This paper recruits KOA elderly patients to experiment, assess the system's feasibility, and analyze the user experience.

We recruited older KOA patients in the community at two community health service centers in Hefei, Anhui Province, China, from August 2021 to November 2021 for the trial:

  1. (1)

    We screened KOA patients through the community health service system.

  2. (2)

    We invited KOA patients by phone or verbal invitation and introduced our trial. Interested individuals are asked to contact us for more information.

  3. (3)

    Each eligible participant is pre-screened and confirmed to participate in our trial.

The participants are 50 years or older, and community residents, who are early-stage KOA patients and could perform the motions in the exercise game, are included in the trial.

Exclusion criteria: (1) cardiovascular disease unsuitable for exercise; (2) systemic rheumatic disease of the knee; (3) meniscal or knee ligament tears; (4) planned knee surgery or other lower extremity surgery or post-operative; (5) presence of hearing, visual or cognitive impairment.

In the experiment, we invited three knee specialists to form a group of physicians to evaluate each participant's knee function and formulate the exercise plan. These physicians all had expertise and experience in knee rehabilitation.

This study followed the accepted ethical, scientific, and Declaration of Helsinki principles. The Hefei Institute of Materials Science, Chinese Academy of Sciences Ethics Committee approved this study (NO.SWYX-Y-2021-13). All study participants are informed about the experiment and gave their agreement.

Experimental setup and procedure

For convenience, all participants experimented with their nearest communities:

  1. (1)

    The physician team formulated the exercise program for each participant based on patient function and entered it into the exergame system. Participants logged into the system for individualized rehabilitation training. Participants are trained to use the exergame system until they could use it autonomously and proficiently.

  2. (2)

    We recorded the participants' exercise videos twice, one week between recordings. And the user experience questionnaire is filled out after the second recording by the user.

  3. (3)

    The accuracy of motion recognition and the analysis of user experience data is counted according to the video recorded by the system.

Evaluation indicators

In this system, the core of the exergame feasibility is the accuracy of pose recognition. Specifically, it assesses whether the system can accurately recognize the exercise motion to control the game process. Whether the user's motion is completed and whether the system makes recognition can be identified manually by video. Therefore, the recording video of the user's experience allows for a quick comparison of recognition and statistics on the number of accurate motions completed by users and the number of accurate recognitions by the system for statistical purposes.

User experience is used to analyze users' perceptions and reactions to the use or expectations of a software system or application service, including users' affective expressions, convictions, appetites, perceptions, physiological/mental responses, behaviors, and fulfillment. There are various empirical research frameworks in the field [31]. In this study, we used a short version User Experience Questionnaire, namely UEQ-S [32, 33]. The UEQ-S is a quick, simple, and reliable questionnaire for analyzing the user experience of the interaction system. UEQ-S includes hedonistic and pragmatic quality with eight items. Hedonic quality refers to the non-technical aspects of user behavior. Similarly, the quality of utility refers to the perceived technical focus to achieve the design goals of the software system or application service. These are shown in Table 2.

Table 2 The evaluation items of the UEQ-S
Full size table

The instrument scores seven Likert statements to indicate quality, with -3 (most pessimistic) and +3 (most optimistic). A value between -0.8 and 0.8 represents a neurological assessment of the corresponding scale, with values greater than 0.8 representing a positive assessment and values less than -0.8 representing a negative assessment. Value over 1.5 indicated excellent quality. Further, confidence intervals and scale consistency are calculated. Confidence intervals are a measure of the accuracy of the mean estimate. The smaller the confidence interval, the higher the accuracy of the estimate and the more accurate the results.

Results

Participants

A total of 27 KOA participants agreed to participate in this study. Of these, 5 participants are excluded because they did not meet the inclusion criteria, and 2 participants did not complete the entire plan. Overall, 20 participants (10 female and10 male; age: 60 ± 5 years) are included in the final analysis.

The result of gesture recognition.

Patients are able to interact with the game proficiently through pre-applicability training for 20 participants. Then, we chose to capture each participant twice, with a 3-day interval between the two acquisitions. The complete process of the training is recorded by computer video. Finally, we recorded the number of completed motions and the number of system recognition for each motion by manual recognition for statistics.

We integrated the number of training motions and the number of system recognition for all participants for the statistics, and the results are shown in Table 3.

Table 3 The result of motion recognition
Full size table

Table 3 shows that the recognition rates of the walk, squat, front straight leg raise forward, and hip abduction is high, reaching more than 95%. The recognition rate of the straight leg raises backward is the lowest, but it also achieved 90.5%, mainly because the leg is backward and occlusion, leading to recognition errors. The recognition rate of the up and down step is 86.5%. We consider that it may be due to the occlusion of the step, which led to the abnormal joint estimated of the foot. For the knee extension in the sitting position, we use a 45-degree angle for capture, which can identify the legs' shape, avoid the error of overlapping the knee and hip joints on the front side, and avoid the mistake of overlapping the lateral sides. The overall motion recognition rate reaches 95.2%, which can meet the needs of interactive interactions.

The result of user experience analysis

Table 4 illustrates the mean value at 95% confidence intervals for the hedonic quality, utility quality, and overall user experience scales. The value indicates that the users gave the exergame system positive reviews. The average value is: pragmatic quality (1.63 ± 0.85), hedonic quality (1.75 ± 0.86), and the overall value are 1.69 ± 0.86. The value above 1.5 indicates that the system gives an excellent user experience.

Table 4 The mean for UEQ-S scale
Full size table

The results of the comparison of the baseline data according to the UEQ-S data analysis tool show that the value for utility quality is good (scoring higher than 75% of the subjects assessed in the baseline data and lower than 10% of the subjects assessed); the value for hedonic quality is excellent (scoring in the range of the best 10% of the baseline data). In contrast, the value for hedonic quality is significantly higher than the score for pragmatic quality, as shown in Figure 5.

Fig. 5
figure 5

The UEQ-S scales benchmark outcomes

Full size image

Discussion

In recent years, 3D human pose estimation based on images has been rapidly developed with increasing accuracy and efficiency. It is applied to rehabilitation training to improve the scientific and popularization of rehabilitation training effectively. Common rehabilitation training systems such as KEEP APP, Joint Rehabilitation APP, and Ring Fit Adventure by video-guided demonstrations or additional sensors generate additional motor and economic burdens [34]. This system does not need any sensor and can complete the rehabilitation training guidance using a common camera. The experimental analysis results show that the recognition rate for the user's motions is high and can meet the requirements of essential rehabilitation training guidance requirements. It indicates that the system has good accuracy and scalability and is suitable for popularizing public health.

The system uses games to encourage patients to engage in active rehabilitation training, which is more beneficial for promoting rehabilitation training. On the other hand, the game's incentive, reward, and punishment measures improve the fun of patients' rehabilitation training and avoid the boringness and fear of heavy training. The user experience analysis shows that the system's user experience is good and can meet the user's experience needs.

The research in this paper also has some limitations. There are not enough types of movements and games included at present, and we will test more types of movement recognition and game scenarios in the future to enrich the training content and meet more training needs. In terms of experimental analysis, the current number of participants is small, and the types of knee disorders are limited. The recent experience with KOA patients has met the clinical requirements, but more experiments are needed to validate and improve.

Conclusion

We developed an exergame system that facilitates knee rehabilitation training based on gesture interaction. In terms of accuracy of motion recognition, the system is able to achieve smooth interaction. The system's user experience is good enough for user experience results. This study provides a good knee rehabilitation tool with high application promotion value.