Keywords

1 Introduction

Musculoskeletal Disorders (MSDs) are the most common work-related injuries in Western countries, reaching up to 2% of the gross domestic product (GPD) in European countries. In France, for example, MSDs represent more than 87% of all the occupational diseases in 2016 [1]. In Europe, millions of people suffer from MSD-associated diseases [2, 3], which impact the workers’ life quality and have a substantial economic and social burden on companies and communities [2]. Absenteeism, permanent disability, compensation and medical expenses are the main, tangible consequences while intangible costs of MSDs (such as a family income shortage, mental damage in the workplace, and loss of skillful workforces) are less considered and might have more social and economic effects [4, 5]. Recent etiological MSDs risk models have well established the role of physical, organizational and psychosocial risk factors in the MSDs development [6, 7]. Although all those factors interact with each other, the physical risk factor - particularly awkward and static postures - remains the most hazardous element of industrial tasks [8].

Fig. 1.
figure 1

Concept image of a user playing the game. A feedback is then given on its posture (upper right).

Full size image

Many ergonomic intervention studies have focused on reducing hazardous tasks by job redesigning, training workers and participation of stakeholders (especially workers). Ergonomic interventions are mostly implemented through the use of new equipment (like lifting tools), staff training and organizational changes [9,10,11]. Most of these studies report workers’ participation and training as a key element for the intervention success [12,13,14,15]. However, several recent studies questioned the effectiveness of ergonomic interventions and showed that classic physical and organizational ergonomic interventions have low efficiency [16,17,18].

Classic training programs (such as lectures, on-site training, posters and brochures) are insufficient to change workers behaviors and improve their postures [11, 19, 20]. Although some studies have found classic ergonomic training to be significantly effective in the short term, sustainability of posture and behavior changes was reported in very few cases [15, 21, 22], the post-intervention effects of training often reducing rapidly after several months. Therefore, manufacturing industries need new methods, which not only involve active workers training to improve postures and behaviors [12, 13, 20] but also provide a sustainable approach in the prevention of MSDs. With this study we want to know whether modern technologies such as Virtual Reality (VR) could contribute to a sustainable MSD prevention. A Serious Game (SG) is introduced as a potential option to provide a sustainable training and behavior changes, particularly in MSD care. Although these technologies are a novelty in MSD prevention, serious games as a pedagogic tool for health and safety purposes have already been used in previous studies [23,24,25]. We also want to know whether a fully immersive environment, which provides feedback on the users MSD risk exposure, contributes to MSD prevention in manufacturing industries (see Fig. 1).

More specifically, this paper wants to propose a SG that combines VR and Motion Capture (MoCap) to prevent work-related MSDs. We hypothesize that an attractive SG increases the workers’ knowledge of MSD risk factors and helps them develop new strategies to reduce risk exposure. That’s why we proposed an innovative tool as a possible alternative to replace the classic training approach.

2 State of the Art

2.1 MSDs Prevention

Ergonomic interventions focusing on MSDs risk factors such as static and awkward postures, frequent bending and twisting, and repetitive work could reduce the risk of MSDs. However, the various components of these interventions are difficult to implement [26]. Recent studies propose a combination of ergonomic interventions to reduce MSDs symptoms: job redesign, technical modifications, ergonomic training, postural advice and organizational changes as the most common interventions used to reduce MSDs symptoms [10, 12, 16]. As industrial workers are directly involved in work and influenced by MSD risk factors, ergonomic worker-targeted interventions (for example, physical exercise program, training, ergonomic advice and instruction on working methods) are mostly integrated into the ergonomic intervention studies [15, 16, 21]. Health promotion actions at the workplace are similarly used to inform workers of good practices and promote preventive interventions at work (e.g., stretching program) or at home (e.g. diet or exercise programs) [27]. However, recent high-quality studies did not confirm the effectiveness of interventions (particularly training and postural advice) in behavior changes [11, 19, 22, 28]: McDermott et al. (2012) investigated the practical implementation of manual material handling training within 150 industry sectors in the UK and concluded that training is more efficient when adapted to a specific task or job. Although the majority of industries currently propose regular training based on legislation, the training efficiency remains unknown. Systematic reviews showed inconsistent and insufficient evidence to conclude that current training approach is effective to reduce awkward postures and MSDs [19, 29]. Changing workers behaviors (such as awkward postures and non-adapted strategies in performing a task) based on current training approach (general lectures and classroom-based activities) is a challenge for MSDs prevention. The novelty in MSDs prevention through training would be to develop a new tool that could suppress the deficiencies of current training approaches. An sector-adapted training approach that implies training a worker in a familiar task might improve training efficiency and achieve successful behavior changes. VR and game technologies are new tools that might change workers behaviors in a playful setting and we believe that they can increase the intervention success and reduce awkward postures by providing an efficient training.

2.2 Serious Games for MSD Prevention

Many SGs for the rehabilitation of those already suffering from MSDs exist [30,31,32,33], but few SGs focus on MSD prevention. A French company (daesign [34]) conceived a MSD prevention SG that is meant as a one-time use. It contains a little of gamification by means of quizzes, an interactive desk and a “find the mistakes” game. The other SGs focus on encouraging stretching exercises as a specific MSD prevention technique. Motion tracking is used to validate the user’s action. Rodrigues et al. have developed a SG joining the stretching and game phase where the user must correct the posture of some virtual workers [35, 36]. Freitas et al. did a series of mini-games that focused on the hands and used stretching exercises as an input for the game [37]. All those games target office workers and we need a SG that targets the occupational risk factors in factories.

2.3 Prevention, Training and Virtual Reality

Even if there are few SGs on MSDs, SGs have largely been used in the rehabilitation and health fields [38] often paired with motion tracking or haptic devices and have proven to be efficient [38]. Moreover, it has been proven that Virtual Reality (VR) helps teach assembly tasks/procedure sequences [39] and is often used as a support for immersive SGs [40]. SGs have even been used with VR and Motion Capture in a prevention game [41], but not for MSD prevention.

2.4 Our Approach

This study proposes a new approach to MSDs prevention, with a SG focusing on real-life movements and applied to industrial workers. We used the different workers’ strategies to develop a SG that combines VR and Motion capture and provides real-time feedback on the user’s awkward postures. This SG is decontextualized in virtual reality to get rid of the effects of the environmental components that play a role in MSDs.

3 Data Collection in Real Industrial Settings

3.1 Data Acquisition

To develop a game close to real work settings, we first created a database of the postures and movements (biomechanical data) of industrial tasks selected from different workstations in the automotive and watchmaking industries. We chose four sectors of the automobile industry (namely injection press, painting, change parts and bumper assembly) and four operations in watchmaking (placing watch dial, setting watch hands, casing movement and visiting). The principal tasks from the automobile industry were packing the bumper and its small parts, preparing, painting and assembling different types of car bumpers. The watchmaker first puts manually the dial on the movement and then fits the hands to the right height and correct position. The operator cases the movement after cleaning the glass and closes the case back. We decided to include these tasks in our study after several visits to the workstations and discussions with industrial stakeholders. These tasks seemed to be more appropriate to develop game scenarios. Twenty automobile assembly operators (8 women and 12 men) and twelve women watchmakers accepted to participate in this experiment. They had a good physical condition without health problems or distress. Most of the participants were polyvalent and could work on several workstations allowing us to measure their postures and movements in different situations. We used nine light T-motion sensors (32 g, 60 \(\times \) 35 \(\times \) 19 mm) to continuously measure the upper limb joint angles at a 64 Hz frequency. Each sensor includes a triaxial gyroscope (it measures angular velocity in degrees/sec), a triaxial accelerometer (it measures linear acceleration in m/s2), and a triaxial magnetometer (it measures magnetic field strength in uT). The T-motion sensors were set by adjustable straps on the head, thorax (fixed on the back), pelvis (located on the hip bone), arms and wrists according to the previous literature [42] and the instructions of the manufacturer (TEA, Nancy, France). The participant was equipped in a separate room near the workstation, then T-motion sensors were set at zero in a reference position. The anatomical reference position is described as the human body upright, feet close together, arms to the side and palms facing inward [43]. These reference positions and the relaxed position of the operator were registered at the start and the end of each experiment. Two cameras filmed simultaneously both sides of the worker. We registered ten cycle times of the subjects’ activity after they got accustomed to the devices placed on their body and the camera installed near them (5 min).

3.2 Activity Analysis and Data Treatment

An experienced researcher in ergonomics and job analysis manually coded the videotaping-recorded activity in CAPTIV software. The subtasks were identified, as far as possible, thanks to the job descriptions provided by the companies. On the basis of the biomechanical model developed by the motion capture system developer (TEA, Nancy, France), we calculated the head, upper arms, forearms, wrists and lower back joint angles in 20\(^{\circ }\) of freedom. The data were synchronized with the subtasks identified in the videos. Each subtask finally had a precise measurement of the upper limbs and trunk joint angles. A global risk score was calculated for each subtask based on the algorithm of the Rapid Upper Limb Assessment (RULA) method [44]. This algorithm generates a single score for each subtask which represents the MSDs risk level (score 1–2: negligible risk, score 3–4: medium risk, score 5–7: high risk).

4 User Posture and Gesture Acquisition

4.1 Upper Body Tracking for Coarse Posture Acquisition

In this project context, the main goal of posture and gesture acquisition is to provide an objective and systematic assessment of the trainee posture in regards to widely-used ergonomic standards. The objective assessment feeds into the Serious Game in the form of a score that gives feedback to the user on his posture and gestures ergonomics. To do so, a system combining coarse and fine gesture acquisition was set up, tested and tuned. An posture and gesture assessment tool based on the Rapid Upper Limb Assessment (RULA) methodology was then developed.

Fig. 2.
figure 2

Human body joints captured by Kinect.

Full size image

Furthermore, the acquired 3D fine and coarse body motions were used to animate the avatar in the Serious Game presented in the next Chapter.

In many industry and work situations, operators suffer from upper body (shoulder, neck, arms, etc.) MSDs due inadequate postures and motions while executing repetitive tasks. That’s why the posture and gestures assessment at the upper-body level is crucial.

The system that has been developed for the acquisition of upper-body gestures and motion is based on Microsoft Kinect 2. Kinect provides the 25 joint angles of the human body as shown in Fig. 2. Most of these angles have been used as provided by the Kinect SDK for posture assessment. Specific rule-based post-processing algorithm for shoulder joints has been implemented to resolve shoulder joint orientation ambiguity [45].

Spatio-temporal smoothing filters have been applied to the extracted silhouette in order to avoid the flickering effect [46].

Fig. 3.
figure 3

Human posture assessment using RULA. (a) You are not correctly seated, (b) You are seated at \(90^\circ \).

Full size image

4.2 Hand and Fingers Tracking

Fine hand and finger movements are of high interest to assess operators’ gesture and posture in high-precision manipulation like in watchmaking industry. The LeapMotion device is used to acquire relevant information about hand posture and gestures: parameters like hand rotation angle, finger flexion and extension angles, computed from the LeapMotion phalanx data (Bones), are extracted.

4.3 Gesture Assessment with RULA

In this project, RULA [47] is used for data analysis of on-site capture data (see Subsect. 3.2), but also to assess the user’s movement during the game. To do so, we developed a tool that automates RULA based on the acquired 3D posture and gesture of human bodies. For each acquired joint we implemented an assessment rule regarding the joint angle and orientation. The different assessment rules have been provided by ergonomic experts. The Fig. 3 illustrates a posture assessment based on two colors: green for safe posture and red for risky posture according to ergonomic rules and recommendations.

4.4 Integration of Fine and Coarse Motions for Avatar Animation

The Serious Game described in the next Chapter includes an avatar that mimics the user gestures and movements. On the one hand, the Kinect is used to capture and forward the coarse body gestures and movements to the avatar animation module. On the other hand, the LeapMotion is used to capture and forward the fine gestures and movements. To have consistencies of both types of gestures and motions, the hand and body 3D data should be aligned. In this project, we implemented a simple, yet robust alignment of Kinect and LeapMotion 3D data using geometric transformations. The Fig. 4 illustrates, through a test avatar arm, the result of this alignment.

Fig. 4.
figure 4

Illustration of the hand-arm integration by combining Kinect and LeapMotion data.

Full size image

5 Serious Game Design and Scenario

5.1 Scenario

To define a game concept and scenario, we had to study the output of the measures taken in the industry workshops. To stay as close as possible to real work situations, the game must replicate real life posture and gesture configuration. We had to avoid the pitfall of losing the game part by making an application which is more a gamified simulation than a game [48]. This means that the game should not represent the objects used in the workplace and that the environment has to be decontextualized. Furthermore, one of the significant challenges is the fact that the game should be scalable in order to adapt to industries handling big pieces (as in the automotive industry) and small ones (as in the watchmaking industry). It should also include the option to be standing vertically or lay horizontally on a flat surface. To sum up, these are the constraints:

  • The game must induce movements close to the ones performed in the workplace;

  • The game must propose an environment and activity that differ from the real work ones;

  • The task must be scalable in size to fit different industries;

  • The task must allow horizontal or vertical layout.

To meet these requirements, we imagined a puzzle-like game (see Figs. 1 and 5), using gears that have to be correctly aligned to adapt to different industries (gears can easily be reduced or enlarged). The user has a board with some gears already placed and empty gear spaces. He must complete the puzzle by placing gears in the right place on the board. At the end of the level, a score is displayed, informing the user of the body parts that are most at risk with MSD according to the actions he took during the session.

Fig. 5.
figure 5

Figure illustrating the application including the hardware and the game. (a) Oculus Rift Headset and sensors, (b) Leap Motion controller (head mounted), (c) Kinect V2, (d) Virtual game.

Full size image

5.2 Environment

In a preliminary study on this project [49], it was found that the environment impact is important and that the environment can be changed without disturbing the user in his tasks. To allow future inclusion of these transitions in our scenario, the user was put in a spaceship (see Fig. 6). This leaves a lot of freedom with the game physics or objects apparition and disappearing. More specifically, in the environment transition, it is possible to put the user in a “holographic room” and change the holographic environment at each new level. These transitions have not been implemented yet, but the environment has already been chosen to ease future work.

Fig. 6.
figure 6

Spaceship environment.

Full size image

5.3 Level Design

Levels. One important thing in building the levels is to balance the cognitive load with the game difficulty. If the cognitive load is too high, the user will not be able to learn the movements, as he will be too taken by the game. But if the cognitive load is too low, the user will be bored and lose motivation and engagement in the game. As it is a Serious Game, we do not want to forget the primary aim, which is MDS prevention. We give feedback to the user at each level validation to incorporate MSD into the game. He gets a global MSD score and a detailed score on the problem he encountered. If the MSD score is too low, the user has to do the puzzle again.

Time. The time is a classical game-play element and a normal constraint in companies, so it is natural to include it in our game. However, it must be used with caution. In fact, introducing time constraints too early in the game will only stress the user. Our primary aim is to teach him to do the things right without rushing or making harmful movements. As it is not the main objective of the game, time will impact the score in the first levels but will not prevent the user from going to the next level.

Fig. 7.
figure 7

One level of the game. (a) Starting point, (b) One gear has been placed, (c) Two gears have been placed, (d) All gears have been placed, the level is finished.

Full size image

6 Results

6.1 Gear-Based Serious Game

As described in the previous chapter, the chosen game is a gear puzzle game. To avoid too high cognitive load, the different gears are color-coded. Each missing gear has its colored axle already on the board and each level contains exactly the right number of gears to complete the puzzle. The user must pick the right gear and put it on its axle. If the gear is released close enough to its intended place, it will automatically set correctly. The gear at this point cannot be picked up. Once all the gears are in place, the system begins to rotate and the user can proceed to the next level. Figure 7 describes the different phases of solving the puzzle, as done by the user in Fig. 8.

Fig. 8.
figure 8

Upper part: user in action on the left with the avatar animated on the right. Lower part: User in action with the first-person view in the corner.

Full size image

6.2 First Users in Action

The Kinect and Leap Motion are used to animate the users’ avatar and allow the wanted interactions with the environment elements (see the setup in Fig. 5). The body is tracked, represented in the game and aligned with the camera. To grab the gears, the user can use his own hands.

6.3 Scoring

The scoring in a serious game is important. The main scoring component is the MSD score, computed by analyzing the user’s movements and postures. If the MSD score is too low, the user has to repeat the level. Time is the other score component, which allows the user to have bonus points if the levels are completed within a defined time frame. At the end of every level the user gets feedback, letting him know which articulation has been at risk during his past actions. The global RULA score is also displayed on a scale from one to seven (see Fig. 9).

Fig. 9.
figure 9

Scoring board. On the left, the body is represented with a risk evaluation for each joint. The RULA Score is displayed on the top right and the game score, solely based on the time, on the top middle.

Full size image

7 Discussion

7.1 Conclusion

This paper has investigated how the game technologies (SG and VR) as innovative tools can provide an alternative for MSDs prevention. We hypothesize that SGs have a significant potential to increase workers’ awareness in MSDs risks prevention. A data acquisition campaign was first conducted in several factories, highlighting the most problematic situations in regards to ergonomic risks. Based on these results, a SG was developed, capturing the user’s real postures, analyzing them and producing feedback in the form of a MSD score. To enforce good practices, the user must have a passing score to access the next game level.

7.2 Perspectives

The first results of this project are promising, as the problematic movements for the target industries have been identified and the positions in-game by scenario and object placement. The next step will be to provide seated levels simulating all the positions requiring a real table and chair. So, after letting the user adjust the table and chair as he wants, we will provide him with a feedback. Transitional environments are one of the main features we are currently including in our application [49], allowing a smooth transition from a virtual, learning-adapted and calm environment to a realistic work environment.