Running Sports Decision Aid Tool Based on Reinforcement Learning Approach

Abstract

Recently, an increasing number of people is interested in healthy lifestyle and gets involved in a variety of sports activities such as jogging, nordic walking, cycling, hiking or swimming. In order to better plan and track everyday training, people use a wide variety of smart mobile systems on their smartphones, smartbands and smartwatches. However, many of training mobile systems have their functional limitations and in many cases do not allow the user to perform in-depth data analysis in order to optimise the training. Therefore, in this paper, we propose adaptive decision aid tool that supports running sports practitioners in their daily training activities. This tool aims at suggesting the user the strategies for next training session (e.g. the maximum number of training days that can be skipped, duration of the training, the distance that should be covered) in order to meet the goals (in the experiments we have considered winning the one-year running competition). From the scientific standpoint, the tool adapts reinforcement learning in order to propose the runner suggestions that will allow for improving performance.

1 Introduction

The appropriate training and exercises intensity are the key to achieve goal and to improve the performance. However, lack of basic knowledge about training methodology can lead to serious injuries, discouragement and lack of gain. Therefore, lots of people are using training mobile systems to track their progress, fulfil training goals, and monitor achievements. Nonetheless, many mobile training systems have limited functionality (especially in contrast to the amount of collected data) and do not allow for in-depth data analysis that will support the sport’s practitioners in decision making. In many cases, these tools have hard-coded training scenarios, measured data is sometimes incomplete, and many assumptions or suggestions are too generic. Moreover, recently gaining in polarity so called competition-based social running in an opinion of many experts is not a good approach to training, because it makes people train faster and harder than they should.

Understanding the collected data is an important aspect when it comes to optimisation of training activities in order to achieve better results in shorter time. In many cases, visualising the rough data (e.g. amount of burnt calories or covered distance) is not enough for inexperienced runners to assess the progress.

Fig. 1.

High level overview of the proposed RunApp mobile training system. The mobile application communicates with server (upper left corner) in order to synchronise data (e.g. user questionnaires, feedback and recorded data), delegate core computations and retrieve suggestions regarding training

Therefore, in this paper, we propose the adaptive decision aid tool that supports running sports practitioners in their daily training activities. This tool is based on reinforcement learning that looks for optimal training policy which will allow its user to achieve satisfactory results in constrained environment (e.g. concerning weather condition, user mood, preferences, etc.).

2 Related Work

Nowadays, the wearable computing and smart devices have become popular tools commonly used by runners to monitor everyday training progress. In principle, the smartphones equipped with flexible operating systems and a wide variety of sensors have eliminated the obstacles to create mobile training systems. Therefore, there are plenty of different solutions varying in concept and complexity. For instance, there are systems such as Endomondo [1] supporting different types of sport activities (e.g. jogging, biking, hiking, etc.) enabling users to record key training characteristics (e.g. speed, burnt calories, time, track, etc.), share training details via social media, have insight into training details of other people, etc. However, this topic is also the area of interest for many researchers. The variety of statistical and machine learning techniques have been used in order to support people practising sports and to improve their achievements and performance. In [6] authors have proposed a decision-making system for a multi-step training scenario. The system is based on dynamic programming optimisation and Markov Decision Process formulation. Another system for runners has been proposed in [4]. The goal of the system is to facilitate social running in a city park by planning the route, supporting local running communities, and promoting a healthy lifestyle. In the literature, there are also examples of adapting machine learning and data mining techniques also for other sports than jogging. For instance, in [5] authors have proposed MONEYBaRL system which uses reinforcement learning to exploit baseball pitchers decision making. In principle, the proposed algorithm allows for finding optimal pitcher-specific and general (against a collection of pitchers) batting strategies. The more holistic approach has been presented in [2], where authors proposed recommender system for running professionals and amateurs providing a wide range of personalised information concerning both workouts and diets. This approach is based on a combination of expert knowledge (maintained as an ontology) with a recommender system.

All the above-mentioned works show that different sports activities can be supported with modern wearable devices and advanced data mining techniques. However, when it comes to running sports, it seems that the problem of optimising different runners behaviours has not been well addressed yet. Therefore, in our approach, we focus on machine learning techniques that would allow the runner to (i) evaluate the training progress in different time spans as well as to (ii) propose an optimal strategy for achieving the training goal.

3 Proposed System

As it is shown in Fig. 1, the proposed system consists of several elements that enable the user to track the training progress. The runner is equipped with the mobile device with pre-installed software that facilitates bi-directional communication with the server, reads current position of the user with GPS sensor, measures an amount of steps by means of an accelerometer and provides the user with guidelines. In particular, the software allows the runner to select the appropriate type of training and have insights into current progress and recommendation. All relevant measurements, as well as user feedback (after training session we ask the user to fill in a short questionnaire about the training aspects), are uploaded to the server.

The server-side acts as a cloud storage but it also is delegated to run more complex analysis on the collected data. One element of the analysis is the evaluation of the training session. Here we apply machine learning and data mining algorithms to learn the classifier which will tell the user that he or she is progressing or not. Usually, the evaluation is not straightforward (e.g. few bad workouts does not imply that whole training program is bad) as the runner’s achievements are influenced by external factors (e.g. weather, mood, diet). Therefore, we ask the user to fill in a questionnaire after the training.

The second element of the analysis concerns the training strategies. There are lots of experts opinions how well-organised jogging training plan should look like. However, the amount of information is usually overwhelming for the inexperienced runners. Therefore, here we adopt one of the reinforcement learning technique to analyse the data collected from different users in order to identify optimal approaches for achieving different training goals.

4 Training Evaluation and Decision Support Algorithm

The aspiration to achieve better running results should proceed in a reasonable way, through gradual extension of distances covered and workout duration, taking organism regeneration into consideration. In result, the runner’s condition could be improved and the risk of injury reduced. Due to the fact that users do not have to make progress in every training, the collected data were considered on a weekly basis.

A runners progress was estimated based on the increase in distance and time as well as on the number of runs which were registered in the following weeks. The implementation of machine learning algorithms in the application allows detecting lack of running progress and searching reasons of that on an ongoing basis. It improves the way of preparing personalised tips which help users to develop a physical condition.

Fig. 2.

Proposed algorithm overview

4.1 Optimisation of Jogging Strategy

In order to capture the everyday behaviour of runners, we model different training strategies as Markov Decision Process (MDP). More precisely, let $\lbrace X^{(t)}\rbrace$ be a Markov process with finite state space $S=\lbrace E_1,...,E_n \rbrace$ of n elements (states), where each state represents set of measurements. The number of measurements depends on the scenario. In principle, these contain such information as covered distance, elapsed time, burnt calories, etc. In our case, we assume that transitional probabilities between states are necessarily stationary. In other words, $p_{E_i \rightarrow E_j} = P(X^{(t)}=E_j | X^{(t-1)} =E_i)$ is different for different t. Considering an example where runner starts a one-year competition and in time $t_0$ covers distance $d_0$. Under that circumstances, we may not expect that the runner will cover the same distance during each training session, as we must consider that at some point the performance will improve (as the runner will become more advanced) or deteriorate (e.g. due to tiredness) (Fig. 2).

An example of graph presenting possible states and transitions has been presented in Fig. 3. In fact, it is modified Markov transition diagram, since (for readability reasons) the transitions probabilities have been represented with varying thickness of graph edges (the thicker the edge is the more probable the transition).

The classical MDP is defined as 5-tuple $(S,A,P,R,\gamma )$, where S indicates mentioned above finite state space, A indicates finite action space, P transition probabilities, R a pay-off function (an intermediate reward), and $\gamma$ the discount factor (importance between future rewards and present rewards).

We define optimal policy $\pi$ as a chain of actions $\lbrace a_0, ..., a_n \rbrace$ (that particular runner can take), which will maximise the expected value function V at every state of a Markov process. In other words $\pi (s)$ will tell us what is the optimal action in a state s. The value function is described by equation:

$$\begin{aligned} V^{\pi }(s) = R(s,\pi (s)) + \gamma \sum _{s' \in S} P(s' | s,\pi (s)) V^{\pi }(s') \end{aligned}$$

(1)

In that sense, the optimal value function $V^*$ of a state is the reward for that state, plus the discounted expected reward for following the optimal policy from this state:

$$\begin{aligned} V^{*}(s) = \underset{ a \in A}{\mathrm {max}} \lbrace R(s,a) + \gamma \sum _{s' \in S} P(s' | s,a) V^{*}(s') \rbrace \end{aligned}$$

(2)

In the same way, we can calculate the optimal policy in state $\pi ^*(s)$, using the following formula

$$\begin{aligned} \pi ^*(s) = \underset{ a \in A}{\mathrm {argmax}} \{ R(s,a) + \gamma \sum _{s' \in S} P(s' | s,a) V^{*}(s') \rbrace \end{aligned}$$

(3)

In our case, in order to estimate the P we have used publicly available historical data obtained from Endomondo web portal. The details about the collected measurements have been presented in Sect. 5.

Fig. 3.

Example of users states transitions over time windows (for readability reasons some edges have been removed)

Fig. 4.

Cumulative amount of calories burnt per meter (top) and cumulative distance measured for different runners (bottom) during different evaluation periods

5 Experiments and Results

In this section, we have described two experiments. One is related to an evaluation of training performance by means of machine learning algorithms. For that purpose, we have used the collected measurements of different characteristics measured during the training and accompanied with user feedback in form of filled-in questionnaires. The second experiments concern evaluation of the algorithm for training strategy optimisation.

Currently, our system is under development and the number of active users is still limited to run reasonable data analysis in order to find optimal training strategy. Therefore, to prove the correctness of our assumptions, we used publicly available data of 10 competitors (participating in common jogging challenge) that have been recording their training progress during the period of one year.

Table 1. Final characteristics measured for different runners

5.1 Training Performance Evaluation

In the experiment, the effectiveness of running progress estimation was tested through machine learning algorithms which are implemented in well-known WEKA software [3]. A ten-fold cross validation was used to assess the quality of the generated prediction models. The results obtained have shown that Bayes Net and SMO are the best classifiers for the research of running progress on the basis of a training set. For the first of them, the accuracy of 94,1% was obtained and for the second – 91,2%. The Bayes Net algorithm forecasts sporting results improvement better than SMO (accuracy is 96% for Bayes Net and 88% for SMO). However, SMO is the only one (among tested classifiers) which detects a lack of progress faultlessly. In that case, Bayes Net ensures 88,9% of accuracy. In the Fig. 5 the ROC (receiver operating characteristic) curves are shown for chosen algorithms. They inform how correctly classifiers separate positive and negative class observations. Following the scale of the evaluation of classification accuracy based on the fields under a ROC curve, we can claim that Bayes Net and SMO divide instances with high effectiveness. From among the tested algorithms, Random Forest is characterised by high accuracy (91,2%) of progress prediction. However, it does not manage forecasting lack of progress as well (77,8%). In the application algorithms, KStar and Filtered Classifier should not be implemented. It is justified by low, indicating high forecasting randomness, the accuracy of classification of observations which do not prove sports development. Also the low standard of Kappa statistic (determining the extent to which a suggested model matches reality) for Filtered Classifier (0,29) and a small field under the ROC curve for KStar indicate a large probability of randomness.

Fig. 5.

ROC curve comparing different classifiers performance

5.2 Training Strategy Optimisation

For this evaluation scenario, we have used publicly available data obtained from Endomondo system. We have identified 9 similarly advanced runners participating in the same running contest/challenge, which was one year long. We have divided the data into two parts each 6 months long. First, the dataset was used to train our system while the second part was used for evaluation purposes.

The training dataset was aggregated in 12 evaluation evaluations periods (each 2 weeks the progress was measured). In each evaluation period, we have calculated the number of skipped trainings, distance covered, average distance in the evaluation period, average speed, a number of burnt calories, and amount of calories burnt per one meter.

The statistics calculated for each evaluation period (and each user) have been clustered with k-means algorithm, where $k=6$. This allowed us to roughly estimate k different user training states.

Using the methodology described in Sect. 4.1, we have estimated the training policy and evaluated with the evaluation dataset (remaining 6 months of the competition).

In Fig. 4 there are presented cumulative achievements of 9 different users practising jogging during the period of 6 months. Also in the Table 1 final achievements of selected users are shown.

In Fig. 4 and in the Table 1, there are also characteristics showing achievements for the hypothetical runners (pol_1, pol_2, and pol_3). They follow different (suggested by the system) training policies.

The runner pol_1 follows the policy, which proposes the most probable training path – the one with highest transition probability for the estimated transition-states diagram. As expected, the results for this policy is average (in the sense of the covered distance). On the training data, the pol_1 runner took the fifth place and the seventh place on the evaluation dataset.

The runner pol_2 follows the policy, which proposes the most effective training path (the one which will allow the runner to cover the longest distance). As it is shown in the Table 1 the policy works also on the evaluation dataset. However, the amount of estimated burnt calories is quite high.

On the other hand, the runner pol_3 follows the policy, which balances the effectiveness with a number of burnt calories. It can be noticed that this policy will also work for the evaluation dataset. Nonetheless, a number of burnt calories is significantly lower.

6 Conclusions

In this paper, we have proposed innovative system supporting people practising running sports. There two main contributions of our research, namely: (i) supervised learning approach for training performance evaluation and (ii) reinforcement learning technique for finding optimal running strategies. In the proposed approach we have used two datasets to prove correctness and evaluate the performance of the system. The obtained results are promising and show that the proposed approach can be effectively used to evaluate runners training performance and to support them with suggestions for optimising the running strategies.