Keywords

1 Introduction

Digital technologies fundamentally change the way we train and do sports. They allow ubiquitous collection of structured and unstructured data that help us to run comprehensive data analyses and learn from exact feedback and precise simulations. Digital technologies boost sports performance by measuring and interpreting data—e.g., fitness, tactical, technical, and mental data—at an individual as well as team level. Although studies have shown that 70–85% of sports performance is determined by the mental state and psychological condition of the athlete (Raglin 2001), so far it is rather hard to measure emotions and affective state. Moreover, while sports analytics is a fast growing and multi-billion-dollar business (Link 2018), the measurement of emotions and psychological conditions such as stress, arousal, flow, anxiety, tension, or aggression has been rather neglected (Bali 2015). Thus, our understanding of the mental state of athletes, the influencing conditions, the mechanisms and influence on the performance, as well as the consequences for the physical and psychological health are still rather unexplored.

With this research, we shed light on the role emotions may play in sports and introduce an artificial intelligence (AI)-based method to measure and analyze athletes’ emotions. We illustrate the AI-based method through findings of the “TAWNY Deep Flow” project, where we aimed to measure the flow state of athletes’ in general and particularly of biathletes and its influence on their shooting performance. Our study shows that AI-based emotion recognition methods allow for classification of specific physiological patterns of the athletes, which help to predict their shooting performance. Based on our case study results, we discuss feasible scenarios to apply Emotion AI technology; we also discuss its potential impact on how we train and do sports and how we may experience and consume it in the future.

In this chapter, we use the term Emotion AI in the broader sense as AI-based technology that helps to detect complex human states in the sports context. These human states are predominantly of emotional nature (e.g., basic emotions) but can also cover cognitive dimensions and mental functions such as concentration or being in “the zone” known from flow theory. Both the affective and mental states are part of psychological processes, which are used for the recognition algorithms in Emotion AI.

2 Advances in Emotion AI

Human emotions play a fundamental role in social sciences, politics, business sciences, medical sciences, and sports. The fundamental problem for all scientific disciplines alike is a valid and reliable measurement of emotions. Currently, self-assessments-questionnaires, observations, and equipment-intensive lab-settings are used to detect emotions. A strong measurement bias is inherent to all these procedures. AI and deep learning open up a completely new and fascinating playground for new techniques to detect and classify emotions. With AI, measurements are much faster (or even instantaneous), less biased, location-independent, and possible at a larger scale when compared to traditional ways of emotion measurement. Accelerated by the advances in face detection and the analysis of voice and physiological data, these technological developments drive the emotion analytics market, which is growing with tremendous speed.

Affective computing is the scientific field dedicated to the detection of human emotional states. It is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. The aim is to allow machines or connected systems to interpret the emotional state of humans, adapt their behavior according to those states, and give an appropriate response for the given emotions (Picard 1997; Bartl 2018). In a first step, human data from facial expressions, voice, or biometric data from wearable sensors like heart rate variability (HRV), heart rate (HR), electrodermal activity of the skin (EDA), skin temperature, photoplethysmography (blood volume pulse), motion (accelerometer, gyroscope) are collected. In a second step, powerful neural networks and deep learning procedures are applied to explore interrelations between physiological data and psychological states with the aim to eventually classify human conditions like basic emotions (e.g., anger, disgust, fear, happiness, sadness, and surprise), stress, attention, productivity levels, and psychophysiological states. The classification of human affects is supported by inventories, scales, and tests from psychophysiological and social science such as the Positive and Negative Affect Schedule (PANAS) (Watson et al. 1988), the Oxford happiness inventory (Hills and Argyle 2002), the flow construct (Csikszentmihalyi 1990), and many others.

The demand for emotion recognition technologies pulls from various domains and industries. R&D departments intend to create smart and emotionally intelligent products (Bartl et al. 2017; Richter and Bartl 2018); human resources try to create safe and productive workplace improvements; sales teams try to optimize customer service (e.g., chatbots, call centers, recommender systems); production teams strive to minimize human failures at the assembly line; market research teams are eager to improve emotion measurement techniques; and start-ups want an easy way to test user experience of brand-new digital applications. In sports, emotion recognition has the potential to change the way athletes improve performance and how spectators experience the actions of their adored sports heroes. Generally speaking, recognizing human emotion is a key dimension to design breakthrough innovations that makes the world safer, healthier, more comfortable, and more productive.

3 Measuring Flow

Driven by advances in Emotion AI, the “TAWNY Deep Flow” research project is dedicated to determining if human flow states can be detected in an automated way (Maier et al. 2019). This idea goes beyond current efforts in emotion research to classify either the basic emotions (e.g., from facial recognition) or human affective states within the two-dimensional arousal-valence space. Valence is the positive or negative affectivity, whereas arousal measures how calming or exciting the condition is.

Flow is the mental state in which a person performs an activity with energized focus, full involvement, and enjoyment in the process. In this “optimal experience,” one gets to a level of high gratification (Csikszentmihalyi 1990). The flow theory has been widely recognized throughout history and across cultures and by various research fields like psychology, marketing, and sports. The renowned Professor Mihály Csíkszentmihályi has driven flow research in sports for more than 40 years (Csikszentmihalyi 1990, 1992). In their book Flow in Sports: The Keys to Optimal Experiences and Performances (1999), Jackson and Csíkszentmihályi investigated factors that influence whether or not flow occurred during an athlete’s performance. Some drivers of flow, according to their model, are the level of motivation toward the performance, physical preparation and readiness, confidence, and focus; some combination of which lead to the optimal experience often described as “playing in the zone” or “feeling on a high.” So far, in-depth information about the experience of flow has largely been obtained through interviews or self-report questionnaires, known as the experience sampling method. Typically, self-report questionnaires are answered during or after performing a task. Critics of this approach suggest that it distracts flow, is inaccurate and time-intensive, and cannot be scaled in real-life scenarios. The TAWNY research project tries to overcome some of the limitations of traditional measurement techniques by developing an automated and real-time recognition of human flow states based on physiological data such as heart rate, heart rate variability, and electrodermal activity.

In a recent TAWNY research study, Maier et al. (2019) propose a method to automatically measure flow using physiological signals from wrist-worn devices. The method is the first attempt to apply end-to-end deep learning for flow classification based on a convolutional neural network (CNN) architecture. The model not only allows for recognizing high and low flow, but also allows estimation of whether the user is under- or overchallenged (i.e., bored or stressed) when not experiencing flow. For data collection, a custom version of the mobile game Tetris was created, which has been shown in psychological studies to lead to different affective states like boredom, stress, and flow (Keller 2011; Harmat 2015). The “easy” level was designed to lead to boredom, the “normal” level allowed for smooth playing and inducing flow, and the “hard” level led to stress. While playing, the participants were equipped with an Empatica E4 wrist-worn device, which captures physiological signals such as HR, HRV (based on blood volume pulse (BVP)), electrodermal activity of the skin (EDA), and skin temperature. Figure 1 gives an overview of the study.

Fig. 1
A flow diagram decodes the flow experiences in sports. The process begins with the flow measurement, followed by the flow classification into easy, normal, and hard using M L, and ends with the evaluation of the validation accuracy.

Study overview for measuring flow (Maier et al. 2019)

Full size image

With the collected data, it was possible to create a first-of-its-kind machine learning model that distinguishes three states: boredom, flow, and stress. Mean validation accuracies of 57, 70 and 71%, for boredom, flow and stress states, respectively, were obtained. It was found that during identified flow periods, the player performs significantly better than in boredom or stress periods. Consequently, the game can be adapted dynamically to a player’s state thereby increasing performance and reducing stress. In a follow-up study with a similar setting and larger number of participants, TAWNY classified low and high flow states at similar rates and further refined the deep learning model (Maier et al. 2019). The study opened up new ways for empirically studying flow and making flow a more accessible psychological concept particularly for sports applications.

4 The Biathlon Flow Study

Pursuing the vision of the TAWNY Deep Flow research project to decode the DNA of flow experiences based on human biometrics and AI, we conducted an exploratory study together with biathlon professionals and Red Bull Media House, the production and media company of the Red Bull corporate group. The primary questions that motivated us to conduct the study were the following:

  • What kind of physiological patterns can be observed during shooting performance?

  • Is it possible to classify patterns and states of flow performance—overload, fear, underload, boredom, and control—that differentiate hits from misses?

  • How can AI-driven data analytics in the context of sports be processed as metacontent in media?

The study was conducted with four male junior professional biathlon athletes of team Switzerland, with an average age of 18.5, coached by former Olympic and world champion Michi Greis. The test track was at Lenzerheide in Switzerland. The study design is illustrated in Fig. 2.

Fig. 2
A block diagram illustrates the study design which involves the following. Pretest to evaluate the study structure, interviews with athletes before the performance, experiment, interviews and short questionnaire after performance, and the data science process.

Study design

Full size image

For the measurement of bio signals during two training days, the wristband Empatica 4 and the Microsoft Band 2, as consumer friendly products, were used to collect data such as heart rate, heart rate variability, galvanic skin response, temperature, movement, etc. Additionally, the athletes were equipped with control devices like the Polar H10 breast belt. Cameras documented the shooting results of 100 shots fired within 70 min of training in competition mode. Afterwards, the biometric data was synchronized with the video data. At the beginning, in the main break, and at the end of each training session, the athletes filled out a short questionnaire on their perceived valence, arousal, flow experience, and general state of fitness. To measure flow, the flow short scale with 13 items was applied (Rheinberg et al. 2003) and used to set data labels for the data analysis. Furthermore, interviews were conducted on the emotional and mental challenges experienced in a normal training session, individual procedures for optimal preparation at the shooting range, and strategies to minimize errors. Figure 3 shows the predictive physiological pattern as a result of the data science process.

Fig. 3
A bar and line graph compares the number of shots and the hit probabilities for the hits and misses. The number of shots for a hit is the highest at 17 with a probability of 100%. Values are approximated.

Physiological pattern and hit probability

Full size image

The light-colored bars show the hits. The darker bars show the misses. Where the light bar exceeds the dark bar there are more hits than misses and vice versa. The black line displays the hit rate (right y-axis) in the corresponding characteristic of the physiological pattern. The x-axis shows the value of the physiological pattern, which is used to predict the probability of hits and misses. Several combinations of physiological signals measured with the Empatica E4 device have been examined. In this study, a physiological pattern consisting of 95% HR and 5% EDA has been identified to be the best predictor for the hit probability. It roughly translates to the mental and physical load the athlete experiences in a certain moment. The pattern’s values are relative and range from 0.0 (the minimum value of the HR and EDA pattern the athlete showed during the competition) and 1.0 (the maximum value the athlete showed during the competition, i.e., high heart rate and EDA). At the bottom of the figure, corresponding hit probabilities are displayed.

Based on 100 shots in competition mode, an average hit rate of 77% and overall flow experience of 83% was achieved. There is a non-linear relation between the physiological pattern and the hit probability. The most notable result of the analysis is that there seems to be a threshold above which the hit probability drops severely. When arriving at the firing range, athletes typically show a high value of the physiological pattern. By relaxing and focusing, they can lower the value and increase their hit probability. At half of their individual maximum value of the physiological pattern they reach a hit probability of nearly 100%. Consequently, athletes might be able to improve their shooting performance if they learn to sense if they have already reached the optimal “flow level.”

The results were integrated as meta information in a training video illustrated in Fig. 4.

Fig. 4
A graphical illustration of a computer that displays a shooter pointing a gun. Below it, there is an image of a heart with the text, HEARTRATE = 141 written beside it.

Source https://youtu.be/tiQwXrbwOQc

Metadata in video.

Full size image

As a limitation of the study, it has to be stated that the applied dataset was small and therefore unlikely to represent all possible states and events during a competition. A larger dataset would allow for a more fine-grained estimation of an individual athlete’s optimal state and highest hit probability. Additionally, more advanced machine learning procedures for the data analysis and classification could be applied to a larger dataset.

5 The Future of Emotion AI Technologies in Sports

Sport offers plentiful opportunities to experience flow. Whatever words are used by athletes to describe flow experiences, flow is always associated with the most precious moments of physical performance, mental productivity, and emotional balance. Yet the flow state seems mysterious to athletes and coaches alike and very hard to generate in a systematic way. The aforementioned studies of the TAWNY research team showed that (1) AI-based approaches have the potential to measure and quantify human flow states in the near future and (2) flow measurement can be applied in sports scenarios. These findings suggest that the systematic work on optimizing flow as an affective state is an additional dimension of preparation like training, nutrition, or psychological motivation, useful to achieve high performances.

The measurement, analysis, and interpretation of complex human states in sports contexts like flow may add value to various actors engaged in sports including athletes—pros as well as amateurs, coaches, spectators—in live or media settings, and media and broadcasting. Table 1 gives an overview of different application scenarios of Emotion AI from the perspective of different stakeholder groups in sports.

Table 1 Potential application of emotion AI
Full size table

In the professional context, Emotional AI analyses may be used to enhance athlete’s performance, to better understand why certain states occur, and what impact they have. They may further allow calculation of how much stress is positive, when it turns negative, and even when it shows long-term effects on athletes’ health.

For amateurs, Emotion AI may allow athletes to better adjust their training to their mental conditions and serve as additional stimuli—helping them to stay enthusiastic and train diligently. Athletes’ different mental and emotional states in moments before the start of a 100-meter run, for example, may serve as additional dimension for comparison.

With Emotion AI, coaches do not have to solely rely on empathy or their psychological knowledge to create their training plans and individual exercises. Instead, they can use real-time insights regarding psycho-psychological conditions of athletes to make exercises and routines even more effective and even less stressful.

Sport fans and spectators at live events or in front of the TV could have access to additional information about what it is going on in an athlete’s mind and body with Emotion AI. What is the level of concentration, nervousness, flow, fear of failing, excitement, confidence of a favorite tennis player during match ball, or a soccer hero during a penalty shootout? This would be a totally new information layer in fan conversations as well as in live event experiences.

Sports broadcasting, to a great extent, benefits from the analyses provided by the commentators and experts—often former professional athletes or coaches. They rely on all kind of statistics, video recordings, simulations, comparison, and provided data. Often it is not the hard facts and technical aspects that make sports events such as world championships or Olympic games so unique and fascinating for the audience, but rather the human emotions, successes, tragedies, and the atmosphere. Thus, providing additional statistics and analytics based on Emotion AI as metadata may help to raise media experiences to the next level.

There is still much to explore in order to fully understand what is going on in sports scenarios. Additional Emotion AI-based methods may help researchers and practitioners to gain insights and accelerate the decoding of unknown questions and secrets. We are aware that some of the presented ideas and scenarios will take time to evolve, however, we are quite sure that others will come true in the very near future. Driven by technological advances and coupled with enormous business opportunities, Emotion AI has the potential to radically change the way we participate in and consume sports.