VisuaLeague: Player performance analysis using spatial-temporal data

Abstract

In recent years, the phenomenon of eSports has been a growing trend and consequently, in addition to players, other groups of users, including coaches and analysts, took an interest in online video games and the data extracted from them. Among many types of video games, one of the most widely played is the MOBA (Multiplayer Online Battle Arena) League of Legend (LoL) game. Similary to traditional sports, players and coaches/analysts analyse all game events, such as, players’ movements, to understand how they play to define new strategies and improve their performance. Our main goal is to get a better understanding of which visualizations techniques are more adequate to handle this type of spatio-temporal information data, associated to player performance analysis in video games. To address this goal, we inquired players to identify the analytical questions they need to support for performance analysis and designed the VisuaLeague prototype for the visualization of in-game player trajectories, using animated maps, and events during a LoL match. This paper presents a user study to evaluate the adequacy of animated maps and the analytical strategies followed by players when using spatio-temporal data to analyse player performance. The results support the adequacy of using the animated maps technique to convey information to users in this context. Moreover, they also point out towards a high degree of importance given to the spatio-temporal components of the data for player performance analysis.

1 Introduction

In recent years, the phenomenon of eSports has been a growing trend. Consequently, in addition to players, new groups of users took an interest in video games, and the data generated from them, including analysts and coaches in professional contexts, and spectators in more casual ones. Although these groups may have different goals and analyse video game information differently, evaluating and improving player performance is often a common denominator. One of the most popular genres in eSports is the MOBA (Multiplayer Online Battle Arena), having registered an increasing participation in player base and event viewership [9, 36], such as in games like League of Legends (LoL) [21].

Similarly to traditional sports, eSports provide a great number of relevant spatio-temporal events for analysis. The main difference between traditional sports and eSports is how easy it is to use telemetry techniques to record game data for further analysis [2, 37]. With the evolution of technology, it has become easier and more common to instrument video game code with telemetry techniques to record these events, which in turn, leads to large volumes of data that can be collected over time. Nowadays, these voluminous datasets include spatial-temporal game related data, such as: the players’ trajectories over the game environment, representing their movement; the spatial-temporal events in which they participated; and the actions they performed throughout the game, which can provide an understanding close to the actual gameplay experience and can be used to extract behaviour patterns and strategies [4, 7, 8, 25]. Out of all explored references, the Riot Games (developers of LoL) API [9] is one of the few publicly available offering access to game data, that includes detailed information of player movement and relevant game activities; therefore, making it a potential case study in this area of research.

Previous studies already addressed the analysis and visualization of spatial-temporal data, applied to movement of humans, car or boats, among others, with several map-based techniques, like 2D static maps or 3D space-times cubes [1, 19]. However, choosing proper visualization techniques remains an important and, sometimes, controversial challenge, as previous studies suggest the adequacy of different techniques for different types of visualization tasks [11].

In the context of video games analysis, this problem is also relevant, due to the current interest of video game developers and several independent groups in providing tools showing spatial-temporal information for any type of player, inexperienced in data visualization, to analyse their data, progress, and improve their skills and performance [2, 7]. Although this increased interest, the visualization of games/players in video games it still an unexplored research area [39]. The surveys [2, 40] also concluded that the visual representation in the area of vdeo games is not very explored and pointed that current visualization techniques to analyse game data are mainly limited to display aggregate statistics or heatmaps and not well suited for conveying the spatial-temporal data or the flow/evolution of the game [37].

We aim to contribute to this body of research by exploring adequate interaction and visualization mechanisms to extract knowledge from those gameplay datasets helping players in their analytics tasks.

Considering the aforementioned challenges, this paper focuses on three main research questions: i) Can inexperienced users (i.e. players inexperienced with data visualisation) extract relevant information to solve common tasks (i.e., assess player performance)? ii) Are animated maps useful and well received representations to achieve this? and iii) How do these users analyse the data, i.e. which interaction patterns and criteria are the most used? To achieve these goals, we developed the VisuaLeague prototype [35], which displays the players’ trajectories and the main events of a match, using animated maps, not often seen in the literature. As suggested above, we use LoL as our case study, and despite its highly specific concepts (described below), we argue that this research is still applicable into other related areas/games, given the importance attributed to spatio-temporal information and the similarity in the most important types of events to analyse. Moreover, among the various types of users, casual players consist of the majority of LoL’s player base, making them the study’s target audience.

The remainder of this paper is organized as follows: Section 2 presents the most relevant related work; Section 3 describes LoL, the data used and the VisuaLeague prototype; Section 4 describes the user study and its results; finally, Section 5 is dedicated to conclusions and future work.

This paper is an extended version of a paper presented in the 22nd International Conference of Information Visualisation (IV 2018) [12]. All the sections were improved, but essentially in Section 3 we added the description of the development process and technologies, the prototype was deeply described, in particular the creation of the players trajectories based on all the location data, and all the VisuaLeague components are more detailed. In Section 4 we detailed the description of the tasks and the methodology. Additionally, and in order to present a quantitative analysis, this section presents the dependable variables considered in the user study, namely, task completion time, actions, and accuracy. Consequently, in the Section 4, the Section 4.4.1 was included with a quantitative analysis considering the dependent variables.

2 Related work

The challenges related with the video-games data analysis, specially in the MOBA genre, considered as team sports themselves, can be associated to those faced in sports analytics, and thus in geographic spatial-temporal data analysis. For these reasons, our related work will fall into two main categories: spatial-temporal visualization, including sports-related data visualization, and visualization techniques applied in the field of video games.

2.1 Spatial-temporal visualization

Although applied in different contexts, the analysis of spatial-temporal data, such as the trajectories of players in the virtual world is not completely different from the analysis of real-world human trajectories [7], and, therefore, the results from this research area should be considered.

Several visualization techniques have been proposed to represent spatio-temporal data. Due to their simplicity, the most used techniques are based on static maps. One such approach is to use points and/or lines over the geographic area, in which each point can symbolize a visited location [6]. Other types of information can be represented using different visual variables [2, 37, 40]. For example, the width of a line, representing a trajectory, can be used to highlight the temporal component of movement, to represent an object’s speed [37].

Choropleth and heatmaps are also common approaches, where maps are coloured according to certain metrics’ values (e.g., death rates or population density). While choropleth maps associate a colour to a specific location [2], heatmaps aggregate that information and represent it with a gradient [25, 38]. However, these approaches require a large amount of data to be useful, and can often obscure important information due to design constraints [37]. For example, when a 3D environment is mapped into a 2D heatmap, information related to the height is hidden.

The inclusion of temporal information in various types of maps is already a common concern in data visualization. The spatio-temporal cube (STC) combines these components [15, 20], typically using two dimensions to represent space (coordinates x and y representing lon and lat, respectively) and a dimension to represent time. This technique has the advantage of representing a continuous variation in time and space, enhancing the perception of movement [4] and the understanding of the relations in the information displayed [10]. However, it can become confusing due to the 3D perspective imposed by the cube, often requiring additional interactive controls to minimize these limitations [4, 10].

Animated maps represent singular images in a sequence, enabling the analysis of the variation of events over time in the geographical area [6]. Animations can be temporal and non-temporal, depending if they are used to describe the variation of the data in the map in a chronological order or to demonstrate shifts in dynamic phenomena, such as, a change in perspective [6]. The use of animated maps can be helpful to reveal spatio-temporal patterns that other visualization obscure [24], like the speed of movement. However, past research yielded inconclusive results regarding this technique [22, 32, 34]. Previous studies have concluded that the use of animated displays is generally ineffective for complex analysis tasks [19], despite being often considered as more engaging to use, particularly for presentation purposes [26, 32]. These results suggest that further research over animated maps is needed, and that this technique may have an important role in personal visualization contexts, where users may just want to fill the gaps in time when bored or curious [17].

The application of visualization methods into the area of sports is also highly relevant as it explores many avenues for spatial-temporal visualization and is often used by professional athletes and sports institutions. Overall, the tools developed to aid in the analysis of gameplay differ greatly based on each type of sport. Nevertheless, the objectives regarding their usage share several similarities, including the analysis of player performance through various metrics (e.g. points scored, number of passes), their trajectories, or the analysis and reconstruction of specific game events. The application of visualization methods enabling an in-depth data analysis was already introduced in many sports, such as soccer [13, 27, 33], tennis [29, 30] and baseball [5, 28].

Gudmundsson [14] provides a literature review of the most relevant works that use spatio-temporal data from team sports and identify several research questions. A relevant challenge is the lack of user studies exploring the analytical questions that experts need support for and which types of techniques can be understood by them. Our main objective goes in agreement with this challenge, by studying if players are capable of reaching meaningful conclusions when using animated maps.

2.2 Spatial-temporal video game analysis

Although the most significant body of work presented in the literature is dedicated to game developers, there has been an increasing interest in showing spatio-temporal data to players [37]. These users, despite knowledgeable in regards to a specific game, in some cases even experts, are often inexperienced in terms of trajectory data visualization and analysis. Nevertheless, they may still be interested in analysing their personal data, even if in a more casual scenario to improve their gameplay performance and, consequently, their experience [16, 17, 31]. This growing trend has already been addressed in several projects, including in professional contexts. Researchers, such as Drachen and Schubert [7], have already identified the lack of software tools to support the visualization of spatio-temporal data by inexperienced users as a relevant open issue in the area of visual-game analytics. In addition, some game developers and independent groups have also showed interest in providing visualization tools and APIs that grant, nearly any user, access to player generated data [37]. Notable examples include the official Match History web-page from Riot Games¹ or third-party applications like OP.GG² and LoLKing³, which provide general game statistics referring to the players’ performance or a gaming session. However, previous studies often emphasize that these techniques do not take advantage of their full potential, nor are properly used in function of the data available or the needs of the target audience [2, 23, 25, 40]. Furthermore, althought some of these applications provide complex map or graph-based visualizations depicting the evolution of a gaming session [2, 7, 8, 18, 37], the majority is focused on the display of aggregated statistics. This makes the detection of valuable spatio-temporal details difficult, if not impossible [21] and, thus, it is often necessary to combine them with map visualization techniques, to provide a more informative context [7].

Additionally, current gameplay analysis can also derive from the analysis of replays used to visualize the events of a match, namely the LoL replay system (accessible from the Match History web-page from Riot Games). However, this analysis can be very time consuming and often needs specific software and powerful hardware.

Our work intends to address some of these challenges. The analysis of the state of the art reveals that, to the best of our knowledge, there is no case in which animated maps were studied in the context of visual game analytics, specially, with an inexperienced population in terms of data visualization, therefore emphasizing the necessity to further study this technique.

3 VisuaLeague development

In this study, we use MOBAs as our primary case study, particularly the game League of Legends (LoL). As such it is important to acknowledge the necessity of addressing specific concepts associated to this case study that may affect its understandability. The following sections focus on the most important concepts associated with LoL, followed by the description of VisuaLeague’s main features.

3.1 League of legends

League of Legends (LoL) is characterized by being played by two teams, of five elements each, aiming to destroy the enemy’s base (the Nexus). Matches have no time limit, but generally last 30 minutes divided into three stages: early-game (until 15 mins), mid-game (until 25 mins) and late-game (until the end). A match takes place in a squared map composed of three lanes that connect both teams’ bases (Fig. 1). Along the lanes, there are various defensive structures, called towers and inhibitors, that must be destroyed to reach and destroy the enemy base. Periodically, in each base, automated units (minions) spawn and travel along the lanes to fight each other. Between the lanes, there are neutral zones, the jungle, inhabited by various monsters (e.g., Dragon and Baron).

Fig. 1

League of Legends map and important locations

Players in LoL are known as summoners and can use a unique character to fulfil one of five roles: Top, Jungler, Mid, ADC, and Support. Characters become stronger by earning gold and experience points (XP). XP unlocks new abilities and stats, and gold is used to buy items to enhance a character’s abilities. These can be earned by destroying/killing jungle monsters, minions, structures, and opposing players.

Player vision regarding the enemies’ position has a limited radius, which can be temporarily expanded by using wards, thus supporting the use of strategies around vision control to ambush the opponents.

It is possible to understand how important spatial and temporal information are when analysing this game. Similar to a real-world scenario, where monuments or markets would be considered points of interest, and picking/dropping passengers would be seen as important events, in this context, these would be the defensive structures, the various jungle locations, and the events associated to, e.g., deaths and destruction of structures.

Based on the analysis of several professional matches and players involved in the design process (described below), five types of events were identified as being the most relevant, and have specific aspects in terms of their spatio-temporal locations:

Player kill/death: :

grants the ”killers” gold and experience, and provides a temporary advantage to one of the teams, allowing it to try to take other objectives. This type of event can happen anywhere in the map at any given time.

Slaying Dragons: :

provides the team gold, experience and enhancements in battle capabilities. This type of event can happen anytime during a match over a specific area in the map.

Slaying Barons: :

the Baron provides similar rewards to the Dragon. Although this event is also bound to a specific zone in the map (different from the Dragon’s location), it can only occur after 20 minutes of gameplay.

Destroying Towers: :

destroying a tower provides gold and experience. Players tend to develop strategies around grouping to facilitate the destruction of these objectives. Events related to towers can happen at any time but, since these are stationary structures, they can only occur in a fixed set of locations.

Destroying Inhibitors: :

players can only attack the enemy’s nexus when they have destroyed at least one inhibitor. When an inhibitor is destroyed the minions that occupy that lane are empowered, thus applying increasing pressure to the enemy team. Although this event can happen anytime, it is expected to occur in the later stages of the game. Inhibitors are also stationary structures located in much smaller geographical areas (inside the teams’ bases).

Based on this information, several types of analysis questions can be made that, when answered, can help players understanding their impact in the matches and improve their performance for future matches.

3.2 VisuaLeague prototype

To provide a tool for player performance analysis, VisuaLeague focuses on the use of animated maps to represent the movement of the players during a match, as well as to display the events that took place. This analysis can focus, for example, in the trajectories players take or on their contribution to certain events that happened during the match. Other techniques, such as the ones used in other applications with the same purpose, were also employed to allow the visualization of thematic attributes like gold and items, although, only attributes that have a temporal component associated are represented. An overview of the developed prototype can be observed in Fig. 2.

Fig. 2

VisuaLeague Interface

3.2.1 Development process and technologies

To better understand the requirements of casual players, a series of interviews was conducted with a total of 30 casual LoL players, inquiring them on the type of information they consider the most relevant and how they use it to analyse player performance. Among the most common responses, participants shown a significant interest in the evolution of their spatio-temporal data, alongside sets of thematic information similar to those shown in other applications. This information is then used to associate a player’s impact over the outcome of an event and to compare the actions and decisions of opposing players throughout the match.

The VisuaLeague prototype consists of a web application that can be interacted with using any web browser. The main technologies used to develop the prototype were PHP (for the back-end) and HTML5 (for the front-end). The PhpStorm IDE from JetBrains was the main tool used to work with these languages. The Laravel PHP framework was used as a basis to construct the back-end. This back-end provided functionalities that executed the various necessary calls to the Riot Games API to automatically fetch the data needed for the prototype to display information to the user. This process is conducted automatically, with the user only required to insert the name, region and the champion played (optionally) of the player he/she wishes to visualize in the front-end component. The front-end was constructed using the Bootstrap framework in conjunction with the jQuery JavaScript library, the d3.js JavaScript library and the noUiSlider range slider library.

3.2.2 Prototype description

When opening VisuaLeague, the user is presented with the twenty most recent matches played by a certain summoner (Fig. 3), and it automatically searches for older matches as the user scrolls further down. To perform the automatic search for match data, the user is required to provide the summoner name of the player, the region that player belongs to, and optionally a champion to filter out the results of the search. In this page, there is information regarding the outcome of each of the matches (green for victory and red for defeat), the champion and position played, when the match took place and its duration. By clicking on the details button on any of the search results, the user is presented with a page where the analysis of that match can be conducted (shown in Fig. 2).

Fig. 3

VisuaLeague Match Search page

All the information presented in VisuaLeague was automatically extracted from the Riot Games API and includes the position of each player, every minute, the spatio-temporal events described in the previous section, and various thematic metrics that vary over time, including: player KDA (Kills/Deaths/Assists), gold acquired, minions slain, and inventory purchases.

The information regarding the position of each player is available roughly once every minute. This information allows the visualization of a simulation of the trajectories that players took during a match. Because the information about player position is only available once every minute, the trajectories simulated using only these points wouldn’t be able to accurately describe the movements of a player during the match. All location-based information is used to simulate the players’ trajectories. This is based on the shortest path that does not cross the physical barriers of the game, thus, supporting the representation of realistic trajectories, unlikely to confuse potential users.

To create trajectories that were satisfactory and had a high degree of reliability, enough to allow this type of analysis, a graph was developed. The nodes of this graph represent places where a player can decide to change its path in a significant manner (Fig. 4). The edges represent the path that a player can take from a node to another. It is important to mention that the edges of the graph do not go over walls or other obstacles present in the map. This way, the resulting paths represent routes that any player could have taken during the match because, even though it is possible to use certain in-game mechanics to transverse these barriers, not all champions can perform such actions and therefore they are not represented, as there is no information in the API about when and where these mechanics are used.

Fig. 4

VisuaLeague Nodes of the graph

With this mechanism, if at time t₀ a player was in location P₀ with coordinates (x₀,y₀) and, at time t₁ the same player was in a location P₁ with coordinates (x₁,y₁) then, for both points, the Euclidian distance to the location of each node of the graph is calculated to identify which are the nodes closest to these two points. Let’s assume that these nodes are N₀ and N₁ respectively. After that, to create the approximate path the player took during the time between t₀ and t₁, the shortest possible path between N₀ and N₁ is calculated.

The path the player took during that time is then assumed to be the path calculated between these nodes. It is important to mention that this path is not necessarily a straight line between these nodes since, to go from node N₀ and N₁, it is very likely that one must pass by a set of intermediate nodes N_i that connect N₀ and N₁ in the graph. This algorithm is repeated for every pair of consecutive points that represent the position of each of the players. The total path the player took during the match is assumed to be the concatenation of all the calculated sub-paths. To more accurately create this final path, the position of several of the events where the player participated were also taken in consideration. Some of these events, such as the ones related to the item shop, do not have a spatial component associated but, since they can only be performed in one specific location on the map, that position was also added to the initial set of points that contains the location of the players. Furthermore, players can teleport back to the base at any given point. However, the occurrence of this event is not specifically provided in the information returned by the API. To simulate this event, it was taken into consideration that players need to be in their respective base to interact with the items shop, hence that would mean they either died prior to that event or teleported back before they bought the items. Taking this into consideration, to simulate the players returning to the base, the icons representing them in the map are instantly moved to their respective bases, prior to an event related to purchasing, selling or returning items. The representations of players were also instantly moved to the base after a small delay (respawn time) to represent what happens when a player dies.

As shown in Fig. 2, VisuaLeague is composed by three main components: the animated map (i), the interactive controls (ii), and representations of thematic metrics (iii - viii).

To combine the players’ needs with our main research questions, VisuaLeague uses animated maps to display the estimated trajectories of players during a match, and the various events that took place. To minimize the potential learning curve with the prototype, the map used as a background and all colours/icons used in are similar to those used in game.

In animated map (Fig. 2i) the players are represented in a similar way as the one presented in the in-game map, using circles with the images of the chosen champions. These are the representations that will follow the paths calculated above to simulate player movement. The icons associated with players and structures are color coded, with blue representing the team in the southwest corner and red representing the team in the northeast corner. Global objectives are also represented using familiar symbols with a neutral color (yellow). To emphasize the analysis of the direct comparison between two players from opposing teams (commonly done by casual players) and simplify the data visualization in this stage of the study, VisuaLeague allows users to simultaneously analyse the information of up to two players, one from each team. The animated map allows users to visualize the evolution of the trajectories of the players over time, as well as the events where these players participated.

Depending on the type, events are represented with two or three icons in a row (Fig. 5), in the location on the map where that event happened. Different symbols are used to represent different types of events. The represented events have three components: the individual responsible for the action (left icon), the action (middle icon) being taken and the target for that action (right icon). Figure 5(left) shows an example of such an event where the red player (left icon) killed (middle icon - in LoL crossed swords represent a kill) the blue player (right icon). In this case the red player would be the individual responsible for the action, the action would be killing another player and the blue player would be the target of this action. Event representations with only two icons represent actions taken by a player that do not have a target, such as going back to base or placing a ward (Fig. 5 (right)).

Fig. 5

Event representation using three icons (left) and two icons (right)

As mentioned above, most events have either a spatial component associated with them or can only occur in a specific place and thus, it is possible to represent them. That is not the case for ward events. In this case only the temporal component is available and, therefore, these events’ location will be attributed to the place where the champion is when the event occurs. This decision might not reflect with total accuracy the location where interactions with wards took place but, it provides enough information that, in conjunction with game knowledge, might help users understand vision control strategies being employed.

The evolution of the trajectory and the events being shown on the map, as well as the information displayed in the other visualizations, evolve over time. As expected from common animated applications [1], the controls shown in Figs. 2ii) and 6 allow users to select the time instant/interval they want to analyse, to pause/restart the animation. adjust the animation’s speed and the duration that events stay visible in the map.

Fig. 6

VisuaLeague Animation Control (zoom of Fig. 2ii)

The slider in Fig. 6a) allows users to select the instant of the game they want to analyse. This feature allows players to control what moment in time they are observing, much similarly to what they would be able to do when using the replay system. When the animation is playing, this slider has one handler that controls the instant in time being observed in all the visualizations. When the animation is paused, two handlers are visible and the user can select an interval of time where all the events that happened during that period are shown in the map. In this mode, the position of the players in the map as well as the information in the other visualizations, corresponds to the time referenced by the handler that is farthest to the right. The solution using the two handlers was not initially conceived but, upon further discussion, it was concluded that it might benefit players who are interested in analyzing what events occurred during fixed periods of time, as well as to help users visualize the region of the map that was more impacted by certain players.

The mechanism in Fig. 6b) can be used to start, pause and restart the animation. Through it, it is also possible to view the current instant or interval being displayed, as well as the total duration of the match being analyzed. This feature, in conjunction with the slider mentioned above gives users full control of the animation being displayed in the map. The slider on Fig. 6c) further improves the control of the observed animation, by allowing users to adjust the animation speed, i.e., to adjust how fast time will advance, which directly reflects on the speed of the animation being displayed. The slider below it, Fig. 6d), allows users to control how long the representation of events is displayed on the map, after they took place, during the animation. This control was created so that users could adjust how the representations are displayed while the animation is being played, since the time needed to assimilate the information during playback might vary for each user.

The control on Fig. 6e) provides a mechanism to filter which types of events are displayed, thus, allowing users to reduce the amount of information at any given time. It’s important to mention that, apart from global events that are always represented, only events directly related to the selected players are displayed. This feature was designed as a means for users to filter out information related to some events that they might not find relevant to analyze a certain scenario.

Lastly, the controls on Fig. 6f) allow users to select which players are being displayed, thus, expand their analysis through the rest of the players. Even though, in the current state of the development, only a maximum of two players can be visualized at once, this feature was still introduced so that users can select and visualize data related to other players to extract information that can be useful in their analysis.

The information shown in the animated map is complemented with six other components that are representations of commonly used thematic metrics for performance analysis. These include: a plot with gold over time (Fig. 2iv); the player score (Kills/Deaths/Assists [KDA] - Fig. 2v); the number of minions slain (Fig. 2vi); the player’s inventory (Fig. 2vii); and a list of the global events that each team conquered, i.e., destroying towers/inhibitors and killing special monsters (dragon/baron/rift herald) (Fig. 2viii) and lastly, a table that shows the order by which players improved their character’s abilities (Fig. 2iii)). Despite varying in importance, depending, for example, in the role occupied by a player or the chosen champion, these metrics allow for the game and player state to be visualized over time and can contribute for the analysis of player performance. For instance, by observing the position of a player in relation to an event it is possible to know if he had a positive impact if his score and gold rise. Furthermore, since these visualizations incorporate the temporal component by changing as the animation unfolds, they provide a much richer analysis when compared to their static versions.

4 User study

The evaluation of VisuaLeague consisted in a user study that aimed to address the following research questions: i) studying the importance given by players of spatio-temporal information, in the context of visual game analytics; ii) understanding if and how the techniques developed meet the users’ needs; and iii) uncovering meaningful interaction patterns and analysis criteria from the users.

In the following subsections we describe the study, the results obtained and their discussion. First, we present a description of the participants of this study. Then we describe the tasks that participants were asked to execute. Finally, we describe the methodology used during the study, followed by the presentation and discussion of the results.

4.1 Participants

Participants were informed about the purpose of the study and were asked to sign a data consent form before engaging in the study. Prior to performing the tasks, the participants were asked to fill out a form with some profiling questions, e.g., gender, age and occupation. This form also included questions related to how long the participants had been playing League of Legends or similar games, what was their current ranking and which role they filled more frequently when playing LoL. Lastly, participants were also asked to disclose which applications they currently used for player performance analysis.

A total of 30 participants (unrelated to the design interviews mentioned in Section 3.2) volunteered for the study (29 male), with ages between 17 and 31 (μ = 22.23; σ = 3.14), recruited based on their regularity as players and their availability. All participants were students, with 77% of individuals studying Informatics Engineering or related areas.

All participants played LoL at least once per week, with 80% of them having experience with it, and similar games, for at least 3 years, with one mentioning having ten years of experience with games of this genre. The majority (93%) of the participants were distributed across the lowest ranking divisions of competitive play (Bronze-17%, Silver-33%, Gold-43%). In terms of player role, over 50% prefer to play Bottom lane roles (30% ADC and 27% Support), 17% were Top, 13% were Mid, and 13% were Junglers. There were no participants without a preference in role played.

When inquired about what tools they used to review and analyse player performance after matches, all participants confirmed occasionally reviewing their matches, 93% using third-party applications, such as OP.GG⁴ and LoLKing⁵, 77% mentioned using the official Match History web application regularly. Although all the participants had previously interacted with the LoL replay system (accessible from the Match History web-page from Riot Games⁶), only 16% mentioned using it regularly, which is the only application that currently displays spatio-temporal data dynamically. Despite that, none of them had any formal experienced in terms of spatio-temporal data visualization and analysis.

4.2 Tasks

Based on the feedback provided in the aforementioned design interviews, the participants were asked to execute two tasks, association, and comparison. These tasks are commonly performed by players, spectators and analysts, since, in order to evaluate player performance, it is important to understand the relation between the actions the players conduct and their outcome, and to understand how these compare to the enemy’s actions.

In the association task (T1), participants were presented with a scenario that features two players of opposing teams playing similar roles during one match. This task aims to simulate a common exercise performed by players, in which they analyze their actions and the actions of their peers, during a specific event, in an attempt to identify mistakes that can be corrected to improve their performance. This analysis is also very common in professional settings, for example, the individuals who live comment professional matches typically analyze the contribution of each player to an event to compare performances. The scenarios that participants were presented varied depending on the role that they most frequently played. This was done to help participants be familiarized, among other aspects, with the champions that players chose and the actions being displayed. During this task, participants were asked to first identify where and when certain events occurred, and who participated in those events. Following that, the participants were asked to classify, between 0 and 10, and to justify the performance of both players under analysis, based on any set of criteria they considered relevant.

This set of actions was repeated by the participants five times during this task, for each relevant event mentioned in Section 3.1:

• T1.1 - First Player death

• T1.2 - First Dragon slain

• T1.3 - First Tower destroyed

• T1.4 - First Baron slain

• T1.5 - Inhibitor destroyed

In the comparison task (T2), the participants were asked to compare the performance of two opposing players, in a similar role, during different periods of the match, namely early-, mid-, and late-game. Similarly to the associate task, the participants were free to select any criteria they considered appropriate and, at the end of the task, they were asked to indicate which of the two players had a superior performance, and to justify those decisions based on their selected criteria. The participants were also asked to classify the overall performance of the two players during that match, between 0 and 10, and justify those scores. The purpose of this task was to simulate an analysis commonly conducted in both casual and professional competitive environments, where players are evaluated and compared during these time periods to classify their overall performance during all stages of the game.

In both tasks, the criteria used by participants to justify their choices were intentionally left open with the purpose of understanding the importance given by the participants to spatio-temporal information when analyzing player performance.

During the conduction of the tasks, VisuaLeague automatically recorded task completion time and information regarding the interaction with the prototype, namely, where the participant clicked or what values/filters were selected in a non-obtrusive manner.

After completing the tasks, the participants were requested to fill out a small questionnaire⁷ regarding their experience and satisfaction towards VisuaLeague. These questions required participants to rate, and whenever possible to justify, their agreement with various statements regarding VisuaLeague’s usefulness and ease of use, its ability to save time for player analysis, comparatively to existing tools, and if/how its features are appealing to use and meet the users’ needs. Lastly, three open-ended questions inquired participants on VisuaLeague’s most positive features, limitations, and any suggestions that they might have.

4.3 Methodology

The experiment took place in a controlled environment where subjects had access to a computer that could be interacted with a mouse. The screen participants used had a resolution of 1680x1050px and was placed approximately 50cm away from them. VisuaLeague was previously loaded into a web browser with all the matches that could be used opened in different tabs. The matches were selected by the observer based on participant’s preferred role. In the first task and at the end of each subtask, the state of the application was reset to its original configuration. During the experiment, the observer sat next to the participant, with access to a laptop to take notes based on the performed actions and the comments. It is important to mention that the participants were not able to inspect the notes taken by the observer during the experiment, as to not influence their answers or interaction methods.

In this experiment all the participants carried out each task individually. At the beginning of the experiment, participants were briefed about the nature of the study as well as the functionalities of VisuaLeague. Then, participants were able to freely explore the prototype’s functionalities, so that they could get familiarized with the controls and interface and were able to clarify any doubts. The data used during this training phase was originated from a match that is different from any of the matches used during the tasks.

After this, the participants were asked to perform the association task and, then, the comparison task. To minimize possible learning effects, the order by which the events were asked in the first task was organized based on a Latin square model [3]. To validate if the participants identified the events correctly and if the justifications used were plausible, all data used during the experiment refers to matches played by one of the authors of this paper.

The following dependent variables were considered to measure the study’s results:

Task completion time :

recorded from the moment the participant was instructed to being the task until he/she gave the final answer. This variable is used to measure task complexity and the efficiency of users.

Actions :

participants performed when interacting with VisuaLeague. This covers interactions with all the controls available below the animated map.

Number of actions :

performed can be used as a measure of task complexity.

Type of actions :

performed can be used to understand interaction strategies as well as the importance of certain components of the displayed data.

Accuracy :

of the participants when identifying the events. This variable serves as a method to detect possible design flaws that impair the ability of users to interpret the information displayed to them.

The grades attributed to players by participants of the experiment, as well as the justifications for those grades were transcribed. The scores attributed to players were noted to compare the degree of coherence of participants when evaluating the performance of players. All the justifications for the grades were transcribed to extract the criteria used to justify the scores. This information is going to be used to understand the role of spatio-temporal information when evaluating player performance. Furthermore, this information will be used to understand if specific roles are evaluated similarly by different participants. At the end of the experiment, the participants were invited to answer to a small questionnaire covering their opinions regarding the VisuaLeague prototype, its features, and suggestions that they might have to improve the prototype.

4.4 Results and discussion

The following sections describe the main results obtained in the study. For a matter of simplification, these focus on the most significant results obtained, those close to be considered as such, or that may reveal important patterns. The first section is focused on analyzing the quantitative data recorded during the experiment. The second section describes the qualitative analysis performed on the answers given by the participants.

4.4.1 Quantitative analysis

To analyse the task completion times, the results were subjected to the Shapiro-Wilk test of normality. Since no valid data transformation could provide a normalized data set, a non-parametric approach was used through the Friedman test, followed by a Wilcoxon signed-rank test, with a Bonferroni correction, for pairwise comparisons. Figure 7 shows the distribution of task completion times across all tasks and subtasks.

Fig. 7

Task completion times per task and sub-task

The results showed significant differences in the associate task (X²(4) = 34.1;p < 0,0001), where participants took more time to detect when the first Baron (Task 1.4 - V = 81;p < 0,0005) and the last Inhibitor were destroyed (Task 1.5 - V = 74;p < 0,0005), comparatively to analysing the death of the first Dragon (Task 1.2). These can be justified since the events pertaining to tasks 1.4 and 1.5 usually take place at later stages in the game and often involve the participation of several players. In addition to imply the necessity of users spending more time detecting the event (particularly, if they decided to play the entire animation), these tasks may also suggest the analysis of a larger volume of information associated to the other participants of the event.

The analysis of the number and type of actions performed followed a similar methodology to the one mentioned above. Among the type of actions detected, we analysed the differences in animation speed selected by the users. Figure 8 shows the distribution of animation speeds selected with the corresponding slider (Fig. 2c) by the participants across all tasks/subtasks. The horizontal line depicts the default animation speed set by VisuaLeague (where 1 second of animation corresponds to, approximately, 1.5 minutes of gameplay, up to a maximum of 6.5 minutes).

Fig. 8

Distribution of animation speed

The results revealed significant differences (X²(5) = 15; p < 0.01), where the values of this parameter were significantly decreased in the comparison task, comparatively to when analysing the death/destruction of the first player, dragon, and tower (associate subtasks 1.1 to 1.3, with, V = 320;p < 0,0005, V = 253; p < 0,0005, and V = 331;p < 0,0005, respectively). These results make sense, since during the comparison task, participants would need to inspect with detail various events throughout the entire game, comparatively to the associate task, where the analysis was focused on singular events, occurring in shorter periods of time.

In terms of accuracy, since there were only 3 instances with participants committing mistakes during the first task, and these occurred in different subtasks performed by three different participants, no statistical pattern was identified, and thus this metric was not further analysed. The mistakes observed in the first task happened when the participants pushed forward the timeline very early in the game and the event that they must identify had happened earlier. However, everyone who made mistakes while doing the other tasks noticed that they had made a mistake and mentioned it. Furthermore, when justifying their answers, these participants noticed their error without any help.

4.4.2 Qualitative analysis

The results of this analysis address the answers given by the participants to the various tasks, their interactive strategies with VisuaLeague, and the overall feedback provided in the end questionnaire.

The analysis of the answers given by the participants was divided into two categories: the grades assigned to players and their justifications. The grades were compared by verifying if participants gave the same relative scores, i.e., if the same player was chosen as having the best performance. Table 1 shows the concordance ratios between participants. The results of the task T1.1. (first player death) were not included, since the surviving player was expectedly always considered to be performing better.

Table 1 Concordance ratios of player classifications

The results showed that on average participants agreed 82.17% of the time on who had a better performance, with the lowest agreement rate being on average 68% and the highest 93% among participants who analyzed the Support and ADC, respectively. Support players were evaluated on their ability to help other players and accumulate assists during fights. The fact that Support players are also heavily judged on their ability to control vision is a key component to this relationship, as controlling vision helps protecting teammates from being ambushed.

To analyse the justifications given, an iterative coding process was followed where two researchers conjointly created a codebook. Each researcher coded the answers given by the participants separately, during which the codebook was refined and merged. Following that, a Cohen’s k measurement was performed to determine the agreement of the two resulting codifications, which showed a high degree of agreement (k = 0.79).

Out of the various codes identified describing the justifications given, the five most common ones were: the player being far away from a specific event (LocFar); the player providing help to its team (HelP); the player being able to achieve an advantage over its opponent (AdvP); the direct participation of the player in an objective (ObjPart); and the death of an enemy (EnKill).

The analysis of the users’ criteria and their agreement reveals three important aspects. First, despite the overall high agreement across all tasks, the analysis over the support role had the lowest agreement, particularly in the compare task. As the name suggests, this role implies that the player should be performing actions beneficial to its team; however, which actions are considered as the most supportive greatly differs according to players. This particular issue goes beyond the scope of this study; however it shows that situations with a lower agreement were resultant of the subjective importance given by users over specific information, rather than VisuaLeague’s techniques not allowing them to understand it.

Second, among the most common justifications given, the majority are related with spatio-temporal information. LocFar, HelP and ObjPart are directly related with the players’ location relative to an event. While more complex, AdvP is often dependent on the remaining ones (e.g., player A being in an advantage over player B due to its participation in previous events). This is important, since it reinforces the relevance of spatio-temporal information for this type of users, even if other metrics are available, and shows the usefulness of animated maps towards their analysis over time.

Third, despite the users’ agreement, in terms of conclusions and criteria used, the way they analysed information was considerably different, and can be classified into three main groups: Narrators, Storytellers and Speculators.

The first group, the Narrators (23,3%), refers to participants who justified their answers simply by describing the observed information (e.g., This player had a better KDA but did not participate in global objectives since he was always far away when they happened).

The second group, the Storytellers (43,3%), used the information displayed to create coherent narratives that explained the participants’ point of view. They often discussed the relation between the actions and the events observed and how they affected other metrics. Some storytellers also used their previous experience with the game to further interpret the data (e.g., This player was ahead in KDA and gold. Because of that and the champions’ style as ’split-pusher’, she played more by herself, causing her to not participate as much in objectives).

Finally, the Speculators (33,3%) were similar to the Storytellers; however, they would often create complex scenarios based on what was being observed that were not verifiable through the application and sometimes some parts are speculations. Despite that, these narratives were often coincident with the actual gameplay that took place (e.g., This fight allowed this red player to destroy the inhibitor; however, since the red team overextended, they got chased down the lane and some members died. Then, the blue team over chased them leading to some of its members dying too, allowing the red team to slay the Baron).

Following this analysis, a Spearman’s Rank-Order Correlation test was run over the results to determine the relationship between the participants’ game experience (in years) and the attributed profile. The results showed that there is a strong positive correlation between these factors (r_s = 0,594;p = 0,001), suggesting a relation between the participants’ experience and their likelihood of being a Storyteller/Speculator. These results are relevant, since in addition to showing different ways in which the same information can be analysed by inexperienced users, it suggests the necessity of studying the potential design implications to deal with each group. In particular, Speculators can be the most challenging case, since rather than just studying how to show data and make their relations clear, it is important to prevent the inclusion of wrong information by the user and/or maximize the likelihood of said information being accurate. Whether this could be achieved, for example, by including additional thematic information or emphasizing the uncertainty of specific data (e.g., realistic trajectories), it remains a challenge to be addressed in future studies.

During the study, the participants used three main interaction strategies with VisuaLeague. First, after configuring the animation speed, some participants simply watched the animation playing, until they were ready to answer. Others rarely played the animation; instead, they would constantly drag the slider (Fig. 2b and i) to seek out and specific moment or period in time, sometimes also simulating the animation by themselves by dragging the slider. The last group had participants playing the animation until they decided an event required a more careful analysis. When this happened, they would slowly drag the slider to inspect the information. It is important to mention that individuals often used the filters to select which events should be observed and other players that were not necessarily the focus of the task. In addition to complementing the previous results, this further emphasizes the importance of giving control over the animation to these users, even if they may adopt a more passive behaviour.

Regarding the questionnaire, the participants showed a very positive feedback towards VisuaLeague’s features, particularly the animated map, which they were not used to, but still described it as being innovative. They mentioned that, comparatively to what they are already used to, VisuaLeague showed more useful information for their analysis. It was also mentioned, by the participants who use the replay system, that VisuaLeague was less time consuming to use and that, if made publicly available, they would use it. Furthermore, most of the participants emphasized the importance of spatial-temporal data in their analysis, saying that, because it was available, it provided a better insight into the data that other static and thematic information cannot. This feedback is particularly relevant, since despite VisuaLeague’s potential limitations, its representations were very well received, including techniques users were not familiar with, such as the animated map.

5 Conclusions and future work

This paper addressed the usefulness of animated maps for spatio-temporal data analysis by an inexperienced population in terms of data visualization, using data analytics of MOBA games, like League of Legends, as its case study.

A prototype, VisuaLeague, supporting animated representations was developed and evaluated through a user study with LoL casual players. The results suggest that, despite their inexperience, users are both capable and interested in visualizing game-related data through the use of animated maps, to solve common analytic tasks, as evidenced by the positive feedback received and the lack of any relevant mistakes during the tasks. Furthermore, they emphasize the importance of providing full control over the animation, and highlight different usage strategies with animated maps, through the use of those controls, and through different analysis profiles, such as Narrators, Storytellers, and Speculators.

Future work should be conducted to expand on these results. The existence of different analysis styles suggests that it is important to understand the causes of such differences, and the possible design implications these may bring, such as how to minimize a more speculative analysis in favour of a more factual one (through narrators and storytellers). In addition, as the results provide a relevant insight over the usefulness of animated maps, it is important to compare these with tools that players already use to study their performance. Finally, VisuaLeague focused on the display of two opposing players, only partially supporting the analysis of team-based behaviours. As such, future work should also address the animated map’s capabilities in representing team dynamics to analyse any given match. Moreover, we intend to explore deep learning techniques to address the capability to analyze more than one match simultaneously.