Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

FormalPara Take Away Points:
  1. 1.

    It is vital to discuss the varied backgrounds and epistemological biases for each of the game development disciplines with stakes in game analytics.

  2. 2.

    Understanding the different expectations, potential benefits and requirements that each type of stakeholder brings to the table when discussing game data analysis is required prior to establishing an efficient analytical strategy.

  3. 3.

    A number of concrete examples are provided for each stakeholder involved in an ideal analytical strategy.

1 Introduction

The process of creating a game is a perfect example of a multidisciplinary effort. Development teams are extremely heterogeneous, involving writers, programmers, artists, designers, engineers and architects, just to name a few. Each discipline brings a legacy of assumptions, practices and values that, more often than not, are far from aligned with each other. For example, designers often discuss with programmers in order to modify existing features or introduce new ones based on their past experiences, theoretical assumptions or “gut feelings”. User researchers often act as mediators in these transactions, and starting these discussions based on objective evidence and telemetry data has proven beneficial. It is, therefore, necessary to understand and chart the different possible benefits, biases, angles, needs, costs and requirements that each discipline brings to the table when it comes to game data analysis, since a thorough understanding of these facets can considerably help in defining a strategy to select interesting game variables to monitor.

In this chapter, the principal stakeholders who might have an interest in game analytics are listed and examined in light of their background, the expectations and requirements and potential benefits that game analytics can provide and finally, where relevant, examples of game analytics are given based on the content of this book.

  • Background: who are the stakeholders, what are their main tasks and goals.

  • Expectations and requirements: the underlying needs of different stakeholders, the limitations in terms of output and input: what data, information and format a certain stakeholder requires before starting any analytics and the outcome that each stakeholder expects from game analytics practices.

  • Benefits: how analytics can improve the performance of different stakeholders.

  • Example: a case study that will be explored in detail in this book.

2 Game Designer, Level Designer, System Designer, Game Director

  • Background: Designers are probably the most heterogeneous group of stakeholders; they have a wide variety of possible backgrounds from architecture to film studies to computer science. But they share one important trait: they are all responsible for creating the design and mechanics for the game.

    When dealing with game telemetry and analytics, designers often generate very specific hypotheses that need confirmation or falsification to fine tune and balance elements of the game from weapon damage to navigation flows. In these cases it is fairly straightforward to pinpoint the relevant game variables, transform them into metrics and extract features. Often game directors and lead game designers are also interested in exploratory analyses aimed at creating models of player behavior.

    Addressing a request for precise answers, Microsoft Game Labs pioneered telemetry-based systems to support user testing of Halo 3, generating metrics-based analyses of player progression and heatmaps to answer concrete questions from designers (Thompson 2007; Kim et al. 2008).

    Until recently it may have been slightly challenging to convince designers to make use of analytic intelligence to corroborate their decisions, these days more and more designers have learned to appreciate the edge given by precise information; an important challenge for designers in the future could be to avoid growing completely dependent from analytics practices but to maintain a certain trust their in own vision.

  • Expectations and requirements: designers often expect to use analytics to improve the design of the game systems they are developing, specifically in terms of gameplay, navigation and balancing issues. Analytics are required to provide designers with low-level and detailed information about player behavior, such as interaction with mechanics and other players. Another important application that transcends a single game is tracking players’ preferences in each installment of a franchise for long term development of that franchise.

  • Benefits: to designers, analytics provide a great way of fine-tuning player experience, provided they have a solid initial vision of what experience the game should elicit.

  • Examples: in Chap. 14 a plethora of hybrid techniques are utilized by designers to gather insights, in Sect.14.3, a case study is presented that shows how designers leveraged analytic intelligence to improve the distribution of ammunitions for the game Kane & Lynch 2: Dog Days (Square Enix, 2010). Additionally, in Sect. 14.4, a further case study showcases how analyzing the different cause of death allowed phenomenological debugging for Fragile Alliance 2, the multiplayer experience of Kane & Lynch 2. Another analysis of causes of death in Tomb Raider: Underworld (Eidos Interactive, 2008) is presented in Sect. 14.5. Chaters 18 and 19 showcase several examples of design intelligence gathered through visualizations, Chap. 17 focuses on spatial analysis, while Chap. 12 demonstrates how data mining can be used by designers to assist designers in balancing the game and discovering game design issues.

3 Producers and Project Managers

  • Background: producers and project managers are concerned with managing people involved in development and assessing people’s performances and production schedules; it is therefore necessary for producers to have very precise estimates for all sorts of tasks. Beside individual estimates given by each team member for each task, producers and project managers also take advantage of automated tracking of processes and pipelines: the metrics of interest are mostly related to turnaround times for new content, barriers and slowdowns on the development pipeline, time for each production iteration, etc. (Mellon 2009). However, defining which metrics to collect in order to monitor production processes is not straightforward since evaluating people’s performance often involves qualitative assessments; the problem is amplified when dealing with artwork since it is very difficult to create objective metrics that take into account art directors’ evaluations of different assets and their level of polish and quality.

    A secondary area of interest for producers is the relation between production costs and player use: any feature, mechanic, system, character or asset in the game has a cost; its acceptability is relative to its centrality to the core of the game, its visibility/accessibility and its use by players. So producers can easily assess a game element in terms of its cost in man/hours and how used/visible it is.

  • Expectations and requirements: producers expect to use analytics to maintain better overview and control of production schedules. Executives want high-level summaries on production status and game feature use, in order to better control the scope of production.

  • Benefits: for these stakeholders, analytics provide a speedy, more informed way allowing them to make production decisions regarding personnel and resource allocation.

  • Example: Chater 15 contain an interview with producers Aki Jarvinen (Digital Chocolate) shedding light on how management can better maintain control and set goals for production of digital games, directing where to focus the team efforts and how to distribute resources.

4 Marketing Managers

  • Background: Marketing managers are mostly interested in monetization, leveraging both traditional and alternative revenue streams, such as micropayments, pay per play, freemiums, etc. They are specifically interested in telemetry that tells them: who buys what, how often, where and when. The most notable efforts in this direction were spearheaded by Zynga, where dashboards are included in several games allowing rapid A/B testing, leading to marketing-driven development models (Pincus and Gordon 2009). Other developers and publishers, such as EA, are investigating ways to monetize gameplay analytics and intelligence and are actively researching into player retention for their sport franchises (Weber et al. 2011).

  • Expectations and requirements: marketing managers expect to use analytics to increase game and micro-transaction sales and they require an analytics system that provides instantly updated information about every in-game or in-store transaction.

  • Benefits: for marketing managers, analytics is valuable as it can pinpoint piracy very early and act rapidly to prevent it. Furthermore, it can provide an insight towards which game items and features are more lucrative. This intelligence is widely used to plan new content according to specific demographics and psychographics.

  • Example: Chater 5 presents an in-depth discussion of Zynga’s practices related to analytics and marketing, while in Chap. 24 Simon Møller of Kiloo discusses a co-production model that aims at capitalizing on experiences and strategies from game to game to maximize revenue streams for the mobile market.

5 User Experience Research

  • Background: Traditionally user research departments have adopted qualitative approaches as part of their practices given how they are mostly concerned with subjective, soft concepts, such as player experience, satisfaction and engagement. Therefore, phenomenological debugging is carried out through interviews, think-aloud sessions and thorough observation. Recently there have been several success stories of qualitative and quantitative triangulations: where game data analysis aided user researchers in pinpointing soft, subjective states, such as frustration (Sørensen and Canossa 2012). Game telemetry and analytics are starting to appear as the perfect companion to other existing methods, such as usability testing (measuring ease of operation of the game) and playability testing (exploring if players have a good experience playing the game), to offer insights into how people are actually playing the games under examination (Thompson 2007; Kim et al. 2008).

  • Expectations and requirements: game user researchers expect to use analytics to help augment the current user research methods and to give researchers a clear and detailed indication of what the player is doing at any given time. Researchers require analytics systems that can provide them quickly and easily with detailed information to aid them in their investigations for the perfect player experience. Large-scale datamining provides information after a game has shipped; hence, purely quantitative methods are applicable only to games that can be easily modified such as web games and MMOGs. User experience research for traditional console titles requires a fast turnaround fitting the various iterations and stages of the production cycle. In fact, only a handful of studios can afford the massive beta tests with thousands of players that are necessary to gather statistically significant datasets. Most developers rely on a much more limited number of testers, between few dozens and few hundreds. These numbers are rarely sufficient to justify heavily quantitative analyses. Furthermore, qualitative methods have proven successful in gauging subjective emotional states such as elation, frustration, fear, etc. It is therefore necessary that quantitative game analytics blends smoothly with existing practices employed in user research departments.

  • Benefits: for game user researchers, analytics provide very clear and concise clues which can be used to show what players are doing; these clues can support the conclusions that researchers draw about problems in the game. Even though interviews and other methods provide valuable insights, it is often beneficial for user researchers to show numbers to other stakeholders in order to prove a point or to easily convince designers of the necessity of the changes suggested. Furthermore, several examples in the book show how triangulating qualitative and quantitative methods can provide answers to questions that could not otherwise be asked.

  • Example: Chater 12 showcases several examples of a user research manager cooperating with academics to address specific questions asked by designers for a number of different Square Enix games. Chaters 21 and 22 discuss several examples of game user researchers working on triangulating telemetry with other measures and the discussed benefits and successes.

6 Community Managers

  • Background: community managers are in charge of leveraging the community of players around a certain game. Usually they are dedicated fans that acquired professional formation as communication experts and have extensive knowledge on how to leverage diverse communication channels to benefit a certain game, brand or franchise. They keep under control the social connections between players, organize events and challenges, try to maintain high level of interest in a game and sometimes handle customer support requests.

  • Expectations and requirements: community managers expect information regarding top performers, opinion makers, play styles, achievements and social relations between players. This information is crucial to grow brand awareness and maximize game lifecycle. This type of stakeholder is usually concerned with high level, highly aggregated and abstract statistics that can give an insight on how to increase player retention (Weber et al. 2011). Social networking features have also become key areas for viral communications, while forums have provided for years valuable intelligence about the community’s attitudes.

  • Benefits: Game telemetry has been widely used by game studios to build and maintain communities. For example, Valve was among the first to make available to their community highly aggregated data about player behavior (Steam Games Statistics). Bungie took it a step further by not only providing aggregate statistics about large number of players, but also presenting each player with personalized reports (Bungie Games Statistics). Community managers can harness this information to create ad hoc challenges, promote and reward certain behaviors and maintain overview of the community’s attitude towards a game. Often, benefits to community managers translate directly into benefits for players, as seen by the rapid diffusion player’s dossiers.

  • Example: Sections 18.3.2 and 18.3.3 presents interesting case studies of visualizations produced by third-parties or players that can be utilized by community managers to explore and exploit social networks.

7 Third Party Data and Analytics Providers

  • Background: Outside the companies that actually develop computer games and other forms of interactive entertainment, a rapidly increasing number of third-parties have emerged in the past few years to provide analytics-related services to companies. These form two broad categories: consultants and middleware providers. The first category specialize in advising companies on how topics such as how to develop an in-house analytics solution, training of staff, advice on what aspects of performance, processes and users to track and how to analyze the resulting game intelligence (see Chap. 13), and other topics that a company might have use for. The middleware providers supply the actual software and sometimes infrastructure needed to build an in-house analytics capacity, this commonly through one of four channels: (1) The provider constructs an in-house infrastructure and supplies the necessary software; (2) The providers deliver a software package that permits the game developer to collect data, analyze them and develop reports. Usually the developer licenses the software. The vast majority of middleware providers in this category are focused on user telemetry, including typically in-game behavior, purchase records, user accounts, and sometimes more exotic data sources such as forum post tracking. The coverage and quality of analytics packages currently on the market varies substantially. Examples include Tableau and other business intelligence packages. (3) The providers deliver software-as-service (SaS), in which case the game developer uses the middleware system to design hooks in the game code, which transmits player behavior telemetry to collection servers, from which the developers can interact with the data via an accompanying UI. This is one of the currently most popular business models among small-medium sized developers. Usually payment is scaled according to traffic, e.g. number of messages. As with the licensed software option, the input data sources accepted varies from product to product, with current SaS-companies generally focusing on Free-to-Play (F2P) games, i.e. supporting the tracking of player behavior and user purchases, as well as allowing for the importing of fourth-party data, for example click-streams or user account information, in order to permit funnel analysis, marketing channel analysis and similar (see Chaps. 4 and 12). Some companies provide an option to choose between SaS or licensed solutions. At the time of writing there are relatively few middleware providers solely dedicated to games. Examples of middleware providers include HoneyTracks, Game Analytics, Games Analytics and Playtomic. (4) An entirely different third-party option, commonly used by players, are performance tools such as P-stats, which provide a small application that is installed on the local client, which transmits personal play behavior data to a database, which the player can interact with via a web-based interface. Usually these solutions are open, i.e. everyone can see the statistics of all members of the network, and thus forms excellent tools for e.g. Battlefield clans to keep track of member performance.

    Typically the people behind third-party companies are professionals with experience from large-scale data storage, database construction, programming, game data mining, game analytics, information visualization, game design and game development.

  • Expectations and requirements: Middleware providers have no expectations or requirements from the data feeding analytics, being usually focused on delivering data to the game industry instead. However, third-party consultants will have requirements that match the analysis they are expected to run. For example, if asked to analyze the in-game economy of a massively multi-player online game, the consultants will need telemetry on various aspects of the flow and aggregation of financial resources in the game world system. Usually, if third-party providers are granted more access to data this results in deeper, more comprehensive analyses.

  • Benefits: The immediate benefit to a game company for using third-party providers to help with game analytics – as with in any other consulting/middleware solution – is that they avoid the requirement for building expertise and/or tools internally, saving considerable resources. How much and what depends on the type of solution chosen.

  • Example: While there are no chapters dedicated exclusively to third party providers of analytics middleware or consultancy companies in this book, Chap. 10 describes a basic RestAPI-based telemetry system, and Chaps. 12 and 17 discuss in brief the role of third-party middleware providers for supplying the logging, transformation, storage and access to telemetry data, as well as varied degrees of analysis and visualization. Darius Kazemi, interviewed in Chap. 11, was President of Orbus Gameworks, one of the earliest game analytics companies.

8 Programmers

  • Background: Different programmers have different concerns. For example, engine programmers might be interested in frame rate, memory loads or compiling errors, while network programmers are more concerned with server stability and hardware performance. A common trait for all the different metrics needed by programmers is that they belong to the class of performance metrics, which are rather straightforward to collect and analyze but require a communal effort. Concerns regarding the complexity and cleanliness of the code are more difficult to address since it is often a subjective measure and, as such, harder to quantify. Mellon described the process of collecting performance data from automated testing sessions in The Sims Online (Mellon 2009).

  • Expectations and requirements: If hardware and software performance metrics, such as server performance, data archiving and bug tracking, are ­represented in orderly, logical manner, understandable by anybody in the team, then it is possible for everybody to have constant access to the specific game build status. Programmers often require detailed information about the symptoms of a run-time or compile time error that has occurred in order to diagnose the problem and fix it. Thus, they would require tools and methods to show details of actions that led to the specific errors. Telemetry can be used to give a clear indication of actions performed prior to the error event. Visualizations, such as heat maps can show where in the level crashes have occurred. Other programmers are more interested in efficiency issues, and thus benchmarks are important as well as a picture of what the context is like when specific delays happened.

  • Benefits: a coordinated effort allows programmers to tackle eventual problems as a team and not as individuals; enhanced coordination results in faster operating time.

  • Example: In Sect. 18.3.1 it is shown a system that clearly addresses the programmers’ need to maintain overview and coordinate efforts during bug tracking. Chaters 6 and 7 also dwell further on these cases.

9 QA Manager and Testers

  • Background: This category group of users are concerned with the number and severity of bugs: the benefit of integrating metrics tracking systems with existing bug tracking systems is that it would help in figuring out how to replicate a bug if all game variables were tracked prior to the manifestation of the bug. Only in this case it would be recommended to indiscriminately monitor all features, and only because no analysis has to be performed on the dataset, just a review of the metrics events that preceded the manifestation of the bug.

  • Expectations and requirements: testers often require knowledge about bugs: how and when they occur. Thus, they would require tools and methods to show details of the bug that occurred and the actions that led to the specific errors that can be relayed to programmers. QA managers and programmers have similar requirements in terms of bug tracking, but programmers expect more aggregated reports from QA managers.

  • Benefits: comprehensive and automated tracking of in-house QA testers greatly increases efficiency

  • Example: Chater 7 presents more information on these cases with a case study based on SkyNet, a system developed at Electronic Arts.

10 Third Party Services for Players

  • Background: third party services focus on aggregating data from several games with the purpose of providing added value to end users, players at large, by offering recommendations, matchmaking or social networking features not for a single game but for a large number of titles developed and published by companies that, more often than not, are in competition with each other. These services analyze game data to understand player behavior and connect players. The challenge is then to homologate often inconsistent data from very different datasets, for example Metacritic (2001) is forced to normalize game scores from a number of reviews that sometimes do not even provide a numerical value.

  • Expectations and requirements: leveraging the capability of bringing together information from different sources, these services can provide players with precious information about the games they play, their own play experiences across different games or compare their experience with their peers. Considering how dis-homogeneous the datasets can be, it is required to enforce some form of consistency within each dataset, whether it is review scores, achievements or raw data such as play time.

  • Benefits: utilizing these services players have access to a wealth of information not always provided by developers, across several games ranging from summaries of stories, characters and mechanics, to scores, achievements, progression, on-line presence of friends and detailed data about their – and their friends’ – purchase history or play experience such as time spent with each game.

  • Example: Section 18.3.2 provides several examples of third party services.

11 Players

  • Background: players are the end users of the game artifacts. Developers devote considerable resources to provide players with analytic tools in the form of statistic tracking or player dossiers, and they keep improving them, for example the WOW armory (reference) has been through a number of iterations. Yet, dedicated players, using APIs provided by developers, often build their own tools for exploring game data either because they want to expand on the limitation of the tools provided by the developers or because there are no tools available. An example of the first case is seen with replays of Starcraft 2 and the player-built tool SC2Gears, while an example of the second case is Minecraft and Minecraft X-Ray (see Chap. 18).

  • Expectations and requirements: players desire to use game data to be informed about their performances, and to augment their gameplay experience and would greatly value easily accessible APIs and developers’ support.

  • Benefits: by creating additional tools, players project their own cultural habits, modifications and ideas on to their gameplay

  • Example: Section 18.3.3 provides several examples.

12 Conclusions

These are just a few of the types of stakeholders that collectively contribute or affect the development of a game. There are many more, and although they do not figure in this book, they will take an active role in shaping the discipline of game analytics; for example localization producers already expressed interest in understanding exactly which parts of the game were used by foreign language testers, even writers, concerned with the clarity and intelligibility of their texts, are making active use of player metrics – which dialog choices are the most popular – to revise different drafts of their lines.

As we have seen until now, each group of stakeholders requires a different feature set. Some require data to be collected during production, others after launch and others again during the whole lifecycle of the game. As we will see in Chaps. 7 and 18, multimodal, dynamic reporting and visualization systems seem to have been able to address the varied nature of expectations and requirements set forth by all stakeholders on the development side. In fact, drill-down reports can supply both high-level information to executives and at the same time generate detailed reports for team leads from design to programming; eventually each single developer has the opportunity to generate individual reports with very different foci: for example, mapping the impact on performance of a certain shader, or the player interaction with a certain system or the navigation flow on a certain portion of the level. No matter the efforts that developers devote to ­provide the community with tools to explore their play data, there will always be space for both third party services and player-created tools to bridge the irreconcilable gap of information that competition between different publishers inevitably leaves open.