Keywords

1 Introduction

The design, development, and creation of games, serious games, VR, AR, immersive worlds, etc. arguably follows one of two approaches – using methodologies from the design and analysis discipline of Human-Computer Interaction (HCI), and practice-based approaches typically embraced by the arts, creative practitioners, and in games and immersive VR studios. While there is some crossover, these two groups largely co-exist independently.

HCI has a long and successful history in iterative design and development, user/player-centered, participatory-design, prototyping, and analysis, evaluation, and assessment of interactive and digital media. While HCI has long looked to the arts to inform methods, and includes creatives, performers and artists in HCI design and development teams, the arts and creativity within HCI typically is incorporated within a traditional engineering and technical design and development iterative cycle.

Practice-based is another successful approach that has a long history in creativity, creative arts, design and making cultures and disciplines. Practice-based approaches are part of a larger family of reflective practice and action-based approaches following Schön’s “The Reflective Practitioner” [1] which includes disciplines outside the arts such as, healthcare, medicine, and education, etc. At the heart of action-based approaches is an action-reflection cycle. In the arts, this is iteratively performed in a cycle of making and creating, evaluating/reflecting-in-action, and informing making, and so on. Although creative, making, and practice-based approaches have typically dominated the games industry, and despite the games industry being one of, if not the most profitable interactive digital media industry, over several years there has been several criticisms, and calls for more formal approaches and tools to support and inform game development practitioners in game studios/industry. The advantages being to better support large-scale projects, keeping them on-track, within time and budget, and so help to reduce costs.

As Katherine Neil identifies in her 2016 Gamasutra article [2] “How we design games now and why”, that over the years, numerous calls have been made in the games industry to address the making-focused approaches, and make available more games design tools, methods, and formal approaches to support games design and game design thinking in the games industry. For example, going back as early as 1999, game designer Doug Church proposed the need to develop ‘formal, abstract design tools’ in his well-known article published in the industry’s trade magazine Game Developer [3]. Later, Raph Koster (2005) drew our attention to “the imprecision of natural language as a tool for designing gameplay”, and proposed a graphical notation system in his well-known Games Developers Conference presentation “A Grammar of Gameplay” [4]. While game designer Dan Cook (2007) disparagingly said games design occurred accidentally through habit and guesswork [5].

One thing is clear, that as game and serious games designs and projects became larger, the design documentation also correspondingly became larger. This increases the difficult in managing and keeping track of projects. We’ve come a long way since these early days with a host of standard texts from writer-practitioners on games design [e.g. Ernest Adams, Tracy Fullerton, Chris Crawford, Katie Salen and Eric Zimmerman, David Perry, etc.] used throughout the world, to introduce, educate, inform and enlighten game practitioners and students (of games design courses and degrees), and researchers, academics and students of HCI and interaction design interested in learning about the craft of game design. In particular, more recently one driver for the HCI communities comes from the strong interest in gamification – applying game elements and characteristics to digital, interactive, online, apps, and services, etc.

Similarly, in the emerging discipline of serious games we witnessed early calls for serious methodologies and design approaches incorporating both research and development to address the spate of published work focused on development and practice alone [e.g. 6, 7, 8]. While more recently we’ve seen an escalation in reports, publications, workshops and conferences focused on serious games design and thinking, and in particular focused on mechanics in serious games these typically adopt HCI focused approaches over art, making, creativity, and practice-based approaches.

Herein, it is argued that both approaches have advantages and could inform and inspire, and feasibly be incorporated with one other. Interestingly, at the heart of both HCI and practice-based approaches is an iterative approach. In HCI, it’s the iterative design, development, analysis, evaluation and playtesting cycle; and in practice-based, it’s the iterative making/creating, evaluating/reflecting-in-action, and informing making cycle. So for example, it’s not difficult to imagine a practice-based creative iterative approach incorporated within an HCI iterative design and development cycle, and vice versa. This paper describes an approach that informs both the HCI design and development cycle and an action-based/practice-based approach for the design and creation of games, serious games, VR and immersive environments. This paper outlines the latest work of an on-going long-term sustained research effort to extend and operationalise Leontiev’s (1981) original hierarchical activity theory framework [9]. The framework described herein describes initial work that builds on and extends earlier work to provide a tool, notation, grammar and building blocks to represent gameplay mechanics and narrative, scenario, story-based activities; from game design concepts and ideas, through modelling and analysis, to informing implementation.

This paper is organised as follows. In section two, related and previous work to extend and operationalise Leontiev’s (1981) original hierarchical activity theory framework is described. Section three describes the notation, grammar and building blocks of the extended framework, and provides examples of serious games, VR and immersive environments gameplay mechanics and narrative, scenario, story-based activities represented through the framework. Section four concludes the paper summarising the work and advantages of the framework.

2 Previous and Related Work

Previous work building on the hierarchical activity theory based framework and approach [10,11,12,13,14,15,16] has made considerable advances to extend and operationalise Leontiev’s (1981) original activity theory to model and support the design, development and analysis of serious games, virtual environments and virtual reality for a variety of purposes along the serious games continuum [17], as shown in Fig. 1.

Fig. 1.
figure 1

Serious games continuum: showing from left to right, games for purpose (with traditional game characteristics) to experimental & experiential environments for purpose (with minimal game characteristics), as follows: Oceans We Make [18] third activity with game characteristics with purpose to collect plastics in the ocean; The Reef Game [19] game-like to slow down or stop harmful human activities to the Great Barrier Reef (second activity); VR Slow Reef Experience [20] swimming through the beauty and wonder of the Great Barrier Reef (first activity); Oceans We Make [18] first activity preparing to go for dive in the ocean

Full size image

While the previous work [21] developed a framework using activity theory, and interestingly draws on and mixes concepts and frameworks from the two main activity theory systems of Engestrom (1987) [22] and Leontiev (1978) [23], the focus of this work is exclusively on educational serious games. Other earlier activity theory work similarly focusing exclusively on learning games is for example [24]. So while several other published work has explored activity theory and games and serious games, invariably this work focuses on [22] version or hybrid of [22] and Leontiev’s later and revised works [23], and focuses on learning or educational games. Interestingly, the English translation of Leontiev’s [23] second publication appeared in print before Leontiev’s [9] first publication which was translated from Russian to English and published in 1981.

Our activity theory work focused on Leontiev’s [9] original activity theory containing the powerful activity concepts of motive and objective that define “activity proper”. Our earlier work, referred to as HABS (Hierarchical Activity-Based Scenario approach) focused on interactive narrative, scenario and story-based activities, composed largely of written description or statements of actions/play, similar to a script/scriptwriting, although structured accordingly in a hierarchy of activity, actions, operations, and intention/motive, directed towards objective.

The framework’s lens-like ability provided a way to describe and represent any level of abstraction from high-level descriptions of activity to zoom-in to any low-level of actions, sub-actions, sub-sub-actions, and so on, and use of tools and artifacts responding to conditions of action. In [10], we presented the first-steps demonstrating the flexible and powerful ability of the HABS framework applied to the learning game “2020Classroom” (NSF-funded) to zoom-in to any level of detail and complexity, and study results demonstrated the mechanism to analyse, assess and reason about the success of purpose (for undergraduate students to learn about processes of human organs) – through objective outcome coinciding with motive. Further refinement to this work was presented in [13].

While this earlier work focused on task-based single-activity and multiple-actions we further extended Leontiev’s (1981) activity theory to incorporate experiential-based activities through the outcome of objective merging with motive, as well as task-based multiple-activities and actions [14] performed within a Sphere of Engagement [16] corresponding to Huizinga’s “play-grounds”, “arenas” and “magic circle” [26]. However, the focus of this earlier work on narrative, scenario and story-based activities didn’t capture or extend well to gameplay mechanics of loops of interactive play activity/actions that is key to reinforcement of an idea through repeated action [27, 28] and that typically characterise games and games for purpose. Considering the serious games continuum [17] that captures all purposes from all sectors irrespective of platform, it can be seen that our previous work focusing on interactive narrative, scenario and story-based activities, was less effective at capturing or extending to games for purpose with game elements, characteristics and mechanics identified predominately on the left-hand side of the continuum.

Therefore, this called for the next phase in the evolution of the Leontiev’s [9] activity theory, to further extend and operationalise our activity theory-based framework and approach to capture core and other game mechanic loops of play activity/actions that typically characterise games and games for purpose, as well as, represent interactive narrative scenario story-based activities.

3 Extended Activity Theory Framework: Notation, Grammar, Building Blocks for Mechanics, Narrative and Story

In this section we present the initial work that builds on and extends earlier work to provide a tool, notation, grammar and building blocks that informs both HCI and practice-based approaches to represent gameplay mechanics and narrative, scenario, story-based activities. The framework is intended to facilitate and inform HCI transition from requirements, ideation, design to implementation, and helps practice-based approach transition from ideation, design docs to creation. It’s intended to support all delivery platforms and extend to all purposes/sectors (education, health, esports, business, documentary, tourism, social impact, culture, etc.) across the serious games continuum: from games for purpose to experiential environments for purpose.

As demonstrated in our previous work, blending both mechanics and narrative/story provides powerful strategies for design of serious games and immersive environments for purpose. For example, as demonstrated through our exhibited, showcased and published works: The Reef Game [19] and Oceans We Make [18] which incorporate both gameplay mechanics for purpose and experiential environments for purpose, and the VR Slow Reef Experience [20] that incorporates an experiential environment for purpose. As these serious games and immersive VR environments extend across the serious games continuum (Fig. 1), we focus on these examples in this section to demonstrate the extended activity theory framework.

Following Koster (2005), at the heart of our framework is notation and grammar that we have developed. Notation systems are found in the arts, dance and music. As shown in Fig. 2, examples include, the Labanotation which is a visual notation used to record and analyse human movement and the staging of dance and design of choreographed movement sequences, and Musical Notation used to visually represent music played with instruments through the use of symbols and other signs such as for durations, rests, tone dampening and sustaining. In our framework, the notation is derived from extended activity theory concepts and games and serious games design elements and characteristics.

Fig. 2.
figure 2

Notation Systems: (left) Labanotation records human movement sequences for archive and choreography of dance, and (right) Musical notation represents music through symbols and other signs for durations, rests, tone dampening and sustaining.

Full size image

In the next section we describe the extended activity theory framework, notation and grammar for narrative, story and mechanics, and provide a number of examples including the exhibited, showcased and published works: The Reef Game [19] and Oceans We Make [18] which extend across the serious games continuum (Fig. 1), and the experiential environment VR Slow Reef Experience [20] to demonstrate the versatility of the framework.

figure afigure afigure afigure afigure afigure afigure afigure a

4 Conclusion

This paper presents on-going research to extend and operationalise Leontiev’s (1981) original hierarchical activity theory framework. This is the latest work of an on-going long-term sustained research effort. In particular, the main focus of the work presented herein was to extend our activity theory framework to mechanics and narrative, scenario and story-based activities performed in sphere of engagement equivalent to Huizinga’s “play-grounds” – from high-level to low-level. The framework herein provides a tool, notation, grammar and building blocks that can be incorporated into both HCI and practice-based approaches. The framework is intended to support all delivery platforms and extend to all purposes/sectors (education, learning, health, science, esports, business, tourism, social impact, social justice, cultural heritage, etc.) across the serious games continuum: from games for purpose to experiential environments for purpose. To highlight the versatility of the initial work to extend the activity theory framework, we described several examples of serious game and immersive VR for purpose developed in research projects, and in the professional immersive content creation Singapore-based studios of Warrior9 VR.