1 Introduction

As the genres of Augmented Reality expand, they will eventually proliferate to serve any number of cultural functions, including that of gamification. One of these is to place Augmented Reality-based art into the service of performing certain serious and purposeful tasks, such as the solution of complex visual/spatial problems, sorting, etc. while interacting with an Augmented Reality/Mixed Reality artwork. Gamification comes from this impetus in the form of taking a task/problem and subjecting methods of solutions to game-based constraints. There are a number of screen-based examples of gamified problems like the protein folding game, Foldit. Darf Designs’ Hermaton architectural installation also hints at task-based artistic Augmented Reality through its specific formal qualities of being a task-based game. Conversely, games like Pokemon GO integrate task completion in terms of gaining experience to ascend game levels, much like RPGs like Warcraft and GO’s predecessor, Ingress. But why are there not more applications/artworks that address the notion of gamification in the realm of Augmented Reality? In this chapter, the cultural effects that lead into the practices of gamification, such as STEM (Science, Technology, Engineering and Math) and STEM to STEAM (Science, Technology, Engineering, Art, and Math) curriculum, screen-based gamification will be examined, as well as the notions of digital labour and how it may be placed to the service of performing certain tasks, like education, problem solving, etc.

One of the key issues when thinking about gamification in its application to Augmented Reality Art, is that there simply is not a great deal of it to date, or at least not as much as other genres, such as screen-based games and gamified Virtual Reality. This essay, while seeking models that could qualify as part of the conversation, will also point toward future models where gamification could be applied. There are instances in which one can argue for task-based interaction in Augmented Reality Art (Darf Designs Hermaton) or certain sorts of labour (Pokémon GO). But few Augmented Reality art-based applications have instances in which tasks are “usefully” being put in services of finding solutions to a given problem defined by the game/art/experience. Most are in service to a metaphor, interact as spatial installation (as in Fantastic Contraption in Virtual Reality), but rarely are the interactions being placed in a role that is accomplishing a given set of labour that is outside the interaction with the work. For this study, works will be selected that fit general models of gamification and then that study shall expand on the possible instrumentalization of the tasks involved.

2 Considering Gamification

The core notion behind the rise of Gamification as a cultural trope comes from models of user engagement to make subjects more pleasant that are not considered pleasurable or accessible and ascribe labour to experiences that generate some sort of practical use value. To understand some of the thought behind gamification and its application to Augmented Reality Art, one must create a context for our subject, where it comes from, and frame the artwork that falls in this category. While there are multiple strategies to discuss, the idea is that there is a common thread; the invitation to engage with subjects considered difficult or mundane as a form of play, as well as where these impulses come from. It is important considering initiatives behind gamification that many of these come from U.S. educational initiatives or ideas related to increasing productivity in technology-based businesses.

For example, a strategy related to gamification is STEM, then STEM to STEAM, a set of imperatives developed by a 2006 declaration by the U.S. National Academies, and then formalized by the America Competes Act of 2007 sought to improve underperforming U.S. standards in Science, Technology, Engineering and Math. Over the first two to three years, STEM programs did little to improve flagging scores, so in 2009 the Next Generation Science Standards (NGSS) were created. The problem is, as with many U.S.-based educational initiatives, these programs have been poorly funded, with the assumption that initiatives alone would result in solutions to the problem. But as Sousa and Pilecki suggest (Sousa and Pilecki 2018), the introduction of arts-based components into the STEM programs provide inspirational methods that give context, application, and meaning to learning science and technology. In this case, STEAM attempts to place the arts in the litany of buzzwords like innovation and creativity, and place art as an incentivizer, as play does in gamification.

Even though a deep discussion of STEAM may seem peripheral to the notion of gamification of art in Augmented Reality, doing so reveals much about the cultural scaffold that evinces the phenomenology of these regimes of thought. For instance, another aspect of the STEM to STEAM impulse (the “A” denoting Art) is the preservation of arts and humanities-based curricula in an increasingly instrumentalized educational system in the United States. While STEAM as such is considered a incentivization strategy in terms of education, gamification is used in a number of modalities, including problem solving and data processing as well as education. Part of gamification’s usefulness is the creation of media which is “sticky” or invites repeat usage.

The notion of media’s “stickiness” or its habit-forming (habitual) qualities is essential to gamified media. In Hooked, How to Build Habit-Forming Products, Nir Eyal describes the mechanics behind the creation of addictive media. Eyal and Shiv discuss what they call the “Hook” model (Eyal 2014), that incorporates four elements: Trigger, Action, Variable Reward, and Investment. Where this model differs from a basic feedback loop is in the last two elements. A variable reward creates intrigue; in the case of gamified environments like the Foldit protein folding game, the task is always different and novel as the problem gets solved. This leads to the next step of investment, where the system is improved or develops based on the actions of the last action or user. This creates a sense of progressive reward as the process continues. The case of gamified task, the user gets an idea that meaningful work is being produced. Perhaps in models of gamified Augmented Reality Art there would be evidence of progress with the work regarding persistence in construction, evidence of communal contribution in public spaces (as in Membit) and so on. This is different from Augmented Reality experiences like Pokemon GO, JC GO, or Niantic in which only the player develops and the environment stays the same unless there is a global “Season” change. Non-game online environments like Second Life used the stickiness of user investiture to enhance its user experience by centering that environment on user-generated content, and the “seasonal” nature of games like Fortnite (and the sandbox nature of its Creative mode) bring a fresh take on the MMO. However, gamification strategies have also been used to critique these methods, with Ian Bogost’s Cowclicker being a prime example.

3 Stickiness and Critique: Cowclicker

Ian Bogost’s Cowclicker (Tanz 2011) was a critical game commenting on Facebook games like Zynga’s FarmVille, but could also be construed to be a criticism of gamification as a whole in terms of monetization and habituation. In it, players clicked on a sprite of a cow every six hours. That’s it. The game allowed certain actions, where players could let friends’ cows onto their pastures, collecting extra “moo-ney” for each click (Fig. 10.1). Bogost created Cowclicker for a presentation on social gaming in 2010, using it as a deconstruction of social gaming, monetization of “freemium” games, poorly designed educational games, and gamification. The fact that it went viral, reaching nearly 50,000 users at one point reveals the relevance of Bogost’s critique. Cowclicker was developed and continued until its end in 2011 one year later with the Cowpocalypse (Burgess 2018). This led an anticlimactic game to an anticlimactic end. When answering one user in that the game itself wasn’t very fun, Bogost replied that “The game wasn’t that fun in the first place.” (Leigh 2013). His reflection that monetization and social gaming being built into the Facebook infrastructure suggests intrinsic abuses of these genres through stickiness and monetization, and suggests potential abuses of gamified AR, as seen in Matsuda’s Hyper-Reality video explored later in this text. However, one application of gamification in desktop gaming that revealed gamification’s positive potential, and this is the protein folding game, Foldit.

Fig. 10.1
figure 1

Courtesy Ian Bogost

Cowclicker (2011),

Full size image

Foldit (Ponti et al. 2015) is an online game created by the University of Washington Center for Game Science, in collaboration with the University of Washington Department of Biochemistry to solve certain protein folding solutions in regard to biochemical problems. Created in 2011, Foldit is a multiplayer online game in which players know little or nothing about the actual biochemistry except for the rules of docking molecules and thus creating “recipes” which are shared with other players. In the first year, over 5000 recipes were created, and according to Scientific American in September 2011, users solved critical issues in enzyme production in enzyme sequencing in the replication of HIV viruses within three weeks (Coren 2011).

The strategies employed by Foldit use the human brain’s 3D-processing capability, and tie this to communal strategies of problem solving through online social space and represents a highly effective solution for solving these spatial problems. Research by the Foldit team suggests that the crowdsourcing of gamification evidenced itself as one of the most effective strategies for solving this genre of problems, and is more efficient than brute computation. What is important about Foldit to this writer is the application of this environment to basic research, as opposed to the direct extraction of use value for financial profit, in the case of the pay-to-win games critiqued by Bogost. This brings our discussion to the notion of labour, non-labour, and playbour in terms of digital media, the model of labour and extraction of use value would translate into gamified AR Art.

4 Labour, Non-labour, and Playbour

As gamification ostensibly deals with task completion as part of game play, the mention of the task brings into question the involvement of labour, and with that labour, the conferral of use-value (Gebrauchswert). The engagement with this subject opens a discursive space which is a continuum in game-play of vectoralist labour, non-labour, and “playbour”. This requires a reflection on the notion of games, play, labour, use value and their relationship to gamification and digital interaction, as well as extraction of value from digital media.

A good method of determining this relationship is to draw an epistemic arc from game to labour in relation to gamification. Definitions of what constitutes a game vary from Wittgenstein to Salen, all of whom define various models/families of rules, interactions, and/or conflict. However, one of the simplest definitions of a game is by Kevin Maroney, who stated a game is “a form of play with goals and structure” (Maroney 2005). Conversely, play, as defined by the Oxford English Dictionary, is to “engage in activity for enjoyment and recreation rather than a serious or practical purpose” (OED Online 2018).

Considering practicality in distinguishing gamified tasks versus games proper, which the OED connotes as connoting frivolity is also key in the notion of gamification. The mention of ‘seriousness’ as in opposition to the notion of gaming calls into question notions such as ‘applied gaming’. This genre, coined as “Serious Gaming” by Clark Abt in 1970 in his book, Serious Games (Abt 1970) and popularized by Ben Sawyer by the Serious Games Initiative in the mid-2000s (Sawyer 2007), proposes alternative problem solving through the competitive, associative, and ludic strategies of gaming. Educator James Paul Gee also championed gaming, especially computer games, as instrument of literacy. Gee reminisces that one of his first experiences with gaming in long sessions with the game Time Machine, resulted in, “confronting what was, for me, a new form of learning and thinking was both frustrating and life enhancing.” (Gee 2003, 118). What Gee was doing in the early 2000s was some of the early (not only) popular articulations of the use of computer gaming for the engagement of difficult mental labor, such as learning.

Gee’s articulation of novel forms of learning and thinking as applied to the solution of complex problems is a precursor of Sawyer’s notions of Serious Gaming. The center of the difference between Gee and Sawyer is that of interactive design, or the direction of the rhetorical vector; where Gee is learning from his new discovered environment, Sawyer is intentional in terms of proposing games that are intended as tasks for the explicit solution of problems. Learning from solving Myst or Oblivion is fundamentally different from making the folding of proteins in DNA segments, although either could constitute a challenging, enjoyable task. While one creates a space where the individual learns, adapts to, and completes difficult tasks in finishing a difficult game, gamified tasks have specific outcomes which are used for the solutions which they were designed for. The user may learn and execute difficult problems, but the important difference is that with gamification, the outcomes are applied.

Hence, the application of gamification, or its instrumentalization of play to the performance of intellectual tasks shifts the gamified space from that of pure entertainment to enjoyable “useful” task. Returning to our previous definition, the gamified task adopting the role of “practical” play leads us to the question of the use-value of the solution of that task and the value assigned to the labour to complete that task. This leads to a discussion of vectoral labour, and the notion of “playbour” coined by Julian Kücklich in 2005 in his article Precarious Playbour echoed in Wark’s thought on labour in the digital age. To consider this is to look at how degrees of labour could be valued in games such as Pokémon GO.

In the Hacker Manifesto (Wark 2004), Wark proposes the addition of an expanded model of digital capital and class dynamics. This includes the addition of two classes, the Hackers and the Vectoralists, and their respective systems of capital. In Wark’s model, these new classes exist in opposition in their relation to intellectual capital. The Hacker class is the epitome of Stewart Brand’s motto, “Information Wants to be Free” (Levy 2014). The Hacker is the generator of free intellectual property, music, code, art, data, knowledge. A key point to understanding the Hacker Class’ relationship to intellectual property is “We do not own what we produce - it owns us.” This notion of ownership places the Hackers in the position of endless Long Tail (Anderson 2004) abundance and free value, not scarcity.

Conversely, in contrast to the Hacker model of digital abundance are the Vectoral Class, or the corporatists whose goal is the extraction of value from intellectual property. Conventional examples of vectoralism are content providers like Netflix, Amazon, Hulu, as well as major dotcoms that extract value from the means of informatic production. For the sake of this discussion, it might not be useful to include social media, from Facebook to YouTube, WhatsApp, etc., as they intersect on this essay’s discussion of playbour, but do not address the specific modalities, and thus a distinction is intended. This is where a jump from Serious Gaming and game-based learning paradigms to digital Marxism when considering the gamification of Augmented Reality, Augmented Reality Art, and even games such as Pokémon GO are not such abstract leaps. To consider the differences between these concepts is linked to labour, use value, and an interpretation of Kuchlich’s “playbour”, especially in terms of deriving value from interactions with augmented content, art, or gaming. In Precarious playbour: Modders and the digital games industry (Kucklich 2005), Kuchlich initially framed the term to talk about certain practices in game culture, like modding. Modding is the practice of taking pre-existent content, as with games like Skyrim, and changing designs/substituting content to change the experience of the game. Famous examples are Doom and Quake, but others include Skyrim, where users substituted the model of the ancient dragon, Alduin, World-Eater, Bane of Kings, for a model of Thomas the Tank Engine in the game mod, Really Useful Dragons (Lambo_96 2013). So, at the point in Skyrim where the player is about to be beheaded, and the great dragon is about to appear, the “toot” of the engine is heard in the distance. Thomas appears, breathing fire, and laying waste to the village.

Kucklich’s point that this would be writ large through social media culture is that of the fans doing unpaid work for the company by developing the game further for their own enjoyment, amounting to playbour. This also happens with other fan cultures, as in the case of science fiction programmes like Star Wars and Star Trek through the generation of fan media. What results from the fans is an enhancement of the intellectual property through a sort of hacker-class labour, which creates a vectoral use value.

The next step in the understanding of playbour is that of social media, and especially platforms such as Facebook and YouTube, where user media generation is a direct example of vectoral labor creating use-value for the platform, and is extracted through advertisement revenue or pay services. Keep in mind that the model for playbour discussed here is related to the extraction of value from user content and user interaction, which may differ slightly from others’. The connection between Foldit and Bogost’s critiques of vectoralist social games like FarmVille (which offer in-game purchases that allow for a much greater ease of advancement) relate to the notion of extraction of capital.

The concern here is the ability to extract capital, whether financial, intellectual, or other from a gamified interaction, and whether this is a beneficial transaction to the designer and the user. In Augmented Reality environments like Membit, user engagement qualitatively adds value through the collective aggregation of memory through the insertions of photo-based augments in public space. But as of this writing, no extraction of value through ads or the like are evident. In social media, such as the case of Facebook, scandals like the Cambridge Analytica case (Confessore 2018) make clear that the relationship between Facebook and the user is currently akin to a sort of Stockholm Syndrome, where the user becomes so dependent upon the use of the platform, that abuses in value extraction are met with little recourse.

Fortunately, there are few or no Augmented Reality apps, art, games, or otherwise that create this sort of relation, but Google Glass, with its potentially “always on” gaze created a possibility for surveillance-as-playbour situations with Google. The issue is whether there is an ability to extract vectoral capital from a gamified task in augmented public space. Although personal spatial (augmented) computing such as the Hololens and Magic Leap are still relatively new, Keiichi Matsuda’s Hyper-Reality shows a model in which living in augmented space could result in ubiquitous extraction of labour from any form of attention. Before discussing this example, let there be a comparison between the notions of interactions and tasks.

5 Interaction and Task

Gamification implies the performance of an interaction with the media that generates a useful outcome or task. A detailed exploration relating to User Experience Design pertaining to task flows, types of tasks and decompositions are beyond the more qualitative nature of this discussion. In our case, the gamification of art, Augmented Reality or not, returns to the OED definition of play as having no purpose, and as art and games frequently have the quality of not being instrumentalized, and gamification places the gamified artwork at the service of a purpose. For now, the definition of a given interaction set to a task will be considered as a product of labour. The question is, in the case of gamification, what is the use-value of the labour?

6 Tasks and Labour in Augmented Reality Games/Art, from Screen to Space

Gaming certainly has genres based around fictional labour; Blizzard’s World of Warcraft has levelling that players “grind” through, consisting of killing ten Razorbacks (or whatever monster, for that matter) to get a certain amount of experience points. This is straightforward labor with a given use value which converts into exchange value in the form of experience points. When this is translated into AR-based art, whether gamified in terms of education, monetization, or knowledge production (among others) brings into question what models can be used for progressive social models of gamification in augmented space. For this discussion, we will consider some Augmented and Mixed Reality experiences (a mix of game, art, and design) as possible examples, then look at two near-future fictions that take critical stances towards gamification of augmented space.

Pappenheimer, et al.’s skywrite (Pappenheimer 2012) employs a collective experience of skywriting in Augmented Reality for the potential signalling of desire for political change. Users using the app write their desires in the sky (Fig. 10.2), ostensibly to communicate them to others. But what other uses could it have? For example, in the case of natural disasters such as the massive fires in California, could this technology be used to mark areas under that part of a sky? Or, drawing a more direct line to gamification, could image recognition be used along with the skywrite technology to crowdsource meteorological research, or verify the accuracy of weather prediction? Suggesting the instrumentalization of an activist app not the intention of Pappenheimer and company, but on the other hand, using the produced data could have positive social benefit as just mentioned.

Fig. 10.2
figure 2

Courtesy the artist

Skywrite (2011), Pappenheimer, et al.

Full size image

Membit, a photo-based geolocative public app created by Jay Van Buren et al. (Hills-Duty 2017) allows users to place photos of moments at certain places and spatial orientations, or the placement of an image in space. The user goes to a place on the Membit map, takes a shot, and the app records the image, place, and also orientation of the image. To recall the image, the user accesses a channel or searches the area for media, and then reorients to the image in space to the point that it reconstitutes. This is a fairly unique use of Augmented Reality in that it largely creates asynchronous experiential windows into the past of the site. For the AWE 2016 conference, Van Buren placed banners depicting Augmented Reality pioneers in the Santa Clara Convention Center, suggesting illustrative uses for the platform.

Taking this spatial/photographic form of Augmented Reality into consideration, gamified Augmented Reality art utilizing this schema could include education, wayfinding, or possibly even landscape notation. For example, Van Buren’s initial example of documenting memory resembles a scavenger hunt, suitable for historical documentation, with the potential for basic scoring, competition, or research-based collaboration. This documentation could record longitudinal studies of change over time in a landscape, which could be useful in times of climate change, in cases like sea encroachment on coastines due to climate change. In some ways, this software’s application could reflect that of skywrite in sharing relevant information about the landscape. However, Membit’s application to models of stickiness like Eyal and Shiv’s concepts of variable reward and investiture seem less clear, with variable reward being the most difficult to resolve, as the problems may be generated by user-created contexts. The next example of spatial Augmented Reality that could adapt to gamification is a spatially-oriented puzzle, Darf Designs’ Hermaton.

Hermaton, a “buzz-wire maze” 2013 game from London-based Darf Designs (Lichty 2014), creates a gaming environment that is designed to fit inside the built environment. Hermaton is located by large-scale printouts in the room being used as augment targets for a large, multi-faceted machine that players have to activate (Fig. 10.3). Players of Hermaton move a digital ball through a maze, activating parts of the machine that are part of the installation. Parts are machinic, with intricate mechanisms in the modules to others that are more about the movement of the ball. Hermaton, as seen at the Augmented World Expo 2013, was an example of an excellent architectural visualization with a game component, but still limited itself to that of the game format and did not extract any performance metrics.

Fig. 10.3
figure 3

Courtesy Darf Design

Hermaton (2015).

Full size image

If turned into a gamified model, Hermaton could be instrumentalized in several ways. One is a form of physical therapy, where movement of the ball is done to accompany movements, and progress could be quantified longitudinally through multiple trials. A user could be seen as perhaps moving the ball though the space slower at first, then measure cognitive function as they learned the space, or require the user to reach high or low. Another could be a training aid, where the Augmented Reality conceit can be an orientation device, helping the user learn some spatial task, such as dis/assembly, which is one of the first applications of instrumentalized Augmented Reality. The use of Augmented Reality linked to a device in the hand and pushing the need to move the device in ways that challenge the body or the conception of interaction could be a key aspect of gamification. One other spatial puzzle/device game that hints at gamification strategies is Fantastic Contraption. Although Contraption is originally produced as a Virtual Reality game, it could be deployed as an Augmented Reality title in platforms like the Magic Leap headset.

Fantastic Contraption (Allain 2017) is a room-scale physics-based Virtual Reality game in which players use various materials to create machines that overcome certain obstacles (Fig. 10.4). This could range from navigating a room to moving a pink ball to a wall. To accomplish these tasks, the player has only a few materials: a cylindrical motor, a wooden beam, and a balloon beam. From these objects, one can make robots, vehicles, even trebuchets (for the ball). Like other physics- based games (like World of Goo), the solutions can vary, but all have a fairly well-defined range of solutions based on the physical properties of the constituent parts. Motors turn, but can reverse when placed in an opposite orientation, wooden beams give hard machinic linkages, and balloons stretch, cushion, and can even be popped. YouTube documentation has even shown Fantastic Contraption as a Mixed Reality experience (Romaine 2018), giving this author the notion of considering it as applicable as a gamified Augmented Reality STEM educational experience.

Fig. 10.4
figure 4

Courtesy Northway Games

Fantastic Contraption (Mixed Reality),

Full size image

The educational/STEM possibilities of Fantastic Contraption are clear when placed in an Augmented Reality context, as it deals with creating physical solutions for machinic problems. Manipulating machines in physical Augmented Reality contexts makes sense, but where the environment is defined in the case of Virtual Reality, the actual context of the surrounding environment must be taken into consideration. Several options as to how to contextualize the task are evident, with two main decisions having to do with using the physical environment as a context for the solution, or merely using an open space. In the latter case, the game can work as in the Virtual Reality version, but when taking the physical environment into consideration the situation becomes more complicated, when considering gamified STEM education. For example, does the exercise limit itself to certain structural constraints (“Find a wall and a clear area of floor”), or will it place itself into a dynamic context with the surrounding environment and define problems on the fly using the parameters of motion, height, etc.? Secondly, how would metrics (score) be determined in such a case; would points be assigned for speed, height/number of obstacles, number of elements, etc.? This use of context-driven Augmented Reality for the solution of gamified physics problems would present infinite challenges but would also present difficulties in ascertaining metrics.

Expanding on the physical construction notion, Mel Chin’s Unmoored (Selvin 2018) has little to do with gamification per se in an overt sense, but points at elements of it. In this piece, onlookers use a Microsoft Hololens to observe a Times Square transformed by climate change. Boats sail and bump around one another in a flooded Square. Created in partnership with UNC Asheville’s STEAM Studio students, Chin created this work, and Wake, a large-scale animatronic sculpture of singer Jenny Lind and a 20-m ship hull situated at Broadway Plaza (Fig. 10.5). While not indicative of gamification in themselves, the piece’s situation in this context suggests gestures towards education, simulation, and construction. The use of spatial computing in large public areas like Times Square and Broadway Plaza prove the viability of public augmented computing. It also shows the potential for gamified interaction with the larger constructed human environment. While this is a speculative scenario, the potential for smart city paradigms/Big Data combined with crowdsourced interpretation of this data could spell a symbiotic human/AI relationship in urban spaces, and this could be a place for a larger conversation. On the smaller end, LEGO Augmented Reality Studio offers direct potential for application in this study.

Fig. 10.5
figure 5

Courtesy Mel Chin

Wake+Unmoored, (2018)

Full size image

Lego AR Studio (Kobie 2017) presents a direct but unrealized opportunity for gamified learning in Augmented Reality paired with the proper conceptual frame, this application is a logical progression from Fantastic Contraption. The ARKit-based app allows for scenarios to play around various constructions. However, LEGO AR Studio (Fig. 10.6) consists only of simulated blocks and narratives surrounding them. For example, users can play with pirate ship, a hospital, a robot, a man on a dragon, a police station, a truck and a train. Different combinations cause various behaviours, such as moving the dragon overhead causing people underneath to move around. If this app were tied to construction and AI, to track blocks and act as a play partner of sorts using automated strategies of LEGO construction (Kozaki et al. 2016), the app would be far more interesting, and block tracking metrics could offer insights in cognitive science and spatial construction for LEGO building.

Fig. 10.6
figure 6

Courtesy Lego

Lego AR Studio,

Full size image

Given the theoretical considerations and examples which point towards a gamified AR/Art, apps like LEGO AR are glorified advertisements for the plastic bricks. However, there are short films which clearly indicate an Augmented future, and the argumentative stretch here is for that of gamified Augmented Reality Art is in the form of science-fiction based speculative design stories. Two short films, Hyper-Reality and Sight offer dystopian insights into the future of ubiquitous AR, with gamification being central to living under this particular ontology. As with many dystopian Sci-Fi near future stories, technology is seen as the threat to individual freedom.

7 Synthesis: Sight and Hyper-Reality

Keiichi Matsuda’s 2016 short, Hyper-Reality (Matsuda, 2016) depicts the logical extreme of gamification of Augmented Reality in a video artwork. It depicts the journey of a hapless Medellin, Colombia teacher turned courier, Juliana Ostrepo, and a day in her life. As we enter the first scene, she is on a bus playing a cat-themed matching game, earning “Loyalty points” (the currency in this scenario), when she is called by her AI-based boss, the Job Monkey Motivation Guru (Fig. 10.7). She plaintively asks that as she is trained as a teacher, isn’t there a better job than being a shopper? He informs her to trust the app, it always gives the best jobs. She swipes him out of sight in frustration, then considers wiping her online account in disgust before deciding that she would lose all her loyalty points in doing so.

Fig. 10.7
figure 7

Courtesy the artist

Hyper-Reality, by Keiichi Matsuda.

Full size image

As Juliana’s day progresses, it shows more social commentary on the future- Augmented Reality scenario. She is harassed by a hacker, where she turns to the grocery store’s AI service department for help. They restart her device, briefly revealing the bleak fiducial marker reality that Medellin has become, and her headset restarts. However, her AI helpers show her a blue line that leads to a biometric identification center. As an aside, at this level of sophistication, her device not having iris scanning is a plot hole… She follows the blue line into the street, only to find that it’s all been a hack. She is confronted by an attacker who is cloaked in augments and stabs her in the hand. Apparently, this allows the hacker to steal her biometric information, and her device is wiped clean. Despondent, she looks around as her now defaulted device restarts, and she sees a shrine to the Virgin of Guadalupe, asking her to make the sign of the Cross and become Catholic. She does, and her life begins anew, with the irony being that the Catholic app is run by Bully Entertainment, the same one that ran all her apps before the attack. Such a commentary hints at the ubiquity of corporate interests in online space and evolutions of their algorithms in their manipulation of users’ desires and even lives. Jaron Lanier points this out in his 2018 interviews and articles on how corporate algorithms are evolving and shaping our desires in real time (Kulwin 2018). Would the gamification of public glasses-based Augmented Reality result in the commodification of our conscious time, as Facebook and Instagram do for so many, as Matsuda seems to suggest?

Arguably the most direct media-based depiction of gamified Augmented Reality as everyday life is the short film Sight (May-raz and Lazo 2015) by Eran May-raz and Daniel Lazo. Sight depicts a day in the life of Augmented Reality developer Patrick.s, a systems-level programmer at Sight Systems, a fictional company that has created Augmented Reality audiovisual contact lenses. As Sight begins, Patrick is playing a more Virtual Reality-like flying game in an almost empty room, leaving an important point as to whether our future display devices will be Augmented Reality, Virtual Reality, Mixed Reality, or merely changing based on the context on whether one’s eyes are closed. Secondly, the idea that decor and all but basic living structures will be depicted question whether a Sight-like augmented experience could result in decreased consumption of consumer goods, optimization of food use, etc. as we will see. Patrick’s day continues, more interestingly when he prepares his lunch using an app called ChefMaster, which encourages the player to optimize cucumber slicing and fry an egg based on gamified models. When he mis-slices a cucumber, he sweeps away the vegetable, to “restart the level”; showing that the program suggests wasting the cucumber rather than taking a point deficit. What would a gamified life like that which Sight provides be like; would it create positive social effects like decreased material consumption, or also rote adherence to the normative models of gamification? Would you be a better chef, or just a really good cucumber slicer/egg frier?

The dystopic notes in Sight’s vision of Augmented Reality gamification increases as he prepares for his date with Daphne (Fig. 10.8). His helper app, called Wingman, helps him dress for the occasion, and then using AI, context and voice recognition, does everything from suggests conversational subjects, food choices, and relational timing. When Patrick finally gets Daphne home, they sit and toast what Patrick calls “a perfect night”, as Daphne notes his augmented decor. This is when she notices his game board, and more importantly, his score marker for the Wingman dating app. Realizing she has been merely a pawn in a sociopathic dating theatre, she gets up furiously, and turns to leave; Patrick, However, has other plans. Daphne’s concerns about Sight Systems actually creating mind altering implants proves true, as Patrick commands “WAIT!” and she stops mid-stride, as command-line interfaces pop up in Patrick’s display, implying that he has just hacked Daphne at the root (deepest) level. This is also similar to the Black Mirror episode, “Playtest” (Seara 2016) in which the protagonist of the episode is also controlled by a neural/Augmented Reality interface. The tired trope of media of technology’s control of the individual is un-needed in this film as Sight shows a number of harbingers of near future scenarios.

Fig. 10.8
figure 8

Courtesy Robot Genius Films

Sight (2015),

Full size image

In the case of ubiquitous gamified AR, what would the ultimate outcome result in as far as a user experience is concerned, and how would this translate to art, or even design? In the case of Matsuda, corporate driven AI agents might seek to extract every second of attention, day or night, being a not-so-subtle metaphor for social media. This might be done by gamifying everything at subtly layered means, from amusing ourselves with tasks when not engaged with a pressing task, to trying to suggest purchasing/patronage choices when involved in “useful” work, and by translating physical jobs to fungible game points. In the universe of Sight, every task and aspect of lie is quantified and gamified, from monitoring the contents of one’s refrigerator to using predictive AI to seduce prospective mates. Although for some an environment that is “winnable” might be preferable as global media depicts a more unstable existence, Sight bluntly makes its social critique evident in suggesting the medium being more than merely the message, but an instrument for direct neural control. In either scenario, the metaphor for gamification being a metaphor of control by hegemonic/capitalistic scaffolds is a bit of a tired trope, but is also a pointer at the recurrent dreams of the abuses of power extant in current social media and a warning against the same in ubiquitous AR.

8 AR, Art, Gamification, and Formalist UI Design

After a theoretical consideration, examples of potential models, and speculative design scenarios, what are some possible explanations for the relative absence of the genre, that is, gamified Augmented Reality Art? In that Augmented Reality is certainly an artform by 2019 is proven in many texts and most notably in the Geroimenko’s previous anthology (Geroimenko 2018). So, given that Augmented Reality is a long-standing genre for art which is expanding into the 2010s, what are some rationales for the lack of gamification in this area? I will argue a small amount in terms of novelty, but also paradigmatically in terms of the creation of the form, availability, and its terms of expression. This may sound slightly Greenbergian in terms of formalism/minimalism regarding art, but in terms of Augmented Reality and art, the notions of essential form may be crucial here. This study will conclude with suggestions on formal user interface as aesthetic experience that could be taken into creative Augmented Reality applications.

The novelty of AR’s popularization/democratization as an art medium is certainly a consideration. Although practitioners like Berry/Poupreyev have made expressive Augmented Reality in the 1990s with works like Augmented Groove (Lichty 2014), the issue is more of the critical mass of production stemming from the expansion of core technologies for handhelds in the late 2000s. Platforms like Metaio, Hoppala, and the free sections of platforms like Layar allowed for a great deal of creative exploration in the late 2000s that was not available before in public space. With the buyout of Metaio by Apple in 2016, and the increase of paid services on other platforms, Augmented Reality as art slowed slightly until platforms like Unreal Engine, Vuforia, and ARKit allowed greater freedom once again.

Secondly, and it is arguably the technologist’s excuse, Augmented Reality as an art form is less explored than some, and relatively speaking, is a younger genre than Virtual Reality. This is a response in other genres like virtual worlds like Second Life, that ‘We’re still in our adolescence”, which, after 20+ years if the social virtual platforms Traveler Onlive and ActiveWorlds are considered, is a position which increasingly cannot be supported. But if we see the emergence of creative Augmented Reality in the late 90s to the expansion of the field in the 2010s with groups like Manifest.AR. Therefore, while a great deal of work has been done in Augmented Reality -based art, the genre is new enough that sufficient maturation has not taken place to allow for the exploration of things like gamification to any large extent, and this text points toward its potential.

Lastly, one of the issues with the creation of Augmented Reality-based art is that it is a form based on a fundamentally new form of Human-Computer interaction (HCI). For example, a seminal example of spatial computing, the Microsoft HoloLens, uses the surrounding architecture as a extension of the desktop metaphor as a place to place 2D browsers. While this essay lacks scope for the exploration of the UI/UX of spatial computing, it is this writer’s contention that we simply have not developed spatial paradigms of computing. The evidence of this is that the Microsoft HoloLens, although masterful in its use of gestural computing, still uses 2D browsers and menuing. Likewise, the Meta and Magic Leap interfaces still use “Billboard” interfaces with icons. In short, if we are to operate in 3D, our operating system interfaces need to be 3D.

In popular culture, a couple visionary User Interface designs point at what immersive interface design could emerge in Augmented Reality. The movies Johnny Mnemonic (Longo 1995) and Minority Report (Spielberg 2002) both proposed haptic interfaces for the embodied manipulation of data and could be used for interfaces in space for gamification solutions similar to that of the Foldit game. The difference between the two interfaces is notions of the UI; for Johnny Mnemonic, the UI is still set into quasi-physical metaphors such as a triangular widget, keypads and so on. On the other hand, the Minority Report interface, developed by John Underkoffler at MIT, deals with the problem of “Real World Geometry” (Underkoffler 2014) in terms of a practice of process-based gestures, much as the Apple interface paradigm expresses itself not as a technology, but as set of gestures. Underkoffler’s gestural interface translate to general models for the manipulation of informatic objects. This reduction of UI to sets of gestural processes for the manipulation of experiential objects could be the spatial paradigm for ubiquitous Augmented Reality interfacing using hand tracking without the need for a “air-mouse” as with the Magic Leap and Oculus Go.

In the late 1990s, UK-based interactive designers, including Roy Stringer and Roger Harnden, and later Danny Brown through the firm Amaze, and Stephen Holtzmann in the USA questioned the two-dimensionality of the World Wide Web though radical UI designs. Noodlebox (Digitalarchaeology 2011) was Brown’s attempt at describing online informatics in terms as a set of nested cubic boxes that would express the spatial structure of an online space’s informatics. A unique part of the Noodlebox interface was the fact that the interface components were moveable, allowing the user to reconfigure the structure of their interactive space to suit their personal organizational style. Navihedron, (Hutton 2014) imagined by Roy Stringer, would expand on this metaphor by using nested sets of platonic solids to structure information by arranging along the lines of edges and vertices on each solid, and allowing each vertex to be an interactive node for the expansion of a new solid/topical space. Steven Holtzman, creator of the Perspectaview hyperbrowser (Dunn 1998), allowed for a hierarchical tree of information to be flown through in space, similar to Ted Nelson’s Xanadu interface. These technologies would generally scale back to forms of mindmap/hyperbrowser platforms Thinkmap and The Brain.com, which structured themselves on traditional spider-like mind-map cognitive maps.

9 The Dependence on Space

One last issue that constitutes a challenge to Augmented Reality-based artists is the dependence of Augmented Reality to the geometric context of the physical environment. This is a matter of conceptual frame as can be seen through examining the spatial computing frame of the Microsoft HoloLens, and then of a brief text by the American author Terrence McKenna. Perhaps a challenge to a variable reward and investiture (Eyal 2014) may be related to the use of variable spatial context and persistence of media in that environment, which is preserved in the interface methodology of the HoloLens. When activated in a given space, the HoloLens uses structured light to create a scan of the area, and places holograms, browsers, etc. within the user’s view which is congruent with the user’s spatial field. Each of these spatial databases are stored, and then accessed as the user re-enters the space, with the persistent elements of the workplace in position. This dependence of computational space to the environment constitutes a physical context that is meaningful to the given space and paired with the proper problems/content a site for solving meaningful/useful interactions like spatial puzzle solving, object recognition, and the like. This leads us to consider novel forms of meaning construction though task completion in AR.

Considering a speculative model for a gamified space in Augmented Reality which would create a unique gamified experience, the writer Terrence McKenna gives a model in the radio programme, Virtual Paradise (Earwax Productions 1992) in which he discusses a real-time form of concrete interaction in virtual space. Virtual Paradise was an experimental documentary by Earwax Productions in San Francisco featuring luminaries of the time (Terrence McKenna, Brenda Laurel) who were part of a 24-hour event focusing on Virtual Reality. During this documentary, McKenna spoke about his ideas on concrete language in Virtual Reality. His Virtual Reality (Augmented Reality) “fantasy” (sic) is about structural elements of language, such as the grammatical elements, and defining them as geometric elements, having colour and form in space, forming a “tinkertoy-like construction” constructing itself based on the rules of grammar and syntax. He argues that “language is a topological manifold, a set of interacting rules that come down to being perceived as a surface.” (Earwax Productions 1992). He wonders then whether his idea is more than a linguistic gimmick, in that if you could concretize language into a form of post-symbolic communication, you could swap one’s point of view, and actualize virtual empathy. As with many of McKenna’s brilliantly seductive arguments, his description of this scenario as some sort of revelation of Chomsky’s universal “deep grammar” is probably a gimmick when thought of practically. But when considering the arbitrary, subjective nature of language, there is no universal sense of subjectivity. However, when taken structurally, if we take this schema, assign rules, apply rules of speech-to-text, as well as rules of construction in synthetic space, McKenna’s fantasy, in a headset like the Hololens or Magic Leap, could be a model for persistent speech-driven concretist sculpture in virtual/augmented space. Placed within the scope of other rules (types of object/time, structural analysis) could be a poetic game of sort, and therefore subject to gamification. Closing this discussion of Augmented Reality art as a small genre, the proposition for realizing McKenna’s fantasy as a sculptural game of concretized language is a provocation for the development of the form.

10 Conclusion

Although the genre of gamified Augmented Reality Art is small as of 2019, my contention is that it will come. This is not to say that gamification in art or creative media does not exist, or certainly not that Augmented Reality art or design exists. And games such as Pokémon GO or augmented star charts with recognition games all have the potential for the generation of use value from augmented interaction, whether for analysis, education, knowledge generation, or monetization. Given the sensitive nature (e.g. Bogost along with Wark’s models of labour in the digital age) of the possible abuses of monetization, my thoughts for the futures of gamified Augmented Reality art are centered in the other three categories. The uses of gamified Augmented Reality Art, as discussed above, could have ready entry points in STEAM education, visual coding, design, spatial problem solving, play, therapeutic experience, and environmental notation/information sharing, as mentioned above. The issue, as Matsuda asks in his film, is what value/meaning is being generated, who is it useful for, what are the questions being solved/asked, who owns the value, who does it benefit, and who controls it. These are only technical design issues, but social design issues/human factors. And most relevant, what is the artist’s role as investigator of gamified space, and who is their research partner, again asking the questions above. The examples we have explored, from Fantastic Contraption to Membit, and on to the media fantasies Sight and Hyper-Reality, offer models and warnings about the future of augmentation and how they could/might be applied to our subject. And lastly, the basic issues of interface, whether entirely spatial as in the case of Mel Chin’s work (intrinsic to the HCI design of the HoloLens) to the potential for object recognition that LEGO AR Studio hints at hints at. This, when thought forward through spatial interface metaphors developed by Stringer, Brown, et al., one might develop potentials for objective interfaces or the embedding of meaning in visual structure such as those suggested by McKenna. The issue for this writer is the meaningfulness (rather than usefulness) in the experience and the value of interaction with the gamified AR artwork.

Using the language circumspectly, the potential usefulness of gamified Augmented Reality art and the uniqueness of problems it could solve is evident from the relatively short histories of both genres. However, the question is whether to do it is as an exercise of the instrumentalization of art in Augmented Reality, which would be a disservice to the notion. It is my hope that when this genre of interactive media arises its usefulness can be directed that are meaningful, inspirational, helpful, and a tool in the aid of the current global condition.