1. 1.

    Conceptual Artists are mystics rather than rationalists. They leap to conclusions that logic cannot reach.

  2. 2.

    Rational judgments repeat rational judgments.

  3. 3.

    Illogical judgments lead to new experience.

Sol Lewitt, Sentences in Conceptual Art, 0–9, no. 5 (January 1969), pp. 3–5.

“But then, I presume, you spoke on the premise of informing others, whilst I want you to speak with the conscious intention of educating yourself, and so perhaps both rules of thumb can be valid…”

Heinrich Von Kleist, On the Graduate Completion of Thoughts during Speech. UDN|United Designers Network, Berlin London, San Francisco.

1 Introduction

This chapter will examine objects in relationship to augmented reality (AR) to open up the territories they connote. In my exhibitions Synthetic Cells: Site and (Para)Site (2018) (Fig. 5.1) and ClownTown (2016) (Fig. 5.2), the object is inseparable from the AR experience, as they are conceived as a total artwork (gesamtkunstwerk) made up of the sculpture, the two-dimensional image, and the augmented reality interaction. This chapter might be considered one extensive footnote to the works in ClownTown and Synthetic Cells.

Fig. 5.1
figure 1

Synthetic Cells Site and (Para)Site, installation view. Photo by Ken Ek, used with permission

Full size image
Fig. 5.2
figure 2

Concept 3D Sketch for ClownTown. Photo by the author

Full size image

The art object is a sophisticated storehouse of memory that contains contextual information that helps people apprehend its nature. In this context, AR creates a complex semiotic which begins to problematize, explore, and enlarge the imaginative connection between people, systems, and things. By semiotic, I mean a jumble of signs, each changing the last's meaning, context, and framework. But AR, unlike a collage or a painted composition, unfolds in time and, uniquely, with some viewer agency and interaction.

AR can be a portal and a library of imagination and thought. It is an opera, a destabilizing mélange of subjectivities, loosely hung on the framework of their object-ness. It represents the possibility of digitizing everything: knowledge and experience; science and culture; and the consolidation of a world in which every location, target, object, and scene is tagged or transformed into a link. These links can lead you to anything, a comment, a movie, a poem, a 3D object, informational, political, psychological, and so on. AR literalizes the theory of extended cognition, where the brain does not reside entirely in mind but is part of a cognitive process that enlists the world.

This chapter is not concerned with more utilitarian uses of AR such as television graphics overlaid on news broadcasts, first-down markers in football games, or apps where cloud movements pass over geographical maps. Instead, it focuses on its use in sculptural projects.

New technologies carry with them not just tool status but broader referents. They liberally reflect the inquiries of other fields of knowledge and herald conceptual frameworks that suggest unique philosophies through their examination. One can sense the musings of speculative realism and the philosophy’s renewed focus on the object reified in AR. One could also include ideas in physics and biology. And yet, the notion of extended cognition exampled by a memory theatre or memory code, one of the oldest tools of the human mind, might be the clearest metaphor for AR. A memory theatre is a mnemonic method whereby what needs to be remembered is systematically mapped to a location, sometimes an imaginary one built in mind. In the new technologies, the memory theatre is updated to an extreme spatialization mapped upon location through digitization. A difference to the memory theatre remains in that these are not written into our brain but into the programmes that employ them.

This essay aspires to bring various forces or influences to bear as we contemplate what AR affords. It is not only a nifty technology to advertise and present information, but a deep integration of the mind into materialization. Far from being only an analogy or metaphor of the mind, it is a synecdoche in that it represents part of the cognitive process to access the more extensive process. Please note that a synecdoche is a figure of speech that uses a piece of what it would describe as a stand-in for it. “The white hairs winter in Florida”. In my imagination, the various forces working on the object and its augmentation include extended cognition, memory codes, panpsychism, atomic theory, and even computing itself.

Within the framework of a memory code, the state of the object and augmented reality are intimately associated. In sculpture, this association of object plus image plus augment becomes a rich platform of experience. It manifests from the physical to the virtual analogous to how an object is mapped to the imagination (a more organic process). The difference with AR is that you see the transformation rather than visualize it. This sculptural platform is an extension of how sculptures have been used heretofore. It implies multiple binary relationships: object to subject, actual to virtual, physical to metaphysical. But AR reifies this process and collapses into a unified yet strange experience.

2 What is Object?

The presence and trajectory of objects are vast. What are humans without objects? For the tool-building species, objects are survival and carry multiple purposes and intentions. My take on these phenomena and their relationship to technology is to tie them to the idea of valence both in chemistry and linguistics. That seems relevant because it blends natural processes with cultural experience. These thoughts emerge as a response to what has changed in the technological realization of form because of 3D printing. We need to establish how things have changed about the sculptural object.

Firstly, objects have become graphical with the advent of computer 3D design. This process of designing 3D becomes highly mediated. An object could just as easily be sets of specifications as simple geometric constructions as parametric instructions (a computer-aided design technique that refers to the use of parameters or variables that are edited to alter the result of the design.). Although not the work’s intent, Joseph Kosuth’s work “One Chair and Three Chairs” demonstrates what is commonplace to computer practice, namely that code is fluid and could be represented as easily by one thing as another. The same question is at stake, “What is a chair?” This is hardly a question in the information age as it points up the fluidity of information. It is de rigueur that context and the creation of meaning specify the representation. It also assumes that specifications—the geometric constructions and the parametric instructions—are added as one of many possible representations.

Working in 3D design programmes is already a communal act: there are other people in the design space with you. The software platform comes from teams of people making tools to design things. The design is technically complex and intimately connected with the computer environment from which it has come, an enormous mélange of operating systems and specifications, not to mention hardware (Fig. 5.3). This complexity is always true of objects, whether made of clay wood or steel, modelled, carved, or assembled. After all, someone had to harvest the clay; someone had to cut the wood; someone had to make the steel.Footnote 1 But something about the technical presentation of computers, the in-house portal to the world, made this especially piqued for a sculptor in the 1990s and more extensive.

Fig. 5.3
figure 3

Concept sketch for ClownTown, 2016. Image by the author

Full size image

Artisans may have rich recipes for making their exquisite handmade relics, but the algorithmic is a bit different. It lacks intimacy while it has scale and reach. It speaks the language of possibility while complying with modular units of code. If this is not new, it is a significant addition. Here are the phenomena of the object, fixed in space with its attendant memories, knowledge, psychology, politics, information, history attached. One can imagine these as the object’s valences.

An object is emblematic of vast information, on consideration, much more than we might expect to be “stored” there. All of these boundaries and borders are the valence of objects. These are the many energetic levels of an object, or artwork, that although invisible, are communicated by its presence. They are in the history of the time that created the piece, or the artist’s biography; or they can offer a sense about the intellectual processes surrounding the work, the poetry of the piece, or the intent. These valences lie deep in works, and they encompass all of the filters that people may use when they view a sculpture. As in atomic theory, a valence functions as the location of closure and interaction.

Valence is the path of electrons as they move around the nucleus of an atom. Different atoms have different numbers of valences that are called shells. These are also where atoms join with other atoms to make bonds. It seems an apt analogy that works of art at a larger scale have valences, their own complex sets of relationships, shells, and layers. These include material ones that form the physical presence of a sculpture as well as the immaterial, conceptual, linguistic, and energetic values. An object carries its history, valences, and all the intentions and experiences of the audience, the critics, the historians, and the maker. The legibility of these qualities gives an object its meaning and accent. These component parts act in concert to create energy around the thing. Energy is the framework created by the object, and it further informs it. The energy is both phenomenal and informational, but it is palpable. Objects are a repository for some non-physical parts of their presence.

It is also helpful to think about objects in the context of a palimpsest. Imagine a palimpsest and the valences of the object as being two aspects of the same phenomena. Using computer-aided design (CAD) and rapid prototyping processes allows artists a new way to realize dense form and thus more complex content. Using this technical opportunity enables artists to load the sculpture with opportunities. It seems a two-way street, content loaded into the object as part of a form-finding exercise and the object forming the way the content is expressed. The work and its content are a palimpsest which has this multi-valent quality.

Another aspect of the word valence is fitting. In linguistics, the term describes the number and type of arguments that a word, especially a verb, can combine in a sentence. It is drawn from chemical valence, if only metaphorically.

These are two critical aspects of the state of sculpture as an object of interaction. One is chemical or refers to its physical condition, and another is linguistic and refers to meaning within this state. In this construction, the object functions as a kind of relational meaning complex grounded in its physical being as quickly as its metaphysical state.

Despite the technological object reinventing multivalence against the backdrop of 3D printing, my first experience of this was older: Donatella’s “David” in the Bargello Museum in Florence. On seeing it in the flesh for the first time in 1993, I had a kind of extended sensory experience. I could feel the air around it buzzing. It took me over. I could move into and out of the buzzing by stepping away or closer. The buzzing was some kind of information, maybe an auditory and sensory hallucination. I could sense the myriad pages that have been written, the histories that have been verified, the speculations that had been proffered, the people places and things that have absorbed it, or that the sculpture itself had pressed upon people places and things. In short, I sensed its metaphysics. I experienced, in an extrasensory way, an augmentation, the object trying to keep up with the information or the information trying to keep up with the object. It was hard to tell what was richer. I experienced all this as the location of a complex of knowledge and memory and its presence as a celebrated aesthetic.

Author Lynne Kelly expounds on something similar in her book on Memory Codes (Kelly 2018), explaining insightfully that the English rock formations, such as Stonehenge, were memory prompts that contained essential survival and cultural information enacted by performance and ritual. She suggests that it is likely that no aboriginal culture was without a memory code where the knowledge of survival and belief were preserved in sophisticated strata committed to memory through song and ritual. Examples abound in the Lukasa memory boards of the Luba peoples (Fig. 5.6), the song lines of aboriginal Australians, the Quipo, a knotted string from the Incans, and so many more. The innovators of these codes have essentially the same modern mind that we have but cultivated in such a different environment.

Like the fish trying to understand water, we are swimming in our alloyed relationship to objects. They are everywhere, representing everything. It is also important to realize that this overarching influence touches other areas of our worldview.

Kelly’s examples demonstrate the fullness of how objects are used, and there are so many more, especially with the development of the information age.

When we discuss memory storage in objects, it might seem to pull up next to something akin to vitalism, but that is not what Kelly is suggesting. It is tempting to anthropomorphize the things around us either through pareidolia, projection or habit, if not by the tendency towards magical thinking. How do we resolve this analogy in ways that are not rhetorical or have supernatural recourse?

A key element to the object and its contemporary vagaries is the theory of panpsychism. It is the idea that all matter is imbued with consciousness, “perhaps intrinsic to all forms of information processing, even inanimate forms such as technological devices” (Harris 2019). Some scientists have started to take panpsychism seriously to unfold the mystery of consciousness. Harris continues that “consciousness stands alongside the other fundamental forces and fields that physics has sealed to us—like gravity, electromagnetism, and the strong and weak nuclear forces”. This theory seems difficult to swallow. Some people who have written about panpsychism acknowledge this in their titles, for example, Philip Goff’s “Panpsychism is Crazy, but It’s Most Probably True” (Goth 2017).

So, is a rock conscious? The bronze of the sculpture? Inflated vinyl? Augmented tablets triggered by photo markers? It is hard to imagine they are aware in the same way we are, yet one can enlist them to fashion meaning and hold memory and history. As an artist, it is possible to imagine collaborating with material to make it work. So, for now, it is worth enlisting this theory, too, as we examine the art object enlivened by its metaphysics but also with the overlays of information that AR affords.

It feels like these many methods of storing memory establish them as equal parts physical and metaphysical. The metaphysical in this use are the intentions, abstractions, mediations, and meditations that are triggered by, or stored within that object. It is curious to note that the engineers that create automatic manufacture, computer-aided design, and animation software parallel the musings of other disciplines. Jean-François Lyotard's Les Immatériaux, where theory, conceptual frameworks, and intellectual arguments meet specific practices and stagings, seem to have anticipated our current practice (Dimitra 2015). In another example, object-oriented ontology, or speculative realism, does interesting things to objects and metaphysics but indisputably reinvents our interest in them. These tendencies weigh heavy on the sculptural object and influence it deeply in contemporary practice. Enter AR.

3 What is Augmented Reality?

It would be disappointing if these notes gave the impression that objects and AR are in binary relation to one another, a sort of hardware–software construct, although it is one way to think of it. Thinking back to the atomic theory analogy, one need not go far to consider the augment as a valence of an object, but there are still other ways. If we reflect on Graham Harmon’s object-oriented ontology, everything is an object and object relations (Harmon 2002).Perhaps the elements of the sculptural platform cooperate in some “form of information processing”, translating whatever is available from one sculpture towards a photo, towards an interactive piece, and around again. Objects enact a broad array of activities, but especially they frame memory. As memory enactors, triggers of sorts, they may simply be augments. This experience is not new, existing before the advent of technological augmentation, but AR seems to reify it.

My first augmented work was Earth TV Cross (Fig. 5.4) which I later understood as the technological and physical colliding together. At the time, it stood out as an outlier of my sculptures from that period. It seemed to lay dormant all these years only to be redefined by these new experiments and the availability of AR technology as the precursor to these investigations.

Fig. 5.4
figure 4

Earth, TV, Cross, 1981 early sculpture made from compacted earth, wood, closed-circuit video. Photo by the author

Full size image

AR is the reification of a process storing rich digital associations that activate a hardware-software combination. It is a specific technology that extends physical attributes to produce a composite experience of the viractual. According to Joseph Nechvatal, “Viractuality is a theory that strives to see, understand, and create interfaces between the technological and the biological” (Nechvatal 2010). So, the technological scene as it is represented in the virtual interface can be activated by 2D triggers, 3D triggers, geolocation, surface detection, or trigger-less tracking as catalysts to load any kind of digital content into an app. In the Synthetic Cell series and the exhibition ClownTown, tablet computers with custom software designed via Unity (a game engine) and vuforia (a plugin for Unity) (Fig. 5.5) create the effect. In those exhibitions, images are used for triggers. The images are often fixed to a sculptural piece either printed on its side or affixed to different appendages of the sculpture. Then, they are programmed to link other forms of information to the viewer. The information can be interactive or fixed and present in any media that a computer, phone, or tablet can represent. Its rich interface adds information to interact with the world.

Fig. 5.5
figure 5

Walking pig in Vuforia application on a tablet computer, Synth Cell 003, China Wall, Pig, 2018. Photo by Ken Ek, used with permission

Full size image
Fig. 5.6
figure 6

Synth Cell 013, Rope, Fly, 2018. Photo by Ken Ek, used with permission

Full size image

AR has notable inferences. Firstly, it is a constructive medium. Unlike the famous (perhaps mythical) story of the Lumiere Brothers film of the train heading into the audience and the audience’s panic-stricken reaction (Grunhauser 2016), it does not present itself as natural or as a suspension of disbelief. It seems the viewer would always consider what the app offers to them is artificial. Whether it is a walking pig (Fig. 5.5), a common housefly (Fig. 5.6), or a virtual sculpture that can be gesture navigated, whatever event comes into the triggered AR app seems to be an addition to the physical reality. It is not mistaken for the real, although it may supplant or subvert it. Instead, AR appears to start with a question: what is this (the world view available in the camera’s viewfinder) doing with that (the augmented experience that composites itself over the world view in a kind of collage)?Footnote 2 That is a question that belongs to the viractual.

Despite this question, the first time a person experiences AR, something strange, miraculous even, has happened. It is digital magic as if the artist’s thoughts, their imaginings have manifested as a screen-based experience wrenched from the physical scene as rich synesthesia. Although it is hard to say how it is different from previous manifestations of the artist’s imagination and to call that new, the virtual and actual appear as strange bedfellows.

Of course, the mechanism is something more pedestrian than that. It is the programmed experience of the artist or author as they make the target of an object or a picture become a link to further information. But there is still a textural strangeness about it because it is that digital pixilated visual laid upon the flattened (by camera or viewfinder) space. But something important has happened. With its ability to link from one thing to another, for example, a word to a video, the Internet is now located in a real-world site. It is the opposite of what happens on the World Wide Web, where one can access the links from any computer. In an interesting reversal, the link has now become located to the place or the image marker. It has become a site-specific link available from the programmed view.

Even so, its strangeness is multiplied in another way by the overlay of the digital, physical, and interactive information that shows up on top of the real scene. It becomes a portal that transports the viewer to a place other than the one the viewer is immediately in. It adds layers of abstraction and unrealness—strangeness—to the experience. This is part of the constructed artificial. This sense that it is a portal is pronounced because of the illusion that there is a real scene behind the pixelated digital.

The technology also affords that this same trigger can act as a channel for other AR projects. For example, six different artists contributed augments to the triggers in Synthetic Cells Site and (Para)Site. In other words, the link may be split in to as many different experiences as people are interested in adding. This technical opportunity is fairly profound, as demonstrated in the following example. A controversial site where multiple histories overlapped can be channelled in this way to create an official narrative plus the narrative of any sufficiently interested group to interrupt the previously scheduled programming. The site of the AR work becomes a portal for multiple other experiences available to the viewer in this space. Here, AR reinvigorates real-world sites antithetical to how sites are neutralized on the Internet. It enlarges the physical site with any kind and number of experiences that are not native to it and extends the notion of a link to be further divided into other channels.

In Synthetic Cells, each augmented target became a portal to the artists’ experiences: Epigenetics, John Craig Freeman; an ASMR experience, Will Pappenheim; Lucy Trackball, a comedic character, Carla Ganis; The Monument App, Claudia Hart; Gardens of the Anthropocene, Tamiko Thiel, and Only As Beautiful as the Objects it Reflects Chris Manzione (Schneider et al. 2020).

Put against the backdrop of objects, AR seems to be the perfect complement to these developments. AR allows a dynamic semiotic that are metaphysical and transformative of the physical object or 2D target that invokes them. They give rich associations which the viewer must assemble. Regarding this instant semiotic, any sign combined with another would create this resonance, but to carry it across media from the physical to the two dimensional to the interactive is somewhat novel. This expanding dynamic allows for direct storage of experience into an object that is accessible to the viewer with the appropriate technology.

There is no limit to the types of signs that could be included, but when it transforms into time-based and interactive works, the experience becomes drawn out. The sense that you are constructing the event as you are experiencing it is piqued. This is unlike the train that drives at the audience or the suspension of disbelief. There is never a moment when you take AR for real, for biological. It seems stranger than real.

4 Patterned to Fit Work: ClownTown, and Synthetic Cells: Site and (Para)Site

So, a work of art can be made in an entirely different way. Rather than the sum of its parts or the sum of the conceptual pressures working on it, it can be a leap, an irrationality, despite residing in a framework. This is true in both of the exhibitions ClownTown (2016) (Fig. 5.2) and Synthetic Cells: Site and (Para)Site (2018) (Fig. 5.1). Each partakes in irrationality. These shows were companion shows, siblings in my exploration of site, the object, and AR. They were musings on my collaborations with different forms of objects across multiple media. Because of the use of augmented media throughout the works, these sculptures had the sensation of being n-dimensional, a science fiction sort of space.

It is also true that each of the shows treated this viractual space as a piqued space to share wonder with friends or anyone else who cares to travel along to this strange platform. It is as if I’ve invited you to an invented theatre of possibility to come along with me to see a space together, to open up to the opportunities and wonders that it offers. The platform is this extensive, expanded space of the relations of objects, whether material or immaterial, out into space, virtual space, and acting as a portal into the mind. We will get used to this just as we grew used to books. Still, for this moment in time, these digital apparitions associated with image and object are strange: unusual or surprising in a way that is unsettling or hard to understand.

What does a world of objects and their interactions with media portend? It is not the only contemporary question, but it is worthy of deep experimentation. These two shows approached the same problem from, on the one hand, a hard-to-understand obsession akin to postmodern literature (thinking of David Foster Wallace and his elaborate, end noted stories) and the other hand through joy. In joy, the contents of strange are present but restrained in favour of a more playful approach. The former would be ClownTown and the latter Synthetic Cells: Site and (Para)Site.

ClownTown (2016) was a comedic picaresque mediated by a sculptural interface. Each form contains a juxtaposition of imagery and augmented reality that plumbs aspects of Internet foolishness. The ludic tenor of the works in ClownTown points to anxious times and shifting definitions of the world, while a sense of fatalism in the face of political and economic surrealism suffuses each work. In ClownTown, the picaresque novel becomes the structure of a metaphorical house of mirrors.

The clown is an ideal subject of this type of novel, for as one contemplates the visage, one cannot help but be struck by the initial problem of the clown’s face. It is two faces: both a face and a put-on face, the human face and the painted face, the physical face and the augmented face. Because of the makeup, there is often one smiling face while the other frowns. The clown is implicated in each piece: what the clown wore, where the clown lived, what horse or donkey the clown rode, who the clown dated, what the clown said, and so on. But different accouterments appear alongside the clown in different states, further destabilizing the visage of the clown. These create ontographic aphorisms that contribute to the rakish clown and what we know of him.

ClownTown was an assembly of a natural history of materials and material culture techniques in the early twenty-first century. Forms were repeated across media as 3D printed objects, metal objects, and as repurposed interactive virtual objects that were skinned with images. A single form in different materials and medias was presented and represented, appearing here and there, and there again, but each time changed. Each time they were submitted to one or another form of computer representation. For example, the form Long Stom shows up in three different pieces: as a large plaster sculpture in “Slappy Pappy: sleeping clown (thought bubble, speech bubble)”, (Fig. 5.7 left) as a cast aluminium 3D print in “Long Stom Recursive (vintage clown pair1 and long stom with happy and sad),1 Big Bare Feet Vintage Clown Shoes from http://bit.ly/2dhtuFH” (Fig. 5.7 middle); and again as the augmented interactive sculpture, texture-mapped with a sad clown face and a happy clown face in that same work “Long Stom Recursive…” (Fig. 5.7 right). The sculptural forms are carried throughout the show contributing to the sense of a house of mirrors where the presentation of the same form appears in multiple representations through the show. “Abject Weather with Clown House and Kitty Ball” and “Abject Weather, Um and Ah, clown’s mouth, winner/loser” (Fig. 5.8 from left to right) are other examples. These objects come into and out of the clown’s life almost randomly, certainly absurdly, contributing to the destabilizing effect of the combination of these various media but consistent with the picaresque.

Fig. 5.7
figure 7

Sculptural form “Long Stom” shows up in various sculptures and augments in ClownTown. Photos by the author

Full size image
Fig. 5.8
figure 8

Abject Weather in two sculptures, ClownTown. Photos by the author

Full size image

These were purposeful juxtapositions as the show came to the exhibition in 2016 on Donald Trump’s election eve. That is a watershed historical moment for the effect of social media on the psychology of the body politic. It referred to the strange role of social media in life and how things were rapidly changed. At that moment, a person who had prepared to be the leader of the free world through the cultivation of expertise in multiple policy areas was overwhelmed by a remarkable media manipulator who designed to game media to his own benefit. The juxtapositions emphasized how the objects were just as slippery in the exhibition as language had become in social media.

This was especially true in “Abject Weather, Um and Ah, clown’s mouth, inkwell monkey head with winner/loser” (Fig. 5.8). Once the augment was activated, an antique ink well of a monkey’s head with a hinged cap opened and spun around endlessly, showing a sign that alternated “winner” “loser” in an endless gif. Visual logic often enacts the meaning of a single aspect of the work in multiple ways. So “winner”, “loser” was as much about the election participants as it was us, as it was the stakes of digital media in the thumbs-up world of social media. Artists often use a single thing to stand in for multiple implications simultaneously, and they are happy to claim them all.

Other sculptures included “Slappy Pappy: sleeping clown (thought bubble, speech bubble)” (Fig. 5.9), where the augment was a hand-drawn picture of a thought bubble inside a speech bubble plus a speech bubble inside a thought bubble. It was activated by a Technicolor representation of the sleeping clown. The augment was a presentation of the dream of the clown as the clown is dreaming of “thinking of speaking” (speech bubble inside the thought bubble) or “speaking of thinking” (thought bubble inside the speech bubble) while no words are inside of any bubble.

Fig. 5.9
figure 9

Slappy Pappy: sleeping clown (thought bubble, speech bubble). Photo by the author

Full size image

Also “Bitter Pill and Landscape Cube, Wrestlers1, Mathematical Pony2. 1Hercules and Antaeus, Lucas Cranach the Elder, 1530. 2The photo texture of the pony is a sculpture made by Michelle Ray”, (Fig. 5.10) wherein an allegory of digitalia. Hercules lifts Antennas off the ground to separate him from the power the earth gives him while a strange math object appears with the sculpture of a horse texture mapped to its surface can only be seen when the object is turned just so. At other angles, it becomes completely strange, elongated, fragmented, and broken. Bitter pill plays on the cliche phrase of a remedy you must pass through, while landscape cube is a simulation repeated across six surfaces, not of an actual landscape but a simulation. These combinations may or may not add up, may or may not make sense. As in the preservation of finitude, something is withheld—by the artist, object, facsimile of the painting, or their collaboration—never fully delivered as some part of the experience refuses to be translated. They seem to be conversing with themselves—a clownish house of mirrors.

Fig. 5.10
figure 10

Bitter Pill and Landscape Cube, Wrestlers1, Mathematical Pony2. 1Hercules and Antaeus, Lucas Cranach the Elder, 1530. 2The photo texture of the pony is a sculpture made by Michelle Ray. Photo by the author

Full size image

“Queue the music! Send in the clowns! In our farcetectural, internet of things, opinions have the same clown-nose shape form and weight and are equal to every other opinion no matter how scary or banal” announces the press release (Rees 2016). Each work continues with its internal logic, hinted at by the titles of the pieces. The titles were more like an index than titles, where each thing is mentioned and footnoted as to where borrowed images came from, unless the images were anonymous. In “Preservation of Finitude (clown1 triggers jackass2), 1Clown Torture, by Bruce Nauman, 1987. 2Internet photograph from google image search “jackass” or “stubborn pony” (Fig. 5.11), a documentary photograph from Bruce Nauman’s “Clown Torture” (during the scene that the clown jumps up and down screaming “NO! NO! NO!”) gives way to a picture of a jackass splayed across the surface of an abstract construction. It is a torturous narrative about the stubborn insistence of the artist to pursue the most arcane content regardless of audience access or understanding, antithetical to the etiquette of good communication in late-stage capitalism.

Fig. 5.11
figure 11

Preservation of Finitude (clown1 triggers jackass2), 1Clown Torture, by Bruce Nauman, 1987. 2Internet photograph from Google image search “jackass” or “stubborn pony”. Photo by the author

Full size image

So many artists were influenced by this Nauman piece that its meaning morphed from Nauman’s intent into its various receptions. It had become repurposed as it moved into its own identity, separate from Nauman. It had moved into language like the playful reiterations of children, latched onto and repeatedly turned until it was changed and inhabited by the players. As in an academic paper, it is a quote acknowledged by a footnote in the spirit of Fair Use. Many of the titles were lists of what was in the sculpture. There was the feeling that this show had been written into existence instead of moulded or made. In the sense that the entire show was in one way or another developed as some expression of computer code, this was almost literally true. But for the show's author, it felt more like the script of a movie, a play, or a novel than an exhibition of sculptures. The objects became fluid, but the augment clued us into this fluidity.

This inescapable clownish aura of ClownTown is felt as variously exuberant, silly, incompetent, abject, or grotesque. ClownTown looks into a sculptural condition stuck within a transformative trajectory that takes us from the existential to the artificial. The sculptures draw their audience through an ideational house of mirrors, deftly shuffling technologies, medias, images, and characters while playing in this serious game with one's sense of the real—Press Release for ClownTown 2016.

ClownTown was the obsessive, fractal intent to mine the ludic display of human behaviour across the Internet for artistic expression in a picaresque starring the clown. Later, Synthetic Cells: Site and (Para)Site sought to halt this critical approach and instead appeal to the viewer with joy while still retaining something strange within this sculptural platform. ClownTown made extensive use of AR as a house of mirrors to reflect upon the sculptural object in a social experience with a series of false equivalencies. On the other hand, Synthetic Cells layered up a sculptural pastorale which started from a generalized cell bred to a math object that telescoped out to animals and insects commensal with human environments. Although both had elements of absurdity and humour, Synthetic Cells were lashed together in some wild notion of the collaboration with conscious matter via panpsychism. Both shows were playful, but Synthetic Cells was consciously joyous, celebratory.

Synthetic Cells: Site and (Para)Site (Fig. 5.1) is chronicled in the catalogue of the same name (Schneider et al. 2020). As such, it is not necessary to repeat the same information here. Synthetic Cells was an exhibition of large inflated vinyl cells, many of which had a marker image on their side that enacted an augmented experience that was always an animation (the butterflies and ants and sometimes interactive the turtle, the rooster, the pig, the feet, and the fly (Fig. 5.12).)The artist-authored AR app was installed on a tablet computer held in place by a rolling tripod stand so that viewers could move from piece to piece. But it differs in crucial ways from ClownTown in its use of AR, which may explain how AR is folded into this sculptural platform. One key component of Synthetic Cells was the addition of other artists in the AR portion of the exhibition. So, the tablet had augmented apps from six different artists. From the beginning, the show started with biological metaphors as the plan was to host other artists’ work on top of the image markers to create a show within a show.

Fig. 5.12
figure 12

Some examples of animals and insects commensal with humans from Synthetic Cells: Site and (Para)Site. Photos by Ken Ek, used with permission

Full size image

There are different kinds of host–guest relationships in biology, including mutualistic, commensal, or parasitic. The parasite suggests that some harm is done to the host, although this is not true in all parasitic relationships. In the title’s reference, (Para)Site exemplifies symbiotic relationships and takes off from the roots of the word: para comes from the Greek meaning alongside, besides, near, and so on. It may have been clearer to emphasize commensal or mutualistic relations rather than parasitic. Even so, the sense of an artistic organism in a conjunctive symbiosis with another organism is what the title hoped to imply.

We might amplify this to include augmented reality too. There is undoubtedly some symbiotic relationship to its host, whether triggered by the object, geolocation, or marker. In Synthetic Cells, the root experience was the first experience, the site (and sight) of the inflatable cells. Everything extends from there. The images were subsets of the form, and the augments were subsets of the images. Perhaps the parasitic aspect was that these forms seemed to destabilize the other. Although conceived as a single piece or unit, the experiences played out over time as the viewer absorbed them. And yet, the animals and insects which made up the AR experiences were consciously chosen because they occupy shared ecologies with humans. The import of the piece was to playfully connect the trajectory of the strangeness of a math object (or protocell) to an image disassociated with it, to an augment that was a representation of an animal or insect commensal with humans.

The relationships of all the things and subjects in the exhibition, not unlike ClownTown, were ontographic, held together by a “list, a group of items loosely joined not by logic or power or use but by the gentle knot of the comma” (Bogost 2012). The extended mind and various memory codes may be a subset of this ontography. It fits with the stated intent of Synthetic Cells to be a pastorale that was located abstractly in a landscape. It seems in this time of so many layers of culture, economy, politics, and industry layered into nature, these strange sculptures were one way to approach it. They were tunnels through to nature, portals of sorts. The sculptures were created from two chambers that were in equilibrium, and the internal chamber had the feeling of a passage. Even so, the passage was blocked to maintain the double chamber stasis. Often, people expressed their desire to move through the passage and enter the sculpture.

This blockage contributed to the sense of the sculpture being an n-dimensional object. This is like a hypercube in which the various visualizations bring us tantalizingly close to the 4th dimension while it remains elusive, beyond our grasp. In an imaginative feat, the augment steps in to layer up the experience. It allows a leap to another thing, say an animated pig. It tunnels from some “out there” nature to some intimate experience where viewers can interact.

The augmented reality allowed another level of interaction in these sculptures. The app allowed viewers with finger gestures in the AR app to orbit, move, scale, and rotate the augments, create interactions with people within the picture frame, and play with one another. Depending on digital literacy, viewers had different levels of involvement with the tablet. This interactivity created social hierarchies and social roles. For example, technically savant people would easily take to the tablets and explore trial and error until they experienced the interactions completely. People more unsure of technology would watch this performance by interactors as if they were playing an instrument. Anyone playing with the tablet would often bring up their phone to photograph or video what they were seeing. Their accompanying friends often dart in and out of the scene, while the person working the tablet would form compositions then take photographs of the tableaux. People were drawn into the work in various ways, which created a diversity of experience. This was a feature of both ClownTown and Synthetic Cells.

Many of the themes of this paper may have been available to a sculptor before the information age; Donatello’s David is one example. But what has changed in digital media? Are we experiencing new things never available before digital media, or are these experiences simply recapitulations of previous models?

As a member of the bridge generation, digital media is a profound change in how we know what we know. It is an essential investigative tool of knowledge and experience. Augmented reality is one of many innovations that will inevitably and irreversibly alter how we do what we do. Still, it was the tool that brought these tendencies to bear in me. It is an extension of network intelligence as it remakes the world by linking in real life (IRL) sites to diverse channels and experiences. These tools have affected the ability to imagine new relationships across multiple and various media. The provocative questions of this technique bring a deeper understanding of consciousness and extended cognition while updating memory codes.