Keywords

1 Introduction

With numbers nothing is impossible. Modulation, transformation, synchronization; delay, memory, transposition; scrambling, scanning, mapping - a total connection of all media on a digital base erases the notion of the medium itself. Instead of hooking up technologies to people, absolute knowledge can run as an endless loop. (Kittler 2012: 31–33)

Computation and computational media have a rich history in the meandering march of the arts and artworld. Computational technologies have long been ubiquitous within our daily experience, and they are now deeply embedded within all avenues of culture. They have become tools of our intelligence and at the same time, computational technologies, through extension, have increased the powers of our intelligence. Like all technologies, they have increased our abilities to manipulate our environment and understand it simultaneously. One has only to consider quantum mechanics or the Voyager 1 probe to get a glimpse at an example of how this particular tool has expanded our capacities to touch, smell, taste, see, and hear our reality(ies).

We are profoundly affected by this new extension of ourselves. Therefore, due to the nature of the work of an artist, even if an artist only works with analogue or non-computational/ traditional media, the artwork is informed by and influenced heavily by computational media. I am focused now more broadly on computational media because it is the foundation on which artificial intelligence rests. AI is at least an enhancement of computational systems, technologies, and processes, and at the most represents the goal, of which the development of computational media is the stepping-stone, on the road to developing an extension of human intelligence.

In this chapter the AI model used in creative practice is my own stepping-stone to think about the aesthetics and structures of creative processes in working on the 432Hz project and others, as well as how these projects highlight a framework for a foundational language of creative AI which affects these approaches to the creative process and the relationships between creators, technologies, and the resulting works. This book, and this chapter, is an inquiry into how AI has altered our theoretical framework in the arts as well as to explore the properties or the language of creative AI.

1.1 AI and Artistic Experimentation

In this section I will explore the language of creative AI through artistic experimentation and processes of building and interfacing with artificial neural networks and generative deep learning models. Namely, an audio-visual performance piece titled 432Hz that is an experiment in building deep artificial neural networks to calculate, train, and tune numerical expressions of computer-generated sound waves. This project represents an iterative process that explores one way in which artificial intelligence (AI) can be embedded in creative practice. This project and others in the field offer a look at unique aspects to creative practice where AI is embedded as a medium for making or where the AI system becomes the art object. First, we will cover some background on the machine learning (ML) developments as they relate to creative fields and then explore ideas relevant to cognitive systems more broadly before returning to the questions raised by the confluence of AI and creative practice.

1.1.1 Topologies

Topologies in mathematics and in networking are the description of a structure or object in space made from points that have an underlying logic. However, this logic and structure are highly variable and contingent. When exploring this field, and looking at these topological objects and spaces, it becomes apparent how these spaces are a construct that can be easily morphed, stretched, pulled, enlarged, squashed, and expanded. In other words, these objects and spaces are tested and highly contingent, and to a certain extent, they represent non-unique, modular possibilities much like in contemporary architecture and the built spaces we occupy, where “cookie cutter” homes and apartments serve as our spaces for existing. For example, consider building projects like the Hudson Yards project with modular, climbable sculpture The Vessel in Manhattan. If we apply these concepts of topological properties of constructed space to examples in visual arts, we can use the example of Sol LeWitt’s artworks such as his wall drawings or sculptures.

Through my practice, I am interested in these morphologies and contingencies. In my own art making where I utilize data as a media, I have begun to think a lot about how nature and our artificial extensions of nature are expressed and transformed through numerical data and mathematical expressions. At its core, an artificial neural network or deep learning model is nothing more than a huge array of numbers being computed repeatedly. It is this “digital base” which becomes a starting point of which I propose as a lens in considering what are the properties of computational media and subsequently, creative AI.

1.1.2 Convergence as AI

The convergence of technical media, that is evident in the introductory quote by Kittler above stating the “total connection of all media on a digital base,” comes to fruition through the translation of computational systems to simulate our own cognitive systems—this, I believe, is artificial intelligence. This scaffolding is realized at its core through the application of mathematics as means for the control of nature (and natural processes) as exhibited primarily through the harnessing of the electromagnetic spectrum and electrical current to compute bits in the form of electrical pulses in a CPU or across a circuit (i.e., through computation or which is down to its core frequencies of electrical pulses). Put more simply, in the context of this text, what is the language of artificial intelligence (AI) in my creative practice?

Finally, in pondering this question, I will use theories of embodied cognition and nonconscious cognitive systems to provide a model for creative practice as the creation of enacted, embodied meaning or aesthetic experience through numbers as exemplified in the performance piece 432Hz. Through this piece, my practice, and the making of aesthetic experience, numbers are expressed through sound frequencies and are then “tuned” by the machine (AI) over time by way of playing or performing the machine (as instrument or medium). I will use 432Hz as a model for a contingent and embodied experience of feedback loop between artist and machine, and through this case, numbers (or AI) become the medium for an aesthetic object or experience. Through the work, the artist’s playing of the AI instrument is an exchange that generates an aesthetic experience, an exchange between artist and audience, and thus the work arises out of two cognitive systems exchanging sensorimotor feedback and operations.

1.2 Converging at Artificial Intelligence: The Successor to Computation

As I previously mentioned, the long history of the development of media is a history of the development of the expansion or the extension of our capacities for seeing and hearing (as well as our other three main senses) through technical means. This history or development does include not only the advancement of technical media, but also includes, and is represented by, the development of how we produce knowledge (and our perception of our own brains, bodies, and selves) more broadly.

As Siegfried Zielinski states in his comprehensive archeological unearthing of technical development, the history of our technical media (and production of knowledge) can be compared to geological deep time (Zielinski 2006: xx). Against Zielinski’s urgings, I like to compare this history to a root system of a tree (sort of a flipped family tree of media) where the farther back we look, the more diversity in technology we see. Zielinski uses the history of geology and the evolution of the Earth as a starting point to begin to think about a concept of deep time for technical media:

Earth’s history could be explained exactly and scientifically from the actual state of the ‘natural bodies’ at any given moment in time, which became known as the doctrine of uniformitarianism. Further Hutton did not describe the Earth’s evolution as a linear and irreversible process but as a dynamic cycle of erosion, deposition, consolidation, and uplifting before erosion starts the cycle anew. (Zielinski 2006: xx)

Using this example of “geological deep time,” Zielinski applies these concepts more broadly to think about the history of the human species and its progress through technology. He urges us to draw a different picture of progress, and that from a paleontological perspective, a picture or metaphor of progress represented with models of “simple to complex” or tree structures should be rescinded (Zielinski 2006: 5–6). Rather, from this deep paleontological position, Zielinski reminds us of the branching diversity found as we look back on nature’s and our own technical progress:

From this deep perspective, looking back over the time that nature has taken to evolve on Earth, even at our current level of knowledge we can recognize past events where a considerable reduction in diversity occurred. Now, if we make a horizontal cut across such events when represented as a tree structure, for example, branching diversity will be far greater below the cut—that is, in the Earth’s more distant past—than above. (Zielinski 2006: 5–6)

Against Zielinski’s better judgment, I will adopt the tree metaphor to visualize this concept of the “deep time.” What Zielinski is describing here could be thought of as a root structure where when you look into the deep past or below a marker in time, you see much more branching or diversity below or before that point. Or perhaps a forest metaphor would be more apt with its branching roots interwoven together and connected by even more intricate networks of fungal—mycorrhizal networks (perhaps computation would be the eldest tree at the center—the parent of the forest—the conductor orchestrating the sharing of resources and warning messages). As Zielinski has done with his crucial research in Deep Time of the Media, we could make a cut at any point on the timeline of history and find many diverse examples of the various tendrils of thought working out the concept of computation.

Now if we pick up one of these (or several) tendrils, we can examine its inherent properties which exhibit the humans’ capabilities to understand nature through computation or numerical translation, driven by the exercise of pattern recognition. This foundation of mathematical understanding and numerical representation becomes the current running through the syntax of technical media. One of these examples that Zielinski explores is Athanasius Kircher’s personal desktop device called the cassetta matematica. He gives an in-depth description of the “box,” now residing in the Institute and Museum of the History of Science in Florence, and which sits nicely on a desk and has:

a menu of nine different branches and applications of mathematics: arithmetic, geometry, fortificatoria (dealing with calculations for military fortifications), chronologia (measuring time by regular divisions, in his case, the cycles of the moon and movements of the planets), horologia (science of constructing sundials), astronomy, astrology, steganography, and music. Assigned to each of these headings are twenty-four wooden slats, one behind the other, which according to each of the nine mathematical areas, are of different colors and marked with the letters of the alphabet A through I. (Zielinski 2006: 141–143)

Kircher’s apparatus serves as a kind of early calculator or algorithmic/mathematics database. Each of the slats has spaces which contain operations from the different fields and which can be arranged with other slats to recombine or arrange into different components (Zielinski 2006: 141–143). This provides another example of the digitization—the translation to a numerical code of knowledge—of media. As I will discuss later, and which is a theme that runs throughout this chapter, this numerical, mathematical translation of nature seems to be a key aspect of any sort of “language” of this media.

Evidence of the intertwined relationship between histories in art and human history and history of civilization more broadly are apparent whenever one considers the so-called “revolutions” in the history of progress, technologies, and collective thinking. Much like the industrial revolution triggered the scene for post-impressionism and then modernism, let’s consider how computation affords the convergence of media and leads us toward the “post” eras in art and culture. In a Fresh Air interview, historian Walter Isaacson discusses his book The Code Breaker which dives into the history and development of the CRISPRFootnote 1 technology. In the interview Isaacson gives his version of the three revolutions in modern times—revolutions of culture founded on basic particles or kernels:

  1. 1.

    The revolution in physics with a foundational kernel of the atom leads to “atom bomb, space travel, GPS, and semiconductors”

  2. 2.

    The revolution in computing with a kernel of the bit, “And it meant that all information could be coded in zeros and ones and binary digits”

  3. 3.

    The revolution in gene sequencing, with the kernel of the gene, “a fundamental particle of our existence...And in the beginning of this century, in 2000 or so, we sequenced the entire human genome. And now with [the invention of CRISPR] we found ways to rewrite the genome. And so this part of the twenty-first century…will be a biotech revolution, a life sciences revolution” (Isaacson 2021).

Now these revolutions can be debated—and have been—over the course of history. I’m sure some historians would take issue with the idea of leaving out the industrial revolution and other milestones in the history of science–culture–technology. Though here I want to focus on Isaacson’s second revolution: the revolution of the bit. Although rather than focusing on the bit or 1’s and 0’s, I would like to go deeper—and refer to the bit as the harnessing of electrical current or its physical manifestation. The bit comes from the “ons” and “offs” of electrical current pulses—and from that we get computation. We have this convergence of mathematics, physics, electrical engineering, and so on to bring us to computation. Computation also represents a convergence of media—or a convergence of media by which we hear and see through a technical means. Again the introductory quote by Kittler rings true. However, because it is a convergence of this media, computation is also heterogenous in the sense it represents all of our media or mediums. Like the invention of photography, the development of computation was the next inevitable or determined step in the convergence of our technologies and thus a post-medium condition in the arts. The evolution of the arts into its post-medium condition was informed by a computational perspective in culture and assisted by the adoption of the technology itself.

1.3 The Language of the Post-Medium Condition

In this section I outline several key concepts that help us understand convergence of our technologies and thus a post-medium condition in the arts. In their book Rethinking Curating, Beryl Graham and Sarah Cook grapple with the various ways technologies or computational media have changed the endeavor of artists, curators, and the audience in relation to experiencing art (or having an aesthetic experience of a work of art). They define and explore all the different ways “New” media art have affected artists’ and curators’ processes, practices and the properties of art and aesthetic experiences in this new age of digital technology, information, and electronic data networks. The authors highlight some properties of new media art such as variable and hybrid materiality, systematized, time-based, and networked, and they remind us of the definitions of interaction and participation. The curators state:

Interaction: ‘acting upon each other.’ Interaction might occur between people, between people and machines, between machines, or between artwork and audience. However, examples of human and machines or humans and artworks truly acting upon each other are relatively rare. What is popularly termed interaction in these cases is often a simpler ‘reaction’—a human. (Graham and Cook 2010: 112–113)

They go on to discuss and define other aspects of artworks as systems of change between work and audience and define other areas on the spectrum of artistic agency. In the same way that artworks that utilize computation as a medium can be interactive at times, they can also be participatory as well as collaborative in their often evolving realizations. Graham and Cook highlight these various natures of this medium (or post-medium) of artistic practice:

Participation: ‘to have a share in or take part in. Participation implies that the participant can have some kind of input that is recorded…that is, not just getting reactions, but also changing the artwork’s content…Collaboration: ‘working jointly with’. Unlike interaction and participation, the term collaboration implies the production of something with a degree of equality between the participants. (2010: 113–114)

Later in the chapter I will discuss the collaborative nature between artist and machine (AI) within the piece 432Hz. One could argue that all artists “collaborate” in some way with their media, and indeed many artists create works that are participatory or collaborative. Usually this is in reference between artwork (made through computational media or software) and audience. However, I will argue here that there is a certain level of equality—because this equality is afforded to the machine when we consider it to be its own working cognitive system working in tandem with the artist—another cognitive system—to produce the final work. For now, I want to consider the machine—which I use to describe this AI cognitive system—as the medium through which the artist works. What does it mean to consider computation as a medium, and what are the aspects of this medium’s language? Luckily, this “language” and its theory of artistic practice have been around for many decades, and therefore, there are already very solid measuring sticks in place.

In her text Voyage on the North Sea, Rosalind Krauss posits the task of Modernism coming to its conclusion as the implosion of specificity and the heterogenous artworld of comprised of distinct media. She eloquently guides us through a narrative of the modernist endeavor which culminates and then is followed by conceptual art to push the post-medium condition past the finish line (Krauss 1999: 10–20). Rather, I think of this as the beginning or start of something for the artworld and art theory. This collapse of boundaries of or the concept of a medium speaks to the new terrain we find ourselves in as a culture as well as art practitioners. This is where I believe the concept of The Deep Time of the Media and the Kittler quote become very useful tools in framing the idea of a medium in art—and the idea of a non-medium framework (or post-medium) for the artworld and art practitioners.

We can also look to the recent history of the language of contemporary art within the digital and then post-digital context. We can look to, or even borrow from, the previous syntactical systems that undergirded the arts going back to the advent of the adoption of video as a medium for practitioners. Of course, as we’ve seen with Zielinski’s research, we could go back further to find instances of a “language” of the arts that is fueled by concepts driven by digital systems or at least conceptually digital systems. Though I think a good place to start is period directly following the Second World War. This is the time when fields were all newly developing based on theories of cybernetics and computation—a time when computational media began to influence, and in some instances aid in the convergence of, the fields of cybernetics/cognitive sciences (science), technology/engineering (technology–military), and the avant-garde (art).

Krauss also adopts this time frame as she positions the post-medium condition in arts practices. She marks the advent of the post-medium age with the artist Marcel Broodthaers and the beginning of Conceptual art in the 60s:

Twenty-five years later, all over the world, in every biennial and at every art fair, the eagle principle functions as the new Academy. Whether it calls itself installation art or institutional critique, the international spread of the mixed-media installation has become ubiquitous. Triumphantly declaring that we now inhabit a post-medium age, the post-medium condition of this form traces its lineage, of course, not so much to Joseph Kosuth as to Marcel Broodthaers. (Krauss 1999: 20)

With this new art movement, the concept of the Modernist quest to find a medium’s essential essence—or its specificity—was abolished and medium specificity was upended. Starting in the post-war era, art was adopting a more heterogenous application and concept of the medium. As we’ll see later in this section, this time period in art ran parallel to the adoption of heterogenous technologies such as video and computation to make art.

In The Language of New Media, Lev Manovich completes an exhaustive synopsis of new media as an medium for creative practice as well as a form for media more broadly. This “new” medium largely encompasses digital or computational media, but extends and overlaps other media used to create aesthetic objects (such as video or cinema, photography, mixed-media, performance, etc.) Manovich outlines five “Principles of New Media.” These five principles include Numerical Representation, Modularity, Automation, Variability, and Transcoding (Manovich 2001: 27–48). Of course those principles were only the beginning of Manovich’s description of the language of new media. Though these principles set a good foundation for the properties of electronic and computational media to establish its language. Two decades later, there are many, well-established theories and languages of the media.

In their exhaustive and effective survey of the field(s), Casey Reas and Chandler McWilliams outline formal design principles of computational media or, in this instance, of “code.” Many of these principles overlap with Manovich’s while also elaborating on them. These principles included repetition, transformation, parameterization, visualization, and simulation (Reas and McWilliams 2010). These are the various principles or outcomes when computational processes are used to manipulate form (2D, 3D, 4D), and, thus inherently, content. We could also analyze (or include) the principles that come about just from including the inherent properties of the resulting forms brought about by computational media’s physical outputs. That is, we could also include more principles that are unique to say (RGB) light from a projector, 3D printed plastics, and printed pigmented inks. We could, for example, say that light from a projector is described as additive numerical representation of color and light which is transcoded by the machine. It is apparent though that this is not necessary as we already have consensus on the various principles of form-concept resulting from computational media. Even by looking at the example I’ve given of the projector’s light, we already see various principles at work: numbers are parameters that are transcoded into simulated light.

These principles of computational media are already well-established, and so there is no need to elaborate on them here. We can use these principles or these previously mapped aspects of computational media’s language as building blocks—a starting point—for the “language” of creative artificial intelligence. We will return to these aspects of this newly updated language later in the chapter. In addition, I want to focus on what these principles tell us—or how they can shine a light on why creativity and art have morphed into the place we find it in now. I want to utilize these principles in the event they can elucidate the various ways computational media and technology were the catalysts for this “post” era—these eras of post-digital and post-medium.

By Krauss’s measure, it was the introduction of the complex system of the Portapack (video) as a medium of art, which shattered the Modernist dream—like a Benjaminesque moment, where we crossed a threshold. The main part of the Modernist dream or endeavor that was ended by the medium of video was the endeavor of medium specificity. Because like film or the cinematic apparatus, there was no indivisible essence or quality that this media could be broken down into. Video was film’s electronic update in the 60s—adding telepresence, broadcast to film’s industrial qualities of repetitive reproduction and time (among other qualities). The language, or Krauss’s “essence,” of film and then video was too complex, too heterogenous to be reduced down to a homogenous specificity (Krauss 1999: 24–26). One could also argue other points in time as well as point to other technological developments to reference this crossing of a threshold—or ending of medium specificity. What about the artists beginning to work with computers for the very first time?—which by the way, was happening simultaneously with the adoption of the more popular video apparatus as artistic medium. Perhaps it was our embedded experience of our environment which was utilizing our new sensorimotor systems of telepresence, computation, was driving a new perception of ourselves and the world which was more variable and heterogenous. And these new perceptions, like anything in our history, work its way into every avenue of culture—including art which has always been a communicative mirror reflecting our culture’s current epistemological state.

2 The Language of Embodied Cognition

The current state of the field of cognitive science (referred by some as Post-Cognitive era) puts forth new ideas about how cognitive systems, consciousness, and the mind works through the theory of enaction or embodied cognition. In the seminal text by Francisco Varela, Evan Thompson, and Eleanor Rosch, after a survey of the past theories of mind that guided the field, the authors define and present a kind of “none” but “all of the above” theory through their idea of the mind as experiencing reality through a process of enaction.

What is key to these ideas of enaction, is that there is no such thing as a separation between the two entities of mind and body, but actually the mind–body is part of one cognitive system that experiences and takes actions in the world. It is worth explaining this idea as we consider the authors’ concept of perceptually guided action as it relates to the different approaches to cognitive sciences as they draw out differences in the opposing theories. For example, the authors’ state of perceptually guided action that: “We have already seen that for the representationist the point of departure for understanding perception is the information-processing problem of recovering pregiven properties of the world.” (Varela et al. 1991: 173) The authors then go onto speak about how the theory of enaction is different as it is not based on a concept of a pregiven, independent world, but that “the point of departure for the enactive approach is the study of how the perceiver can guide his actions in his local situation” (Varela et al. 1991: 173). In addition, Varela and his co-authors remind us that these “local situations” are constantly in flux and change, and that some of these changes are a result of the perceiver’s activity. Therefore, “the reference point for understanding perception is no longer a pregiven, perceiver-independent world but rather the sensorimotor structure of the perceiver (the way in which the nervous system links sensory and motor surfaces)” (Varela et al. 1991: 173).

I want to highlight this approach or this concept or this theory of perception as it relates to experience and the link between the mind, the body, and the experience of the world. As the author’s state:

The structure—the manner in which the perceiver is embodied—rather than some pregiven world determines how the perceiver can act and be modulated by environmental events. Thus the overall concern of an enactive approach to perception is not to determine how some perceiver-independent world is to recovered; it is, rather, to determine the common principles or lawful linkages between sensory and motor systems that explain how action can be perceptually guided in a perceiver-dependent world (Varela et al. 1991: 173).

So we see that the mind and sensorimotor system that is our body is actually a part of one cognitive system that experiences the world through a process of enaction where there is a constant feedback loop between this cognitive system and its environment through its sensorimotor functions it takes actions in the environment through a complex back and forth of tuning the environment and tuning its own reaction to the environment as it gathers information and takes subsequent action.

This model of the brain or cognition (and consciousness) is built on top of the previous connectionist strategy to model cognition/brains. The connectionist model is based on principles of emergence and self-organization that result from interconnected ensembles of neurons. Varela et al. summarize discussions taking place as far back as the Macy Conferences (the “formative” years of cybernetics):

Rather, brains can be seen to operate on the basis of massive interconnections in a distributed form, for that the actual connections among ensembles of neurons change as a result of experience. In brief, these ensembles present us with a self-organizing capacity that is nowhere to be found in the paradigm for symbol manipulation. (Varela et al 1991: 85–86) (aka the computationist’s model which was connnectionism’s preceding theory of cognition)

The authors go on to zero in on the theory’s foundational, singular explanation by citing ‘Hebb’s Rule’ which:

suggested that learning could be based in changes in the brain that stem from the degree of correlated activity between neurons: if two neurons tend to be active together, their connection is strengthened; otherwise it is diminished. Therefore, the system’s connectivity becomes inseparable from its history of transformation and related to the kind of task defined for the system. (Varela et al. 1991: 87)

So thinking is a process of “learning” which is the transformation of complex connections between neurons that fire together in certain ways for certain events or thoughts or conceptualizations. In this way, the network of neurons or these connections self-organize over time as the brain experiences the world through the body–mind’s sensors inputting physical touch, light/sight, eardrums, taste buds, and the olfactory system.

Let us consider further their trimmed-down example of this process. Take a total number of neurons and reciprocally connect them together. Connect some of the nodes to an input mechanism—say the retina. Then present the retina with a succession of patterns (images made of reflected light bouncing off objects). After each presentation of these patterns to the system, the system reorganizes itself by rearranging its connections to send signals in a very specific way. This rearranging is a process where the system is “increasing the links between those neurons that happen to be active together” during the time when the item is presented to the retinal inputs. This presentation of a whole collection of these patterns makes up the system’s learning phase. Finally, after this learning phase, when the system is presented again with one of the patterns, the system recognizes it because the system “falls into a unique global state or internal configuration that is said to represent the learned item” (ibid.: 87–88).

I am taking the trouble to explain these concepts that elucidate the connectionist model because it helps understand the model of enaction as well as my own use of this type of system in the artworks presented later in this chapter (432Hz). In a very real sense, this is what is happening during my performance while I am ‘training’ the system. I give the system a set of inputs. It feeds forward these inputs (in his case a sequence of numbers that represent soundwave frequencies) through the fully connected network of nodes (again which are only placeholders in the computer’s memory). Over time, connections are built up between specific nodes in the network. So that when I present the system with a set of frequencies, it recognizes the pattern through its subsequent connected firing of certain nodes, and the system responds by “answering” me with a new pattern of soundwaves. I will return to this process of training and my collaboration with the machine later in the chapter when I discuss the performance 432Hz in more depth.

This brings me to the discussion and exploration of the current state of cognitive science through its latest theory of cognitive systems or embodied cognition/experience—that to enaction. In this model of cognition, the mind–body or the entire cognitive system and its environment arise together through enaction within this embodied experience. Varela, Thompson, and Rosch explain their model of cognitive science by defining their theory of “embodied action.” The authors do this by focusing on explaining what “embodied” means in relation to cognition, and they highlight the first point “that cognition depends upon the kinds of experience that come from having a body with various sensorimotor capacities” (Varela et al. 1991: 173). Secondly, the authors point out that they use the term “embodied” because “these individual sensorimotor capacities are themselves embedded in a more encompassing biological, psychological, and cultural context” (Varela et al. 1991: 173). Furthermore, they define the term “action” and their intentions for using the term “action” in order to “emphasize once again that sensory and motor processes, perception and action, are fundamentally inseparable in lived cognition,” and they also emphasize that “the two are not merely contingently linked in individuals; they have also evolved together.” Finally Varela, Thompson, and Rosch define the concept of “enaction” as a model for cognitive systems:

We can now give a preliminary formulation of what we mean by enaction. In a nutshell, the enactive approach consists of two points: (1) perception consists in perceptually guided action and (2) cognitive structures emerge from the recurrent sensorimotor patterns that enable action to be perceptually guided. (Varela et al. 1991: 173)

In this view of cognition and experience, the perceiving actor is embedded within its environment, and as it perceives its environment, through its actions, it alters itself and the environment and thus further alters its perceived experience of it. The two things are entangled together, or what Varela et al. refer to as “structural coupling.” These concepts of embodied cognition can be found in studies in linguistics as well. Varela, Thompson, and Rosch use studies in linguistics by Mark Johnson (who I will look at in more detail later in this chapter) to exemplify these processes of cognition through enaction. Johnson explains how even during our basic categorization process cognitive structures are created based on our bodily experience. Varela et al. explain that these “…image schemas emerge from certain basic forms of sensorimotor activities and interactions and so provide a preconceptual structure to our experience…These concepts have a basic logic, which imparts structure to the cognitive domains into which they are imaginatively projected. Finally, these projections are not arbitrary but are accomplished through metaphorical and metonymical mapping procedures that are themselves motivated by the structures of bodily experience” (Varela et al. 1991: 177–178).

We will look more in depth at Johnson’s studies regarding metaphor and aesthetics later. Though I bring up this study here as it is a good example of the relationship between cognition and embedded, bodily experience. In conclusion, our experience through our sensorimotor actions within our environment dictates the recurrent, neuronal mappings that represent both our thoughts (brain’s activity, conceptualization of the world) and our experience of reality/the world and our feelings—which is to say this is all a sort of embedded feedback loop where we affect change in our sensorimotor experience as the wider physical world, society, culture is affecting changes to our cognitive structures/system. When we understand that this is how our cognitive system(s) works, we can begin to conceptualize different types of cognitive systems that exist in nature or in our constructed environments—different types of cognitive systems that are linked to ours but are not necessarily human. This also leads us to think about Hayles’s concept of nonconscious cognition.

As we chip away at our world, trying to grasp it and generate new knowledge, over the past twenty years, our species’ dive into this unknown has brought up some amazing theories and unearthed so many awe-inspiring discoveries of mysteries that still confound us—those mysteries involving things such as our brains (the interior) and our universe (the exterior). All the while, our ability(s) to hear and see through technical means—to borrow Zielinski’s phrase—has become increasingly sophisticated (continuing to hit milestones along the evolutionary trajectory of our species). This of course has changed the way we see and understand ourselves and our world—within our specie’s own embodied and enacted feedback loop at this moment of its evolutionary history where we find ourselves exactly where we should be given our actions and perceptions within our embedded environment. If we consider again these processes or systems of embodied cognition as a model for our own cognition, we see how these enacted patterns represent our cognition as interconnected or embedded within our environments and experience through our sensorimotor body–mind.

This leads us to think about cognitive systems—what they are and how they work—in a different light. In her text Unthought, N. Katherine Hayles effectively proposes a more encompassing perspective of cognition. The vehicle she uses for this model is a concept of a type of cognitive system she refers to as nonconscious cognition. In order to introduce and define this cognitive system, she first draws from various fields such as neuroscience and cognitive psychology to delineate definitions of cognition, consciousness, and higher consciousness among others. To start, consciousness is the cognitive function that comprises a core position in our thinking stemming from an awareness of self and others (found in humans, many mammals, and some aquatic species). Extended (or secondary) consciousness is associated with abstract thought, conceptualizing meaning, symbolic reasoning, verbal language, mathematics, and so on. Higher consciousness is the “autobiographical self” which is augmented by our inner monologue playing in our heads all day, and this then induces the “emergence of a self-aware of itself as a self” (Hayles 2017: 9–10). She then contrasts these self-aware cognitive processes with the cognitive system she calls “nonconscious cognition” which “operates at a level of neuronal processing inaccessible to the modes of awareness but nevertheless performing functions essential to consciousness” (Hayles 2017: 10).

Again she draws on the past few decades of neuroscientific research to detail some of these functions which include translating somatic markers into coherent body representations and discerning patterns too complex and subtle for consciousness to process (Hayles 2017: 10). Hayles generates a more inclusive definition of cognition as “a process that interprets information within contexts that connect it with meaning” (Hayles 2017: 22). She goes on to unpack the framework for cognition and provides various examples and applications at various levels for this model of cognition. In her parsing she dives into the definition and writes about each part of the definition and provides additional context. About the first part of the definition that states “cognition is a process,” she writes that “this implies that cognition is not an attribute, such as intelligence is sometimes considered to be, but rather a dynamic unfolding within an environment in which its activity makes a difference,” and she goes on to provide an example of a computer algorithm written on paper which is not cognitive until it is deployed to a platform capable of understanding the instructions and carrying out the process.

The next part of the definition “that interprets information,” Hayles points out that this interpretation implies there is more than one option for which a choice is made, and for example, a computational choice would be between true or false or 1 or 0. The connection to the generation of meaning becomes a key part of the definition, and Hayles emphasizes how meaning comes from contingent contexts. Regarding the final portion “In contexts that connect it with meaning,” Hayles writes that “the implication is that meaning is non an absolute but evolves in relation to specific contexts in which interpretations performed by the cognitive processes lead to outcomes relevant to the situation at that moment. Note that context includes embodiment” (Hayles 2017: 25–26, authors emphasis). I want to emphasize how these contexts are pointed out as being contexts of embodiment which serve as contexts for biological cognitive systems. As we discussed previously, or cognitive system is a sensorimotor system connected to its environment. Hayles also wants emphasize this point and states:

let me emphasize that technical systems have completely different instantiations than biological life-forms, which are not only embodied but also embedded within milieus quite different from those of technical systems. These differences notwithstanding, both technical and biological systems engage in meaning-making within their relevant instantiated/embodied/embedded contexts. (Hayles 2017: 25–26, authors emphasis)

When considering these thoughtful definitions and examples of cognitive systems, we see their relation to concepts espoused in embodied cognition. Cognitive systems are enactors that are embodied or embedded within a milieu or context where they are constantly receiving information coming into a sensorimotor system and make a conscious/unconscious/nonconscious enaction (choice) and/or feeding forward new meaning. Following these examples, Hayles outlines a “tripartite framework” specific to human cognition but also used as a way to conceptualize how these various levels interact and also how these ecologies or systems can include biological systems and technical systems. Specifically referring to human or self-aware cognitive systems, she developed:

A tripartite framework that may be envisioned as a pyramid with three distinct layers. At the top are consciousness and unconsciousness, grouped together as modes of awareness […] The second part of the [framework] is nonconscious cognition […] The even broader bottom layer comprises material processes. Although these processes are not in themselves cognitive, they are the dynamic actions through which all cognitive activities emerge. (Hayles 2017: 27–28)

This tripartite framework highlights the inner workings of the various aspects of the interwoven cognitive assemblages, and we can see how other nonconscious cognitive systems (biological or technical) are embedded within the environment and exact changes within these assemblages with “material processes.”

Hayles urges us to expand/broaden our anthropocentrically derived definition of cognition and cognitive systems. Everywhere we look, at various levels of magnification, we see interconnected systems that are cognitive systems at the core, in biological systems and in our complex technical systems as well as our own systems of thought and consciousness. When defining her framework for technical cognitive systems, Hayles, rightly I think, gives a description of this purview, both macro and micro, of technical systems characterized through nonconscious cognition. As we saw above, she highlights the importance of cognitive systems’ workings within a context. Furthermore, she elaborates on how interpretation embedded in contexts as it applies to nonconscious cognitive systems of technical devices such as “Medical diagnostic systems, automated satellite imagery identification, ship navigation systems, weather prediction programs, and a host of other nonconscious cognitive devices interpret ambiguous or conflicting information to arrive at conclusions that rarely if ever are completely certain.” Hayles uses this example to point out how this ambiguous process is exemplified in human cognitive nonconscious by stating “Integrating multiple somatic markers, it too must synthesize conflicting and/or ambiguous information to arrive at interpretations that may feed forward into consciousness, emerging as emotions, feelings, and other kinds of awareness upon which further interpretive activities take place.”

I want to highlight the parallels Hayles is making between these complex cognitive systems, namely that these systems involve feedback loops between input, interpretation, and decisions that feed forward into actions taken in the world. Hayles paints this picture of a complex technological cognitive system when she states that “In automated technical systems, nonconscious cognitions are increasingly embedded in complex systems in which low-level interpretive processes are connected to a wide variety of sensors, and these processes in turn are integrated with higher-level systems that use recursive loops to perform more sophisticated cognitive activities such as drawing inferences, developing proclivities, and making decisions that feed forward into actuators, which perform actions in the world.” And key to her argument she writes how these systems and their architecture work in the same way:

In an important sense, these multi-level systems represent externalizations of human cognitive processes. Although the material bases for their operations differ significantly from the analogue chemical/electrical signaling in biological bodies, the kinds of processes have similar informational architectures. (Hayles 2017: 24–25)

I also want to highlight what the author is emphasizing here that these multi-level cognitive systems represent a system that is similar to the system we saw in the embodied cognitive systems in previous paragraphs. If we consider various interconnected computational or technical systems we’ve employed around and above the earth alongside various micro- and macro-biome systems occurring in nature, we can see how all these systems are cognitive systems that work in a similar fashion as our own embodied cognitive system. I propose to take Varela’s, Hayles’s, and others’ model of embodied cognition and enaction (and nonconscious cognitive assemblages), and take the entity of the embedded mind–body subject and apply this system-body to other assemblages or networks or systems. These concepts will hold up as a model for how these systems (technological and biological) are embedded or intertwined within a cybernetic, feedback loop such as the feedback system I will discuss later exemplified by the performance piece 432Hz. Furthermore, this is an integral aspect of “creative AI” as well as the artist’s “creative intelligence”—that is the recursive, iterative feedback between an actor and what she outputs.

2.1 Embodied Aesthetic Experience

What does embodied cognition or embodied experience mean for the language and models of aesthetic experience? In The Aesthetics of Meaning and Thought: The Bodily Roots of Philosophy, Science, Morality, and Art, Mark Johnson puts forth a case that brings aesthetics and aesthetic experience into a central role of cognition and therefore our conceptualization processes, knowledge production systems, and onto our cultural assemblages/practices. Throughout this collection of essays, Johnson lays out his argument where he contends that.

we need to transcend this overly narrow, fragmenting, and reductionist view in order to recognize that aesthetics is not merely a matter of constructing theories of something called aesthetic experience, but instead extends broadly too encompass all the processes by which we enact meaning through perception, bodily movement, feeling, and imagination. In other words, all meaningful experience is aesthetic experience (Johnson 2018: 2).

Here Johnson posits his central argument that all experience is intertwined with the generation of meaning which by nature makes it aesthetic experience. He outlines his task to “… construct an argument for expanding the scope of aesthetics to recognize the central role of body-based meaning in how we understand, reason, and communicate" (Johnson 2018: 2).

Johnson goes onto speak to using the arts as a model for how we generate meaning through this merged (or recombined) lens of “body-based” perception or imagination-emotion-sensorimotor experience. Thus, the arts are instances of deep and rich “enactments of meaning,” and therefore, the arts and their subsequent enactments of meaning, Johnson argues, “give us profound insight into our general processes of meaning-making that underlie our conceptual systems and our cultural institutions and practices” (Johnson 2018: 2). He then goes on to punctuate this ‘embodied cognition’ line of thinking about all experience as inherently aesthetic experience because we are constantly using our experience and perception to derive meaning from our environment. Johnson states:

From this embodied cognition perspective, it becomes. possible to see the aesthetic aspects of experience as giving rise to mind, meaning, and thought. The view of meaning that emerges highlights the body-based, affective, and imaginative dimensions of our interactions with our environments as they shape the ways we make sense of, and reason about, our world. (Johnson 2018: 2)

Here again we have embodied cognition where the cognitive system is enacted through a sensorimotor system within the environment. Furthermore, here the embodied cognitive system arises not only with experience, but with experience that is aesthetic in its nature. As we’ve seen and discussed with the cognitive systems of embodied cognition and then nonconscious cognition, the mind–body, through its sensorimotor enaction perception, is folded into experience of the environment. Johnson points out that this is inherently a creative process as this is a system where cognition is ultimately enacting metaphor and meaning. While referencing Varela, Thompson, and Rosch’s process of enaction, he states that with this view of experience, “it is not correct so say that the mind is merely the brain, since experience encompasses the entire arc of organism–environment engagement, which is an enactive process. Sometimes neuroscientists are criticized (and rightly so) for claiming that all experience and thought take place within the brain. What they should say, according to a pragmatist nondualist ontology and according to good cognitive science, is that thought takes place via structures and processes operating at many levels: in neurons, in a cortex, in a brain, in chemicals in the blood, in an active body, in bodily interactions with one’s surroundings, in social interactions, within cultural institutions, and thus in a multidimensional environment. In other words, any satisfactory account of cognition will have to include the whole creative process of organism–environment engagement” (Johnson 2018: 40). These ideas regarding the functioning of cognitive systems are in line with Hayles’s concept of nonconscious cognitive systems. This seems to expand upon (or confirm) the concept of nonconscious cognition as Johnson points out is a multidimensional, intertwined collection of systems—or assemblages.

2.2 Artificial Intelligence Through Machine Learning: Cognitive, Creative Systems

As I stated previously, artificial intelligence (AI) has developed alongside computation and could even be seen as end by which computation is the means. In this context, it makes sense that AI was mostly theoretical up until only the recent past few decades—computation had to get ironed out first. Jürgen Schmidhuber’s survey article on neural networks goes over the long (although he admits that he may not have caught everything in this complex and rich field) history of supervised learning, unsupervised learning, reinforcement learning, deep learning, and evolutionary computation. All of these techniques, algorithms, machine learning (ML) systems, and models amount to the developments, and the field, of AI. We can look back to Weiner’s cybernetics and the Macy Conferences or Frank Rosenblatt’s trailblazing Perceptron in 1958. Though things didn’t really pick up until the 90s when artificial neural networks became more and more sophisticated, and after that, computer scientists were off to the races solving all kinds of problems and developing very sophisticated unsupervised, generative, and deep learning techniques. Schmidhuber writes:

In the decade around 2000, many practical and commercial pattern recognition applications were dominated by non-neural machine learning methods such as Support Vector Machines (SVMs). Nevertheless, at least in certain domains, NNs outperformed other techniques […] Important for many present competition-winning pattern recognizers were developments in the [Convolutional Neural Network] CNN department […] Good image interpretation results were achieved with rather deep NNs trained by BP variant R-prop; here feedback through recurrent connections helped to improve image interpretation […] Deep [Long Short-Term Memory Recurrent Neural Networks] LSTM RNNs started to obtain certain first speech recognition results comparable to those of HMM-based systems. (Schmidhuber 2015: 96)

These machines are becoming more and more effective and successful which in turn results in a renewed focus on their development in the twenty-first century. Computer scientists in both industry (Google) and academia are now using the ML models in place of normal computation in order to recognize and analyze patterns in datasets. Now of course, they are being put to use in order to learn and predict commodity consumption and subsequently target fine-tuned or personalized ads to the consumer, among other uses (robotics, etc.). This leads to the next generation of computational machine—deep learning generative machines.

To see an example of this development, around seven years ago the engineers at Google wanted to learn more about how the hidden layers in the machine operate which resulted in a new, more powerful machine learning system (Mordvinstev et al. 2015). As I discussed on the ISEA 2020 panel on AI in creative practice, these engineers realized through their investigation, that these models have the means to be able to generate novel images (DiBlasi et al. 2020). Other AI machines were developed and followed this DeepDream model’s footsteps, but now being applied to text and music generation. As I said at the time, I was less interested in this latest iteration of the longstanding debate over who is the artist or author, but rather wanted to focus on what this creative act by machine can reveal about agency and cognitive systems.

As we’ve seen in the previous outlined history of AI and ML, in the last six years or so there have been rapid advances in this machine learning branch of artificial intelligence. As a result of these advances in deep learning and deep generative modeling, these machines are now able to generate novel, creative output such as a musical score, an image, or a piece of text. There are countless examples of artists using this technology in all sorts of interesting ways. Although I am not going to get into too much details regarding these examples. Rather, I want to consider a specific type of use of AI in my creative practice—through performing AI. In this usage of the machine, the AI is one cognitive system, and myself, the artist, is another.

3 Performing AI: 432Hz

The project 432Hz, seen in Fig. 1, is an experiment in building artificial neural networks to calculate, train, and tune numerical expressions that are transcoded into computer-generated sound waves. 432Hz is a live, generative soundscape performance that utilizes the act of training neural networks to generate various soundwaves that evolve over time and fluctuate between the harmonic and the discordant. The piece explores the aesthetics of sound and movement expressed as data in order to create an experience of this information into generative imagery and computer-generated sound waves.

Fig. 1
An image of a person sitting in front of the projection screen. It is a still from the 432 Hertz project which is an experiment in building artificial neural networks.

Still from performance of 432Hz. Custom Neural Network, multi-channel audio, single-channel video. 2021

Full size image

In the past, tuning pitches tended to vary widely before tuning was standardized and based on the 440 Hz frequency. Before this standardization, this pitch was expressed in lower frequencies, and for a time, composers promoted a scientific pitch based on 256 Hz or 432Hz. 432Hz is an exploration of these tuning frequencies and how sound is expressed through these numerical relationships. The multimedia performance consists of generative imagery that evolves over time and mapped to computer-generated sound waves. Various soundwaves or oscillators expressed by the computer through assignment of these numerical values are layered and altered throughout the performance by a custom digital synthesizer created by the artist. The synthesizer is also a custom-built neural network that the performer trains throughout the performance to learn to generate a combination of various sine wave frequencies.

For the Festival Internacional de la Imagen Inter/Species festival I performed the work within the category of Soundscapes through the live production of 432Hz, and the piece was previously performed (with an intimate, spaced audience due to the pandemic) at the Museums Quartier in Vienna. Inspired by the emergent relationships between naturally occurring and artificially generated oscillations, and the evolving relationship over time between the audience and machine (AI) agent and experience of the auditory output, 432Hz involves a performance of a generative audio-visual experience. The development and the performance of the artwork take the form of a live computer-generated set of evolving projection and sound.

The performance 432Hz is an exploration of these tuning frequencies and how sound is expressed through these numerical relationships. The multimedia performance consists of generative imagery that evolves over time and mapped to computer-generated sound waves. Various soundwaves or oscillators generated by the computer through assignment of these numerical values are layered and altered through performance and a custom digital synthesizer created by the artist. The synthesizer is also a custom-built neural network that the performer trains throughout the performance to learn to generate a combination of various sine wave frequencies. The machine “learns” and tries different emerging patterns of combined oscillators. So with this project, I explore AI and the generative neural network as itself the media for artistic output as well as the resulting art object. So rather than having the AI create something for the artist—or program the AI to generate the novel aesthetic object (i.e., to make something under the guidance of the artist)—the performance becomes a conversation between the performer and the AI as it is being trained. Through the performance of this system, 432Hz explores the idea of the performer as simultaneously the builder and trainer of artificial intelligence through the construction of a neural network as itself the media of production. This media outputs an evolving aesthetic experience of sound and imagery that represents the generation of training over time but can also reveal the state of the learning AI at any moment in time.

I proposed as a model to elucidate a series of properties or principles for the use of AI for creative means. First of all, the neural network, or AI itself, becomes the created object—the aesthetic object to experience, rather than the AI’s generated output. Secondly, throughout the work, the AI represents a cognitive system, or technical nonconscious cognitive system, with which the artist, another cognitive system, engages in a conversation or dialogue with the AI system through the process of tuning—or training of the AI. Lastly, I want to consider a certain model postulated within the fields of architecture and experience design. In Richard Coyne’s text The Tuning of Place, he proposes what he calls a metaphor of “tuning” when constructing a theory of how we construct and manage experience within our places and spaces which we can think of in the context of nonconscious cognitive systems. Therefore, our places are cognitive systems that are made up of physical space as well as embedded, integrated, and pervasive digital media, and we tune these systems as we experience and interact with them. He writes that his use of tuning

is intended to embrace tuning-in and attunement, opening up an examination of the micropractices by which designers and users engage with the materiality of pervasive digital media and devices, including the inexorable accumulation of small changes, divisions, and ticks of such devices. So tuning provides a richer metaphor for the interconnected digital age than Mumford’s trope of synchronization. (Coyne 2010: xv)

This is what happens throughout my performance of the piece: my cognitive system tunes or trains the AI’s cognitive system over time. This is also an integral and unique aspect of the AI system. AI is trained over time where the connections between nodes in the network are tuned to be stronger or weaker based on the relationship between the inputs and the desired outcome. The piece and the experience are contingent as the two systems tune and morph over time based on different sensorimotor actions taken in response to the machine’s generated light waves and sound oscillations.

Through projects such as 432Hz, I want to explore the idea of artificial intelligence—and its evolution—as a medium for creative expression. As a medium for aesthetic experience in itself—the act of training is an act of tuning simulated “neurons”—which at its core are data expressed as a number occupying a space of memory within the larger interconnected network. Using the new research in the field of cognitive science—that of embodied cognition or enaction—as a lens to understand the relationship between myself, as an artist, in the act of creating, but also as the interplay between myself—a cognitive system—interacting or exchanging with another cognitive system. But wouldn’t that make the two parts simply one cognitive system? And what of the audience who is also connected to the work through their own aesthetic experience of the piece which generates various levels of meaning reflected in the work of art or aesthetic experience?

In these various ways, variable and hybrid nonconscious (and conscious) cognitive assemblages are generated and enacted in an embedded aesthetic experience. This idea has always been at the core of my interest in the landscape as an artist and my exploration of concepts surrounding the landscape in my work. How we move through our environment which is changing, as we alter it with technologies, etc., and we change to adjust to new alterations to the surroundings. I’m interested in this feedback loop between sensorimotor data, our navigation through the landscape’s infrastructure, learning its features, and then designing alterations to the constructed and experienced landscape. It’s truly inspiring how I’m engaged in a feedback loop between all of these biological, technological, and cultural systems that make up the environment and that make up myself as an embodied mind–body system.

4 Conclusion

In conclusion, I aim to highlight and propose a model for thinking about AI in creative practice by generating properties or the so-called syntax of the language of creative AI. As the chapter title suggests, I wanted to explore the formal and structural relationship between overlapping, contingent, and fluctuating cognitive systems that collaborate, or more aptly, tune each other and bring about changed states in each system.

In this current moment at the culmination of the interwoven histories of computation, AI and art, we seek to define the properties and structures of the language of creative AI which [I argue] can be seen as a culmination of a variety of languages rooted in aesthetics, artistic practice, and cognitive science. The framework created here is elucidated by a dialogue between various cognitive systems which use this language to create aesthetic experiences and which represent a collaboration between various creative actors and agents involved in this conversation. We investigated the histories of computation and AI and how these technologies have affected the language of the arts as both areas of culture developed and grew.

Through using the lens and theories of embodied cognition and enaction, I propose a collaboration or generative feedback loop that arises between various cognitive systems or assemblages. Finally, I used the concepts of enaction, aesthetic experience, and the sensorimotor cognitive system (or cognitive assemblages) to describe the relationship between various levels of aesthetic experience and artistic production. By creating a custom AI agent (or building the algorithms and mathematical system of artificial neural network) as the art object in itself, and then through performing and ‘tuning’ and training this AI, creative agency and aesthetic experience take shape as a collaboration between these two cognitive systems: the AI and the artist. Which in turn is experienced by an audience which then makes up a collection of other cognitive assemblages or systems.

As we ponder the convergence of mind, body, and experience into a cybernetic feedback loop, I propose to think about how we constantly tune and adjust to our experience, our mind–body systems of the environment which are applied to aesthetic experience and the artist’s research and production of aesthetic experiences and objects. The aesthetic experience (or the object of creative production) becomes a collaboration or a dialogue between various cognitive systems that are enmeshed together: the artist, the AI agent, and the audience. The language used in this dialogue exhibits the topology of embodied, aesthetic experiences that fold into one another and this, in turn, generates a possible model of the highly contingent morphology of these creative cognitive systems.