Sketching Symbiosis: Towards the Development of Relational Systems

Abstract

My approach to drawing using machine learning, data and robotics started as a way to evolve my drawing ability by designing artificial intelligence systems to explore collaboration with a robotic unit. For the past several years, my focus has been on the possibilities of robots as a medium of collective collaboration, communication and connection. The practice investigates the interactions between mark-made-by-hand and the mark-made-by-machine as an approach to understanding the dynamics of humans and systems. I believe interdisciplinary engagement lies at the foundation of speculative constructions of art and technology. In the work, I engage with interactive media like AI, robotics and augmented reality as a space of, and for, doubt and uncertainty rather than narrowly defined ends, creating works that address separation, merging and how we inhabit the relations between human, machine and non-human others. The following chapter is an outline, tracing a practice interested in the contours of where AI ends and the “I”, the individual human subject begins, using technology of the present-day to ask questions of authorship on a space of a canvas and a performance over time. For me, this question of agency within systems is a microcosm of the wider implications of technological governance and its entangled relationship to the human subject.

1 Introduction

The end of the human is not so much about the hypothesis that machines will completely replace human beings, because this may take longer from now than the extinction of the human species, but rather that machine intelligence will transform humans to an extent that is beyond their own imagination. We are in a flux of metaphysical force, which is in the process of carrying humans to an unknown destination.

—Yuk Hui Yuk Hui, On the Limit of Artificial Intelligence¹

Were it not for shadows, there would be no beauty.

—Jun’ichirō Tanizaki, In Praise of Shadows²

Creative practitioners occupy a unique space in defining the intersection of cultural and technical production, inventing languages of artificial intelligence for artistic exploration. More broadly, the term artificial intelligence (AI) has come to describe an increasingly interwoven suite of technologies including robotics, machine learning, synthetic sensing and natural language processing. Indeed, the landscape of AI is vast and its role in society is pervasive and increasing, shaping industries across a multitude of fields. However, “artificial intelligence” as a term itself necessitates more questions than it does propose answers.

For instance, what do we mean by “artificial”, and in what ways does the term presuppose conceptions of the “natural”, of the resulting relation between human and machine subjects? In this sense, it could be argued that all artistic media made with the suite of technologies that comprise an AI system implies specific ontological positions in regards to these definitions. These positions, conveyed through installation, artefacts and performance, extend to shape contemporary ideas regarding the artificial and natural, the relation of human-machine-environment, and the shifting nature of intelligence brought upon by digitisation and the post-industrial revolution.

As such, the work of art and research is an ongoing process of inquiry and invention. As a practitioner, I develop operations, as seen in Fig. 1, using emerging technologies to explore phenomenological constructions, interrelations and alternative configurations of human and machine. The operations produce artefacts, vestiges of presence in the form of paintings, research studies, sculpture, installation and performance. I investigate the computable and uncomputable, interrogating the promises and pitfalls of meaning-making metaphors for understanding complex systems; considering sight as a metaphor for computer vision algorithms, embodiment as a metaphor for multi-robotic systems, and memory and learning as metaphors for data-driven machine models.

Fig. 1

Sougwen Chung, Drawing Operations performance with industrial robotic arm, 2017

Throughout all these operations, the role of the human agent oscillates between relayer and receiver, architect and performer, observer and amplifier, designer and steward. At its core, the work takes a speculative approach using non-speculative research: the tools of the day. It asks:

What are the sensory mixes of the future? Where does “AI” end and “we” begin?

2 Towards a Relational Intelligence

2.1 A Note on Language

One can make the argument that there is no such thing as Artificial Intelligence because there is no such thing as natural intelligence. As Donna Haraway³and others have said, we have been cyborgs for a long time now; that is, the intelligence we might attribute to that of the human, or the natural, has always been entangled with technology and machines. Yet, one presupposition that concerns me about the current discourse around AI is how it reveals the shortcomings of language in describing a suite of emerging, exciting and fallible technological systems. The language suggests a false binary between the artificial and the natural disproven by cognitive science and developmental psychology.

Within the presuppositions of the term “artificial” lie a range of ideas and beliefs, such as the secular atheist accelerationalist’s belief in the artificial as transcendent, as seen in popular proponents of the technological singularity; something other-than-human and inscrutably above it. Perhaps these beliefs are founded on the conflating of resemblance as equivalence. While these systems of artificial intelligence seem to resemble competence in some traditionally-regarded-as-human tasks, to suggest its equivalence to human intelligence misses the mark.

It is along these same striations that a belief in the neutrality of AI systems also takes root. However, while the biases in AI systems have long since been discussed, the potentiality of foregrounding bias as a feature of AI systems is largely underutilised. One can begin by understanding the field of AI as a system of interconnected disciplines ranging from computer vision, robotics, data science and engineering, philosophy, art and so on. I include art and philosophy in the discipline of technological development as both are shaped by the cultural condition in which technologies manifest. I believe the construction of AI as “other” is a missed opportunity for a reinvention of the relations that shape the personal, the technological and the ecological. Within the premises of AI exists a provocation for the transformation of being as connected to tool and world and not separate from it.

2.2 On Fallibility as Ground

The most interesting thing I find in working with AI is its fallibility. Fallibility interests me as it is a shared trait across machines, human beings and the environment in which we are a part.

Recognising and exploring fallibility is a means of constructing relation through common ground. Doing so involves the development of relational AI systems trained on hyper-local, personal, individual datasets. It is my hope that this development offsets the misguided notion of a singular artificial intelligence towards a plurality of diverse, responsive, systems of play and governance.

The following projects follow a multi-generational approach to constructing relation as artistic inquiry, technological invention, and philosophical research and seek to offer constructions for moving beyond an ethno-human-centric view.

3 Practice-Based Research

What is ownership? What is work?… How can we move from a production system in which human labour is merely a disposable means … to a process that depends on and expands connective relationships, mutual respect, the dignity of work, the fullest possible development of the human subject?

—Adrienne Rich⁴

The following sections outline a brief overview of insights resulting from almost 10 years of explorations in human and machine co-creation. They stem from an approach that foreground hybridity of practice, including but not limited to research as art and art as research. A continuation of the antidisciplinary thinking made popular by the MIT Media Lab in the early 2000s, and the outmoded polarities of the humanities and the sciences, respectively.

In practice-based research, science transforms its languages; poetry invents its tongues.⁵ The below perform as prompts and provocations for merging, the continued reimagining of collaborative systems, as well as signposts along the development of a practice informed by emerging concepts in the humanities and the development of technologies of the day.

Introduction to Drawing Operations Unit: Generation 1–4

Drawing Operations Unit: Generation 1–4 (also known as D.O.U.G._1–4) is an ongoing project exploring human and machine collaboration. The work is presented as code, drawing artefacts, narrative, sculpture, installation and live performance. D.O.U.G._1-4 has utilised techniques in computer vision, deep learning on a dataset of two decades of my drawings, custom robotics, AR/VR and bio-sensors explore relation; to catalyse embodied configurations in human and machine creativity. You can think of them as responsive systems linked to my body, movements and biology. Each D.O.U.G. explores mimicry, memory, collectivity and spectrality as speculative prompts.

The artefacts are traces—traces of artistic speculation and investigative research. They mark a process of making that challenges, transmutes and distorts my own artistic agency as drawer, performer and programmer. The drawing operations take place in real time in the studio and in front of an audience, reflecting the processes, possibilities and paranoias of the time in which they were created.

D.O.U.G._1 began as a prototype of mimicry. D.O.U.G._2 expanded on the notion of memory. D.O.U.G._3 constructs collectivity. D.O.U.G._4 explores spectrality and teases at symbiosis through Flora Rearing Agricultural Network (F.R.A.N.) (see Fig. 2).

Fig. 2

Sougwen Chung, drawing operations unit: generation 1–5 diagram, 2020

3.1 Mimicry: An Exercise in Behavioural Empathy

Perhaps it belongs with the avant-garde abstract expressionist movement, under the context of Clement Greenberg’s theory on medium specificity in which contingency is accepted as aesthetic sensibility. Rather, this collaborative performance between the body and a robot arm is musical.

—Naohiro Ukawa⁶

Drawing Operations Unit: Generation 1 is the 1st stage of human and robotic interaction as an artistic collaboration. The robot mimics my drawing gesture via a system that gathers the data of my drawing movement through an overhead camera and analyses the position through computer vision software. The result is a synchronicity of movement between artist and machine in real time, resulting in an interpretive performance and captured as a drawing artefact, as shown in Fig. 3.

Fig. 3

Sougwen Chung, mimicry debut with D.O.U.G._1, New Museum Sky Room Performance in New York, 2014–2015

Shared Fallibilities

The performance is a process of slowing down, paying attention and communicating entirely through gesture. The robot mimics my movement like a partner in an improvisational singing performance. It is a robotic system that embraces every glitch, bug and error. The drawing session, without pre-established harmony, frees itself from aesthetic constraints, while also examining the essence and phenomenon of beauty at the same time.

The artefacts are white ink on black paper and trace the limitations of the robotic positional translation and my own adaptations to drawing with a robotic unit for the first time.

Beyond Automation

The project came at a time when interactive art and media in an installation format created an interaction model in which the visitor in the space acted as a human catalyst for generative machine responses. Drawing Operations Unit: Generation 1 sought to explore an inverted position of human and machine with machine as creative catalyst for a collaboratively composed composition of an image.

Drawing Operations Unit: Generation 1 references and re-imagines the premise of a predecessor in Harold Cohen’s A.A.R.O.N. project in which Cohen as a painter extends his visual language through a flatbed plotter generating sequences of his own brushstrokes executed by mechanical and computational principles.⁷

This work has theoretical ties to Lawrence Shapiro’s research on embodied cognition which describes the role of gesture and spatial reasoning in the experience and development of human cognition.⁸

In this first phase, the notion of collaboration is suggested through the creation of an interactive model beyond mere extension. While simplistic in its execution, the interaction model demonstrated in the D.O.U.G._1 configuration proposes a shift from automation to relation. Automation is the existing, hegemonic relation of human and machine paradigm derived from Industrial Revolution to relation.

3.2 Memory: Where Does “AI” End and “We” Begin?

Generation 2 (Memory 2015–2016) explored memory with deep learning and recurrent neural networks. It is an initial exploration into the machine learning of the drawing style of the artist’s hand. The robotic arm’s movement is generated from neural nets trained on two decades of my drawing archives, the basis of a gestural feedback loop based on my own style.

The white and blue artefacts show a hybrid human and machine drawing, in a sense I’m collaborating with two decades of my drawing as remembered by a machine, as shown in Fig. 4.

Fig. 4

Sougwen Chung, memory artefact with D.O.U.G._2, National Art Gallery in Tokyo, 2016

On a poetic level, the robotic arm has learned from the visual style of the artist’s previous drawings and outputs a machine interpretation during the human/robot drawing duet. Gestures from previous drawings are collected and saved, existing as a memory bank for D.O.U.G._2. Analysis of visual style of historic artists to translate into gesture as well as colour palette as a collective memory bank from which robotic arm will be able to select.

Foregrounding Artistic Bias

The work foregrounds the subjectivity of classifiers within a neural network, implementing human bias as an artistic style. By focusing on robotic memory using a recurrent neural network and point-based 2D path extraction. This project speculates at the beauty of a non-human move using two generations of an artist’s drawing data as training as source material for a training model.

Art as Research Artefact

The project is based on Sketch-RNN, part of the research and development of Google Brain researcher David Ha.⁹ The project uses custom software to convert two generations, two decades of drawing compositions on paper and various styles into machine readable paths. The project utilises an interactive system, an overhead camera in which the artist’s pen position within the canvas is inputted to dog two and the robotic unit responds based on a statistical approximation of previous drawn line work of an artist, as shown in Fig. 5.

Fig. 5

Sougwen Chung, memory data from D.O.U.G._2, V & A permanent collection, 2016

3.3 Collectivity: How Can Computational Ways of Seeing Reframe a Collective Imagination?

Drawing Operations Unit: Generation 3 is a multi-robotic system linked to the flow of a city. In this project, I developed custom robotics and a computer vision system in collaboration with Andy Cavatorta and Larry Gorman, from Nokia Bell Labs to create a co-creative performance that traced the movements of New York City in various locations and times of day. The computer vision system was inputted with public camera feeds from New York City.

Its debut performance Omnia per Omnia reimagines the tradition of landscape painting as a collaboration between an artist, a robotic swarm, and the dynamic flow of a city, as shown in Fig. 8. The work explores the poetics of various modes of sensing: human and machine, organic and synthetic, and improvisational and computational. Through a collaborative drawing performance between myself and a 20 unit, the project explores the composite agency of a human and machine as a speculation on new pluralities as shown in Fig. 6.

Fig. 6

Sougwen Chung, Omnia per Omnia performance with D.O.U.G._3, Bell Labs and New Museum Exhibition at Mana Contemporary, 2018

In Gutai art, “Matter never compromises itself with the spirit; the spirit never dominates matter. To make the fullest use of matter is to make use of the spirit”.¹⁰ The matter the artists of the Gutai tradition explored were paint, metal, canvas, with Akira Kanayama utilising primitive robotics as a remote painter. Today, matter in the form of digital data, biometrics and robotics has become responsive and interactive. Omnia per Omnia explores the interplay of today’s matter with Spirit as defined by Gutai, a performance of human agency communicated through drawing and mark-making.

Omnia per Omnia found inspiration in the Situated Knowledges research from theorist Donna Haraway. Situated Knowledge establishes a view of knowledge that is embedded in, and thus affected by, the concrete historical, cultural, linguistic and value context of the knowing person.¹¹ In machine parlance, each perspective of each camera embedded in urban space comprising the “machine gaze” is determined by the material of the camera itself, the specific modality of the algorithm that constructs visual meaning from the data retrieved.

(Bio)Metrics of the City

The city of New York is a conductor for multi-robotic choreography. Motion vectors extracted from public cameras are linked to D.O.U.G._3’s collective behaviour in the painting duet. I paint with this system of machines in the creation of an ephemeral portrait of a city in perpetual transition, in constant flux.

Surveillance Apertures

What do public cameras see? How do they see us? The positional data for the robots foregrounds the different states of the city via publicly available camera feeds. The optical flow algorithm categorises states of collective movement as density, dwell, direction and velocity, as shown in Fig. 7.

Fig. 7

Sougwen Chung, Omnia film featuring optical flow with D.O.U.G._3, screening at Mana Contemporary in New Jersey, 2018

Fig. 8

Sougwen Chung, mutations of presence performance with D.O.U.G._4, Laurenz Haus Residency in Basel, 2020

Ways of Seeing

The philosophical underpinnings of the computer vision algorithm captures the optical flow of a scene as opposed to the single object; it privileges the action of the collective (the behaviour of the crowd) over individual surveillance (face tracking and recognition), as shown in Fig. 6. Do computer vision systems that view a scene as a composition of discrete objects shape a sense of modern isolation?

Foregrounding a panopticon of eyes within an urban environment the states of the city the computer vision system was derived from a computer vision technique called optical flow which extracts the states within a scene and not an individual object. This is part of O’Gorman’s ongoing research on creating “kinder” cameras which do not delineate the individuals of a scene but instead extract collective states of collective flow for interpretation and research.¹²

This approach to computer vision in the terrain of public cameras is in the direction of the development of privacy-centric deployment of vision systems in the public domain.

3.4 Spectrality: What is a “Body”?

This is not a passive participation. Passivity plays no part in Relation. Every time an individual or community attempts to define its place in it, even if this place is disputed, it helps blow the usual way of thinking off course, driving out the now weary rules of former classicisms, making new “follow throughs” to chaos-monde possible.

—Eduoard Glissant, Poetics of Relation¹³

Generation 4 (Mutations of Presence 2019–2021) explored biofeedback and was developed in isolation during the onset of the pandemic. I focused on internal flows of meditation captured through biometric recording with an EEG headset. I translated those states to the robotic unit as a physical expression of my meditative states during lockdown, as shown in Fig. 8.

Drawing Operations Unit: Generation 4 is a project exploring spectrality, the electrical signals, currents and pulses of human and machine cognition. Utilising EEG and biofeedback technologies measure, analyse and to catalyse alternative marks made by machine using human subject as a conduit. The work complicates the determinative notion of the drawn mark as linked to human intention, exploring a configuration between human and machine operating at a sub-level of experience—that of the electrical signal. The work speculates on a relational recursion in which the human subject learns to draw through a muscle memory of the brain as an organ that can be trained through meditative and robotic practice.

Signal versus Noise

To draw parallels to computational development, the low level of the human system is accessed in Generation 4, the level of the signal versus the processor, the artistic intention of mark-making. This shifts the role of the human as agent to conduit—the human mind itself as a conduit for artistic co-creation.

Embodied Feedback Loops

D.O.U.G. 4’s incubation, conceptualisation and development took place in lockdown during the beginning of the COVID-19 pandemic in 2020. As a response to prolonged periods of isolation during the Laurenz Haus residency in Basel, Switzerland, I recorded my brain waves during experiments in meditation techniques ranging from vipassana (mantra), aural (sound), qi gong and visualisation approaches, as shown in Fig. 8. My biofeedback was recorded daily during 20 min meditation sessions. By measuring EEG fluctuations across the spectrum of alpha, beta, delta, gamma, theta, the extracted data is converted into kinematic positions resulting in a visual representation of the brain's electrical signals, as shown in Fig. 9.

Fig. 9

Sougwen Chung, mutations of presence paintings with D.O.U.G._4, Laurenz Haus Residency in Basel, 2020

Speculating Aura

D.O.U.G. 4 challenges the notion of the drawn line as an intentional, artistic gesture and posits a conceptualisation of drawing as inextricably linked electrical signals in the brain, the biology of the cognising human subject further moving away from the human as a control subject in the human/machine configuration.

An expansive and evolving relation between human and machine that generates movement and sound from the electrical signals generated by the human and robotic body.

The Electric Body: A Language of Unreason, But Which Carries a New Reason

The electric body—at all scales, atmospheric, subatomic, molecular, organismic—is a quantum phenomenon generating new imaginaries, new lines of research, new possibilities.

— Karen Barad

Trans*/Matter/Realities and Queer Political Imaginings¹⁴

The earth flows. We are now aware of the deep historical coproduction, or “sympoiesis,” of all kinds of material flows that we used to study separately. Flows of rock, flows of water, flows of air, flows of life, and even vast cosmic flows of matter are profoundly interdependent (relational) processes. What if we retold the history of the earth from this perspective?

—Thomas Nail, Theory of the Earth¹⁵

Exquisite Corpus is an immersive performance piece that explores human and machine and ecological bodies. The project links my biofeedback to the mechanical unit and the visual immersive environment. My biofeedback is recorded through an EEG headset and streamed real time to the robotic unit and the immersive environment as shown in Fig. 10.

Fig. 10

Sougwen Chung, Exquisite Corpus performance with D.O.U.G._4, Greek National Opera in Athens, 2020

The project exists in four chapters. Chapter 1 engages the audience in a sound exploration, neuro-audio orchestration derived from the artist’s electroencephalogram and the robotic unit’s electromagnetic frequencies in real time. Chapter 2 focuses on the shared gestural feedback loop between human and machine, the machine gestures derived from recorded and processed brainwave data. Chapter 3 integrates the human and machine performance alongside an immersive environment of real-time satellite feedback layered with the artist’s visual interpretation of brainwave data. Chapter 4 combines these elements to speculate at a new co-naturality between the human subject, machine and ecology, as shown in Fig. 11.

Fig. 11

Sougwen Chung, Exquisite Corpus EEG Visualization performance with D.O.U.G._4, Greek National Opera in Athens, 2020

3.5 Sympoiesis

This is the same motion as that of the paintings—from dull to brilliant, and then back to dull, and then back to brilliant. Ecological pulses come from and enable the experience of ancestral power. Indeed, for power to come forth, it must recede. For shimmer to capture the eye, there must be absence of shimmer. To understand how absence brings forth, it must be understood not as lack but as potential. This is where one grasps, afresh, the awful disaster of extinction cascades: not only life and life’s shimmer but many of its manifold potentials are eroding.

—Anna Tsing, Arts of Living on a Damaged Planet¹⁶

Flora Rearing Agricultural Network (F.R.A.N) is a project exploring human, plant and machine co-naturality. The work speculates at an interdependent ecosystem of human, machine and flora. Explores linkages between machine and ecology through the development of a networked robotic system stewarding nature.

The first phase of the prototype is a sketch for a bio-mimetic machine. It lays out a conceptual blueprint for a network of custom robotic machines powered by microbial cell batteries exploring unconventional approaches to generating sustainable energy. The robotic units in the piece will be designed to steward the surrounding flora. What are the synthetic plant and machine hybrids of the future? How can we exist as part of a symbiotic feedback loop of caretaking machines attending to nature and humans as gardeners for both.

The process conceived in F.R.A.N. 1 foregrounds regenerative systems and suggests alternative ways of conceptualising the dynamic configuration of human, machine and nature, as shown in Fig. 12. A configuration not as a productive force or creative force in service of the artistic artefact but as caretaking machines co-contributing to a renewed engagement with plant matter, entangled ecosystems and natural ecologies. F.R.A.N. addresses a shift in these views as a response to the extractive aspects of technological systems and its impact on planetary resources.

Fig. 12

Sougwen Chung, flora rearing agricultural network performance prototype 1, Pearl Art Museum in Shanghai, 2021

4 Summary

By pursuing a long-form process-driven work of this nature, the project posits that perhaps the future of human (and non-human) creativity isn’t in what it makes, but how it comes together to explore new ways of making. Developing new approaches to embodiment, memory and improvisation is what excites me about technology and it’s why I program my own creative systems. It’s why art and tech intersections with robotics, AI and virtual reality matter—its about exploring new ways of creating and becoming-with machines.

In the work of building relational models, the realms of mimicry, memory, collectivity and spectrality are drivers for system-making enacted through the practice of artistic exploration. Relational AI couples technosocial research and art science practices as wayfinding towards new cosmotechnical pluralities and emerging conceptions of human and machine.¹⁷