Keywords

1 Introduction—Historical and Philosophical Background

“Mister Science Meets Earth Mother”—an opening line from Rosenboom’s 1971 presentation, called Homuncular Homophony, that was delivered to the Spring Joint Computer Conference in Atlantic City, the Audio Engineering Society Convention in Los Angeles, and the University of Illinois Festival of Contemporary Arts, broadcasted something about the spirit of how the inspiring emergence of biofeedback in the 1960s, with all its implications and intermingling with cybernetics, computer science, neuroscience, systems theory, artificial intelligence, evolution, complex adaptive systems, studies in cognition and consciousness, and epistemology, among other disciplines, offered a doorway into a space where science and art might meet in meaningful and substantial, deep theoretical territory (Rosenboom 1976, 1997). Today, we might refer to this as artscience. It was imagined then that new developments in neuro-technology might be building a potentially powerful bridge that could link what were thought of as the inner and outer spaces of individual experience, while simultaneously offering both reasonable measures of phenomenal objectivity and rich offerings for creative realization. From this point, and in the whirling historical-cultural context of that time, it was a natural and obvious step to also want to explore building such links among the experiences of more than one human individual. From this environment of inquiry, multi-agent biofeedback emerged early on as a natural and irresistible arena for investigation. The term BCI (Brain-Computer Interface) was first coined in 1973 by Jacques Vidal (Vidal 1973) to describe a direct link between observable neuroelectric signals in the brain and a computer system. Now, decades later, the term has become relatively widespread and even colloquially used. Consequently, we can now conveniently refer to this emergent phenomenon as multi-agent BCI (MABCI), and in music, multi-agent BCMI (Brain-Computer Music Interface).

Several critical concepts about systems organization penetrated this environment deeply. The nature of feedback in developing electronic music and video synthesis paradigms, for example, was—and still is—foundational. The qualities of resonance and resonant emergence, also driven by feedback and observed in a wide range of natural phenomena, including the physical, cosmological, psychological, historical, biological, sociological, and cultural arenas, to name a few, have remained unbroken threads. Investigations into the behavior of systems (Foerster 1981), and more recently, self-organization and non-linear dynamics in the brain and human functioning (Kelso 1995), new understandings about the emergence of order (Holland 1995), and the adjacent possible in models of evolution (Kaufman 2000), are examples of continuous sources of inspiration from science crossing over into music and related arts. Multi-agent BCI in the arts was seen as a manifestation of interconnectivity, a broadening of self-reference to encompass multiple selves.

The meeting of science and art in this arena should not be confused as being one or the other; it is a joint space in which concepts can be exchanged and perhaps influence each other, ideas tried out with the freedom that artistic practice brings, and techniques tested for their relevance to rigorous practice and their potential to illuminate new theoretical models. Rosenboom has written extensively about an approach to composition he terms propositional music. Propositional music involves building proposed models of worlds, universes, evolution, brains, consciousness or whole domains of thought and life, and then proceeding to make dynamical musical embodiments of these models, inviting us to experience them in spontaneously emerging sonic forms (Rosenboom 2000c, 2018). Artistic license allows us to build these propositional models without requiring that they must completely correspond or explain some idea of reality. As Stephen Hawking is reported to have said, “I don’t demand that a theory correspond to reality because I don’t know what it is … All I’m concerned with is that the theory should predict the results of measurements.” (Holt 2018). There are many challenges to predictive model building when linking complex self-organizing systems via mappings in multi-modal stimulus domains. Propositional music, and by extension, proposition art making, may help open artscience conversations where some of our deepest theoretical questions lie: in our languages of description, how we describe what we experience, deduce, induce, propose, and believe that we know. A quote from Biofeedback and the Arts asks, “What is the place of nonverbal communication in the scientific method?” (Rosenboom 1972).

2 Qualities of Sounds in Non-contingent and Contingent Feedback Paradigms

In musical biofeedback paradigms, the qualities of the feedback signal and the nature of the auditory environment became important subjects of investigation. It was soon observed that the degree of success in achieving some control over the feedback signal—increasing the ability to influence the presence and coherence of alpha brainwave bursts, for example—was influenced by the nature of the sounds (Rosenboom 1976, 1997, 2003). Though this relationship might seem to be obvious—sounds conducive to the mental states being associated with particular frequencies of brainwaves extracted from the EEG—deeper investigation in both laboratory and performance situations revealed that the relationship is, indeed, not a simple one. It was found that aspects of attention, the dynamics of focused attention, the musical backgrounds of subjects and their facility with active imaginative listening strategies, all had profound effects on how subjects were able to interact with sonic environments and achieve a measure of success in a biofeedback control setup, be they simple or complex sonic worlds. The effects of musical and artistic backgrounds on affective judgments of aesthetic qualities had already been investigated in what was known as experimental aesthetics (Berlyne 1971). Subjects—or in the case of brainwave music, brainwave performers—could also develop considerable facility in feedback setups with extended practice. This seemed directly parallel to the way musicians gain performance facility with instruments through long, intensive, extended practice. Furthermore, those with experience in meditation, particularly as found in Zen practice, often brought skills that enhanced their comfort with complex sound environments and their facility in manipulating brain states within them, again, particularly with practice (Rosenboom 1976, 1997). The matrix of possible relationships among sound worlds and the cultural backgrounds of participants in biofeedback paradigms is also a complex one. A vast and continuously interesting territory for neuro-musical investigation in BCMI and the qualities of sounds in musical forms remains to be traversed. Elsewhere, Rosenboom has offered an agenda with questions and suggestions for future research in this territory (Rosenboom 2014).

With multi-agent BCMI setups, the qualities of the sound environments seemed at first to be particularly important. Early investigations involved practicing biofeedback exercises in groups. Early examples include various projects by composer Richard Teitelbaum—described in his article, In tune: some early experiments in biofeedback music (1966–74), and in Rosenboom’s early 1970s, carefully structured and immersive Three day biofeedback learning experience for Brown University (both are contained in Rosenboom 1976). These were mostly non-contingent, group biofeedback music setups. That is, the electronic musical feedback did not depend upon features of the performers’ EEGs being detected simultaneously. These group experiences soon lead to what was termed contingent, multi-agent biofeedback setups (Fehmi and Rosenboom 1971). In these situations, various methods of observing EEG features that were synchronous, or simultaneously detected, among two or more participants were developed and used to generate the auditory feedback signals. Sounds that were initially conducive to a group achieving simultaneous, synchronous brain states—simultaneous enhancement of alpha brainwave production, for example—were particularly important for the group to practice effectively. Again, though, more recent work has shown that multiple brainwave performers in multi-agent BCMI setups can achieve positive results in complex sound environments, especially if they have strong musical backgrounds and are active, imaginative, creative listeners. Setups like this will be described in detail later with technical descriptions of recent brainwave music works.

3 Historical Roots for the Development of Multi-agent BCMI and BCI in the Arts

Multi-Agent BCMI has long roots. Around 1969–1970, Rosenboom programmed an interactive game of Alpha Checkers, in which a computer screen would display a checkerboard for two players only when they produced EEG alpha wave bursts sufficient to cross an amplitude threshold at the same time. The players could only play the game when the checkerboard was visible. The task proved impossible to carry out, because when either player was looking at the checkerboard and trying to make a move in the game, the presence of alpha waves would decrease. This was an example of contingent, multi-agent BCI. Access to the necessary technology at the time was limited; so, it wasn’t possible to see whether with sufficient practice the players could master the system and play the game, while continuing to produce simultaneous and sustained alpha wave bursts. Interestingly, similar questions were explored nearly 30 years later in the context of a popular multi-agent BCI game and research project called BrainBall. Created in 1999 at the RISE Interactive Institute AB, the game has two players compete to increase their level of relaxation—thereby increasing alpha and theta EEG activity—which in turn controls movement of a magnetically coupled ball towards an opponent’s “goal” area. The game could also be played collaboratively, wherein players must alternately increase and decrease relaxation levels to move the ball towards the center of the table. Playing the game competitively reportedly resulted in reduced stress, as measured by galvanic skin response, and the players’ attitudes towards the game were reported as generally quite positive with user tests suggesting that players were able to successfully “competitively relax” (Ilstedt Hjelm and Browall 2000). In addition to Alpha Checkers, throughout the late 1960s and early 1970s other explorations with contingent alpha biofeedback setups involving only sound-based feedback were carried out. These proved more successful. Some of these took place in an EEG lab at the State University of New York at Stony Brook in collaboration with psychologist Lester Fehmi (Fehmi and Rosenboom 1971).

Rosenboom’s more substantial work in the multi-agent BCI arena began around 1970, with an environmental, demonstration-participation-performance event, called Ecology of the Skin, which was held at Automation House in New York City. In this exhibition, up to ten participants could wear EEG electrodes connected to portable EEG preamplifiers, filters, and amplitude envelope followers that were connected to an electronic music generating system. In addition, some EKG monitors were available, and stations for electrical stimulation of visual phosphenes were installed around the exhibition space. Most of these employed non-contingent feedback setups. Subsequent iterations and spinoffs from the original Ecology of the Skin, however, did begin to employ contingent biofeedback setups (Fig. 4.1).

Fig. 4.1
figure 1

Diagram of an early example of a contingent multi-agent feedback system (reproduced from Rosenboom 1972). Synchronous alpha bursts from two participants triggered slow rising sweeps of a harmonic series with slight, automatically induced sequence changes through initiation of voltages that determined the starting pitches of resonant filters and randomly introduced bell-like accentuations of various harmonic tones produced by shocking the resonant filters with narrow pulses at the attack initiated by each synchronous alpha burst and throughout the sequence

Full size image

Soon after Ecology of the Skin, Rosenboom and collaborators built a facility called the Laboratory for Experimental Aesthetics at York University in Toronto. Here, students and faculty, notably Richard Teitelbaum, Barbara Mayfield, C. Mark Nunn, and others, developed systems for exploring both contingent and non-contingent, multi-agent BCMI on a regular basis. Various artists developed installation pieces, such as Jacqueline Humbert’s Brainwave Etch-A-Sketch, in which low-frequency envelope followers tracked the amplitudes of alpha brainwaves from two participants, one of which moved a dot on a storage oscilloscope along its x-axis and the other along its y-axis, to create a shared drawing. This was an example of a non-contingent feedback system, as the presence of absence of feedback did not depend on the contemporaneous detection of a specific EEG feature in both participants. Conversely, contingent feedback was employed in another Humbert installation, Alpha Garden, wherein simultaneous alpha bursts from two participants would turn on pulses of water from a lawn sprinkler that irrigated a piece of artificial turf.

In 1972, Rosenboom expanded the scope of contingent biofeedback art with his Vancouver Piece. A darkened, sound-isolated room was built inside the Vancouver Art gallery during an exhibition of sound sculpture pieces. Inside the room were subtle types of visual and auditory displays and equipment to detect the EEGs of two participants at a time. In one of the room’s most intriguing setups, two participants could sit on either side of a two-way mirror with red and green lighting arranged so as to subtly illuminate each participant’s face when they produced alpha brainwave bursts that exceeded a preset threshold. Each participant would see their own face reflected in the two-way mirror when they produced sufficient alpha; but when the two produced simultaneous increases in alpha, their faces would appear to switch positions, so that each player would see their own face seemingly positioned on the other player’s shoulders. The intended effect was to open the participants’ consciousness of self to enable them to explore ideas about shared identity. The result was strongly engaging. It was another example of contingent feedback (Fig. 4.2).

Fig. 4.2
figure 2

Two museum attendees participating in Rosenboom’s Vancouver Piece at the Vancouver Art Gallery in 1972

Full size image

Around 1969–1970, Rosenboom organized a multi-agent biofeedback ensemble, called the New York Biofeedback Quartet. The idea was to gather a group that could practice biofeedback music together regularly. As personal circumstances played themselves out, this ensemble was short lived. However, it lead to several substantial biomusic compositions that have become regarded as early classics.

In 1972, Rosenboom created two works called Portable Gold and Philosophers Stones (Music With Trills) and Portable Gold and Philosophers’ Stones (Music From Brains in Fours) (Rosenboom 1976). The score for the first piece describes a “dual-contingent” feedback system in which musical results depend on simultaneous theta or alpha brainwave activity. The electronic music equipment includes a device for generating sub-harmonically related tone complexes, the spectra of which are scanned with resonant band pass filters that are being tuned by the brainwaves. The resulting tones are mixed into an immersive electronic music texture that is broadened further via an accumulation tape-delay system. This piece was performed, though never recorded.

The second work, Portable Gold and Philosophers Stones (Music From Brains in Fours), expanded the multi-agent BCMI paradigm in significant ways. This time, an ensemble of four biomusic performers is specified in the score. EEG signals from the four performers are routed and processed through a coarse-grained Fourier analysis device, in order to track several EEG frequency bands, and a correlation function computer that measures the coherence times of signal bursts in selected EEG frequency bands. At the time this piece was created, both these analysis functions were accomplished with analog equipment. A fifth performer operates the electronic equipment and routes the outputs of the analyzers to an electronic music generating system, which again includes a set of sub-harmonically related tone complexes. This time the tone complexes are fed into a bank of resonant band pass filters, known as a holophone. The holophone idea was inspired by Longuet-Higgins’s description of a scheme for non-local storage in the time domain, analogous to non-local storage in the spatial domain with holograms (Longuet-Higgins 1969). The performers know that as the coherence times of their selected EEG frequency band bursts increases—as measured by the correlation function computing circuits—, the range of their control over the holophone is also increased. Furthermore, by changing various time constants in the holophone circuitry, the detail of control they can affect is also increased. Thus, initially, slowly moving effects—gradually evolving, drone-like sounds, for instance—may become broader with more fast moving detail—wider pitch excursions with trill-like sounds, for example—as the corresponding performer’s EEG band bursts become longer and smoother (Fig. 4.3).

Fig. 4.3
figure 3

Signal flow diagram from the score for Rosenboom’s Portable Gold and Philosophers’ Stones (Music From Brains in Fours)

Full size image

Over years, Portable Gold and Philosophers’ Stones (Music From Brains in Fours) has been performed many times. One particular performance from 1972 has been released and re-released on vinyl records, CDs, and digital distribution (Rosenboom 1975, 2000a, b, c, 2006, 2019a, b).

In the mid-1970s, Rosenboom’s work shifted towards investigating what can be done with auditory event related potentials (AERPs) extracted from the EEGs of participants in a biofeedback paradigm. Of particular interest was how AERPs might provide information about attention shifts that could be related to various kinds of changes in an evolving sonic form. Through many experiments, this led to another major biomusical work called On Being Invisible, in which AERPs in a biofeedback scheme were used to guide the emergence of a self-organizing musical form. Full technical description of this project and the nature of AERPs is beyond the scope of this article. However, On Being Invisible has been documented extensively elsewhere, performed, and recorded (Rosenboom 1977, 1997, 2000a, b, c, 2019a, b). Mostly, this work was presented as a solo performance, not a multi-agent work, and the technical descriptions will not be presented here. However, it is worth mentioning that a follow-up piece, called On Being Invisible II (Hypatia Speaks to Jefferson in a Dream), which used two performers in a multi-agent AERP feedback paradigm, was created and realized in 1994 (Rosenboom1997). Building on the first iteration of On Being Invisible, the artistic concept was partly to try to see if the idea could be extended so as to create a self-organizing opera, one in which the pathways through the opera’s non-linear narrative would be guided by the AERPs detected from the two performers together. The performers would react primarily to auditory events and be shielded from visual stimulation; however, AERP events with strong P300 components—(a peak in the AERP, occurring approximately 300 ms after the onset of a highly differentiated stimulus event, that is commonly associated with aspects of attention)—would also be used to essentially edit sampled voices delivering bits of text and stored visual sequences for the audience. The results would be different in every performance. A recorded example is available (Rosenboom 2000b) and program notes are available online (Rosenboom 1994).

4 More Detailed Descriptions of Selected Recent Works Produced with Multi-agent BCI and Multi-agent BCMI Paradigms

4.1 Ringing Minds

Ringing Minds is a collaborative work created by David Rosenboom, Tim Mullen, and Alexander Khalil. A first version was produced and performed in 2014, and a detailed technical description was published in (Mullen et al. 2015). Ringing Minds is a complex multi-dimensional, multimedia, multi-agent BCI project in the arts which explores new possibilities in contingent and non-contingent feedback, concepts of “audience-as-performer,” complexity and structural forms in music and the brain, and resonance within and between listeners and performers. Ringing Minds uses real-time “hyperscanning” techniques to model event related potentials (ERPs) and resonant properties of neural activity simultaneously measured from a group of individuals engaged in active imaginative listening during a live musical performance.

The EEG signal processing builds on multivariate principal oscillation pattern (POP or eigenmode) analysis methods for identifying resonant properties of a time-varying dynamical system. Each POP characterizes the response to a specific input of an independent, stochastically forced, damped harmonic oscillator or relaxator. Another way to think about the dynamics of a POP is as equivalent to an idealized string “plucked” with a specific force plus additive random excitation. POP analysis methods had previously been applied to multi-electrode electrophysiological data to identify characteristics of spatiotemporal oscillatory modes in single individuals (Mullen et al. 2012). For Ringing Minds, each of four participants’ single-electrode EEG time series (sampled at the 10–20 Cz location) were instead treated as if generated by a common dynamical process—a “hyper-brain” sampled by four sensors. Within a sliding window, the multi-brain EEG time-series were decomposed into a set of forty POPs, spanning the EEG frequency spectrum. In this manner, each POP may be regarded as an extended neuronal process (e.g. a coherent network) spanning the four brains, oscillating at some frequency and/or exponentially decaying in response to an excitatory input (e.g. a musical event), or reflecting a resonant/synchronous state of this “hyper-brain”. Each POP was characterized by seven dynamical parameters, including frequency, initial amplitude (excitation), and decay (damping) time, which were mapped onto a software-based electronic music instrument, the central core of which is a very large array of complex resonators. These respond to the POP data in a way that generates a vast, spatialized sound field of ringing components, analogous to ways neural circuits might also “resonate” and sustain modes of behavior within and between individuals. POP-to-resonator mappings were chosen to produce an aesthetic interpretation of the precise meaning of oscillator/relaxator for POPs. Periodically, the shapes and temporal positions of important peaks in ERPs, averaged across the four brains within a 1 s sliding window, were applied to modulate the resonant auditory field, sounding as if a stone had been tossed onto the surface of a sonic lake (Fig. 4.4).

Fig. 4.4
figure 4

Simplified diagram of the Ringing Minds system. Hyper-scanning analysis techniques are used with an N-Brain Group. Single-trial average ERPs (Event-Related Potentials) are captured via spatial averaging across the group, rather than the traditional approach of averaging across multiple trials (i.e. repeated events) within a single individual

Full size image

A second version of Ringing Minds was produced and performed at the Whitney Museum of American Art in 2015 during a fifty-year retrospective of Rosenboom’s work. In this version, the work was expanded with the collaboration of visual designers, Matt Wachter and Glenn Snyder, to include elaborate video projection displays showing components that paralleled the EEG analysis and music generation systems.

The concept for the visual display began with the idea that the POP resonances detected from the four brainwave performers were analogous to stones being dropped onto the surface of a still lake; and the nature of the ripples that spread out from the location of the stones impact on the water was analogous to the properties of each POP. This also paralleled how in the computer music instrument, the POPs were mapped onto a large array of complex digital sound resonators.

For each POP, a splash of color was displayed on a screen, forming a visual backdrop in the performance space. The spatial positions of the color splashes were determined by the dominant frequencies and spatial distributions of each POP. The vertical position was determined by a corresponding POP frequency, and the horizontal position was determined by the spatial distribution of the energy contributing to the POP across the four brainwave performers (Fig. 4.5).

Fig. 4.5
figure 5

Photo by Paula Court

Image from the Ringing Minds performance at the Whitney Museum of American Art in New York in 2015 showing color splashes initiated by individual POP (eigenmode) detections from the N-Brain Group.

Full size image

Another new feature in the Whitney Museum performance of Ringing Minds was the addition of a contingent feedback component based on detection of contemporaneous ERPs amongst the participants. ERPs were extracted by averaging EEG signals, sampled at the standard 10–20 Cz electrode location, across the members of the N-Brain group, rather than the traditional method of averaging over successive occurrences of a stimulus in time. A simple template matching procedure was then used to detect the presence of a contemporaneous ERP response across the N-Brain group to unexpected auditory events. The group-averaged EEG time-series were convolved in a sliding window with an ERP template reflecting the average of ERPs from 48 individuals elicited in response to rare deviant (unexpected) tones interspersed within otherwise predictable sequences of standard tones which differed from the deviant tones in their duration. When the correlation between the N-Brain ERP and the template ERP exceeded a pre-determined threshold the shape of the detected ERP was mapped onto a musical pitch space and sent via MIDI to a Yamaha DisklavierTM grand piano. The ERPs were, thus, played automatically on a piano, providing contingent feedback to listeners, as well as to musicians, and adding to the overall musical experience. This occurred during sections of the music in which two musicians (Khalil and Rosenboom) played a lithoharp (a kind of xylophone made of carved stone bars) and an electronically processed violin, in interaction with the brainwave music performers. The musicians could attempt to create and violate musical expectation in the listeners and thereby elicit collective neural responses, which in turn would be sonified. For the brain artists, listening was the performative act (Fig. 4.6).

Fig. 4.6
figure 6

Photo by Paula Court

Four listening brainwave performers participating in Ringing Minds. The shapes of their evoked responses (ERPs) are being played on the Yamaha Disklavier™ piano on the right.

Full size image

4.2 Portable Gold and Philosophers’ Stones (Deviant Resonances)

Following his 2015 Whitney retrospective, Rosenboom composed a third piece in the Portable Gold… series, Portable Gold and Philosophers’ Stones (Deviant Resonances). It was premiered in The House performance space at Plymouth University, UK, as part of a BCMI Workshop in association with the 2015 Computer Music and Multimedia Research Conference. It has been performed many times since then and has recently been recorded for release (Rosenboom 2019a).

Portable Gold and Philosophers’ Stones (Deviant Resonances) is structured for two active imaginative listening brainwave performers and a computer-electronics performer. It employs both non-contingent and dual contingent biofeedback paradigms. The brainwave performers’ task is to remain still and listen actively with eyes closed, allow their attention to be drawn to any features of the sound texture, to actively direct their attention to specific sonic features they may choose, and to continuously notice when they observe that these listening actions may be related to how features of the sound texture evolve. If possible, they may also choose to direct features of the sound texture with their shifting attention; and, in any case, they are to practice immersing themselves in the sound texture and attempt to increase the degree to which they can actively interact with it. This is called listening as performance.

A complex software instrument was created for this piece using the Reaktor Core digital signal processing and synthesis platform. The instrument is designed to receive raw EEG signals from two performers via the OSC (Open Sound Control) data format. Each of the decoded EEG signals is parsed into three individual frequency bands, the upper and lower band limits of which may be freely set and adjusted. Each of the filter outputs is made available individually and is also fed into an amplitude envelope follower algorithm with adjustable time constants for rise and fall. The frequencies of the filter outputs are also tracked and provided as outputs. In addition, they are mapped onto a selected range of MIDI pitch values with controllable scaling and offsets. These signals may be used to play other modules in the overall, complex instrument. The envelope outputs are also made individually available for use by other modules. Other modules in the overall instrument will be described below, along with a tour through the compositional structure.

The composition begins with a sound texture created with sub-harmonically related tonal complexes made with pulse waves feeding banks of resonant band pass filters. Just as in 1972 with Portable Gold and Philosophers’ Stones (Music from Brains in Fours), this bank of resonant filters is referred to as a holophone, this time with refinements. The holophone can isolate and recall multiple tones from the several overlapping harmonic series in the pulse-wave chords that are fed into it. The pitches of the pulse wave chords can be determined in advance and may be changed during performance. (So far, performers have mostly chosen to keep them pretty stable, though this is not required by the composition.) The envelopes of selected EEG bands are patched into the holophone in a manner that gives them control over movement among the tones it produces. The performers are informed ahead of time about which musical voices from the holophone are responding to each of their EEG signals and where those sounds will tend to be located in a simulated 3D sound diffusion space. They know which sounds are responding to which performer and where in space they are likely to be heard. The ensemble may decide ahead of time, or in the moment of improvisation, exactly how to choose among the optional control paths from EEG envelopes to holophone movement. The electronics performer must carefully monitor and adjust the EEG signals received via OSC to make sure they fall within acceptable ranges, adjust the time constants of the envelope followers, and monitor signal flow into the holophone. This section unfolds slowly, usually starting simply and growing more complex as the performers settle into the nature of the exercise.

At a certain point, the electronics performer may choose to activate a second and eventually a third layer of sound elements enabled by the overall instrument. This is analogous to activating second and third sections of an electronic orchestra, all of which are responding to signals from the brainwave performers, while also being guided by the electronics performer. Before describing how these sections evolve in a musical performance, some further technical description is needed. Refer also to the Fig. 4.7.

Fig. 4.7
figure 7

Signal flow among primary components of the Portable Gold and Philosophers’ Stones (Deviant Resonances) system

Full size image

The amplitude envelope values from each performer’s filtered EEG are patched into a module called EEG2MIDI, the algorithm of which includes a delta function that responds when the rate of change in the amplitude envelope exceeds a delta threshold set by the electronics performer. When the signal exceeds the delta threshold, its value is mapped onto an adjustable range of MIDI pitch values and a scalable range of MIDI velocity values. These MIDI pitch values are sent to a pitch-mapping module and used as index values into a lookup table of pitches. This table contains data defining four musical scales that have been pre-composed to contain particular pitch interval sequences: (1) a scale made with pentagonal numbers and interval sequences in numbers of semitones (4 5 2 …), (2) interval sequences that do not repeat at the octave (1 1 1 1 2 2 1 1 1 2 …), (3) interval sequences based on hexagonal numbers (1 2 …), and (4) superimposed sub-harmonic and harmonic sequences on the pitch C.

Three such pitch mappers are used in the system. The outputs of the first two are patched to two very particular synthesis instrument modules, one corresponding to each performer. These instrument modules are known as Touché II. The original Touché digital keyboard instrument that was developed by Donald Buchla in collaboration with David Rosenboom in 1978–1979 inspired their design. Both instruments employ nonlinear wave shaping algorithms as their primary sound synthesis method. Linked to this, and a key to the instruments’ success, is the ability to program many, very complex control envelopes of arbitrary length, with decision logic that can be applied to every breakpoint in an envelope design. These envelopes can be used to modulate every parameter in the instrument’s synthesis algorithms. The original Touché was a hybrid, digital-analog hardware instrument. The Touché II, developed in 2007 by Rosenboom in collaboration with Martijn Zwartjes, is an entirely software-based instrument. Describing its full technical details lies beyond the scope of this article. Suffice it to say that it is an extraordinarily powerful, live electronic music instrument that can traverse a tremendously rich sonic terrain.

The third pitch mapper is connected to another interesting instrument made of a bank of complex digital resonators, with both deterministic and stochastic control functions. These resonators are activated by a set of exciter functions with variable slope controls and ways to inject indeterminacy with various kinds of noise into their behaviors. One can think of these digital resonators as complex bells that can be rung by complex exciter functions. The design of these resonator banks began with the composition of Ringing Minds. Those used in Portable Gold and Philosophers’ Stones (Deviant Resonances) are derived from those used first in Ringing Minds, though they are slightly simplified and reduced in number for practical reasons.

To recapitulate, the electronics performer can activate and deactivate what may be thought of as three layers or sections in an electronic orchestra. The first is the one described above with sub-harmonic pulse wave complexes and the holophone’s resonant band pass filter bank. The second is the complex digital resonator bank with its exciter algorithms. The third consists of the two Touché II synthesis instruments, each of which is also preprogrammed with an array of preset algorithms that can be called up instantly with MIDI program change signals.

A module in the system called Control/Test/Delta is key in managing a performance. It enables the routing of control information and triggers for sounds around the instruments in the electronic orchestra in several ways. First, direct EEG pitch tracking from either brainwave performer can be selected and routed to the Touché II instruments and/or the complex resonator bank. Second, EEG amplitude envelope delta threshold crossings from either performer can be selected and routed to the Touché II instruments. When this is done, each delta threshold crossing will initiate a program change in its corresponding Touché II instrument. Delta threshold crossings can also be merged so that signals from both performers affect both Touché II instruments. Finally, and very importantly, a function can be selected which only routes program change signals to the Touché II instruments when delta threshold crossings from both performers occur at precisely the same time. This introduces the contingent feedback paradigm into a performance.

A word about these delta threshold crossings and their possible relation to attention shifts is pertinent here. It is commonly thought that when a subject is in the process of producing increasing amounts of coherent brainwave signals, such as highly coherent alpha waves, the interruption of these signals can often accompany a significant shift of attention in that subject. The instrument developed for this piece includes an ability to react to the rapid onset of alpha or other coherent brainwave bursts, and also to their quick interruption. In addition, it can respond when changes of this type occur simultaneously in the two performers. In the highly controlled conditions of a laboratory, the validity of these assumptions can be tested. In the environment of a brainwave music performance, they are considered to be very interesting phenomena to explore when intensively engaged in a biofeedback music environment.

It is also important to underscore that it may not be productive to engage with a brainwave music instrument of this complexity casually. It is, rather, an instrument to be mastered with extended practice, just as one might practice any musical instrument with high levels of discipline to achieve mastery. In addition, experience shows that performers experience more rewarding results when they are truly able to engage in active imaginative creative listening. Often such individuals bring prior experience in sound arts and/or music to bear, and perhaps, also techniques for meditation. With this in mind, it is important to differentiate among multi-agent BCI or BCMI designs that require extensive practice to achieve the desired results and those that do not. It is perfectly possible and legitimate to design experiences that are not based on practice, which can generate enriching experiences for participants who do not bring particular kinds of experience to bear—for example in installation-based or audience participation works. Those that do require extensive practice are made with different intentions. Both offer fertile territory to explore.

To date, typical performances of Portable Gold and Philosophers Stones (Deviant Resonances) have proceeded in a kind of arch form, beginning simply with the holophone layer, then adding the other sections, one at a time, until a peak of musical complexity is reached. Then, one by one, the layers are often reduced in intensity and eventually deactivated, until an ending section is reached with textures reminiscent of the opening. Optionally, the electronics performer might play an auxiliary instrument to interact with the brainwave performers, particularly in the central section when the contingent delta threshold detections are active. For this purpose the composer has often used an electric violin to trigger analog circuits designed to exhibit somewhat unstable, chaotic behaviors (Fig. 4.8).

Fig. 4.8
figure 8

Photo by Tom Erbe

Image from a performance of Portable Gold and Philosophers’ Stones (Deviant Resonances) at Fleet Science Center in San Diego presented as part of San Diego Art Institute’s AMT (Art, Music, Technology) Festival in 2017. Rosenboom is seen behind active imaginative listening brainwave performers Susanne Thorpe (l) and Bonnie Jones (r), founders of TECHNE, an education organization emphasizing gender equity and social justice in arts and technology.

Full size image

4.3 The Experiment from Hopscotch

In the fall of 2015, an extraordinary opera, called Hopscotch–a mobile opera for 24 cars, was produced by The Industry, an opera production company in Los Angeles (The Industry 2015). In this extraordinary project, directed by Yuval Sharon, a story was presented in a non-linear fashion as audience members were driven around Los Angeles in 24 limousines along three different routes. Various scenes in the opera were performed inside each limousine and at specific, iconic locations in the Los Angeles cityscape. Audience members would experience the scenes of the opera in different orders, depending upon which route they were on and at what location they began their journey. Rosenboom was one of five principle composers commissioned to create music for various scenes in the opera. The Experiment was one of those scenes.

The Experiment was performed inside one of the Hopscotch limousines. In this scene, as audience members entered their limo, they heard spoken and sung explanations, accompanied by electronic backgrounds, of what they were about to experience, while individual brainwave monitors were affixed to their heads. As the scene unfolds, one of the opera’s principle characters, Jamison, pursues an obsession with understanding the nature of consciousness by singing eleven questions to the audience that progress in nature from seemingly innocent inquiries to somewhat more confrontational probing. Concurrent patterns among the brain signals of the audience members are then detected with signal analysis techniques and used to gauge their collective responses to each question. The results are translated into an immersive mix of soprano voices that sing three possible answers for each question with different musical qualities, representing: (1) an agitated state, (2) shifting attention or alertness, and (3) being focused on one’s inner self and disinterested. These were presumed to come from the inner group psyche of the audience. In the end, instead of finding the answers he seeks, Jamison snaps. After an extraordinarily successful performance run of Hopscotch in 2015, Rosenboom created a concert version of The Experiment, a recording of which is available (Rosenboom 2019a), and a score for which is published online (Rosenboom 2015). Writer Erin Young wrote the texts. A relatively detailed technical description follows.

Raw EEG signals from four audience members are recorded with Muse™ brainwave monitoring headbands and transmitted to a computer via Bluetooth. The signals are received via the Muse–I/O program and sent to software written with the Reaktor Core digital signal processing and synthesis platform using the OSC (Open Sound Control) protocol. The raw signals from each audience member are parsed into three commonly used, brainwave frequency bands: theta (5–8 Hz), alpha (9–13 Hz), and broadband beta (14–30 Hz). Amplitude envelopes of the resulting twelve bands (three for each of four audience members) are detected with variable time constant, low frequency envelope followers. The four envelopes corresponding to each EEG filter band are then averaged to produce collective audience envelopes for theta, alpha, and beta EEG frequencies.

Prior to the performance, eleven sets of vocal parts, corresponding to each of the character Jamison’s eleven questions were recorded by a soprano. Within each set, three kinds of answer texts were also set for the solo soprano voice and recorded. All the recordings were stored in a computer. In a performance, after Jamison sings a question, he pauses. During the pause, the averaged audience EEG frequency band envelopes are used to control the playback amplitudes of the prerecorded answers to that question, sung by the soprano. The dramatic operatic result is that the brainwaves control an audio mix of the three types of answers for each question, as specified in the narrative: theta controls the singing related to an imagined condition of being focused on one’s inner self and disinterested, alpha to shifting attention or alertness, and broad-band beta to an agitated state. A performance then proceeds through the sequence of eleven sung questions and eleven mixes of multiple soprano voices (all recorded by a single vocalist), the qualities of which are modulated by the collective brainwaves of the audience members (Fig. 4.9).

Fig. 4.9
figure 9

User interface showing software control panels for The Experiment

Full size image

What has been described thus far is an example of non-contingent, multi-agent feedback. However, a contingent, multi-agent feedback mode is also included in The Experiment. The alpha frequency amplitude envelopes are also connected to a delta function detector, much like that described above for Portable Gold and Philosophers’ Stones (Deviant Resonances). If the absolute value of the rate of change of the envelope signal crosses a settable threshold, a trigger signal is generated. A hold time can also be adjusted to determine how much time must pass after detections before subsequent detections can be made. The system will respond both to the onset of rapid alpha bursts and to the sudden interruption of alpha bursts, depending on the hold time selected. Finally, the system tests to see if triggers from all audience members occur at the same time. If they do, the triggers from all four are simultaneous, then a special electronic chord is sounded over the mix of the soprano voices. This chord is meant to signal the possibility of a simultaneous shift in the group mind of the audience, whatever that might ultimately mean in their experiences.

4.4 Concurrent Complexity

Uncertainties in stimulus detection, uncertainties in response measurements, and uncertainties in the generation of feedback responses exist at some level in all interactive systems, in both research setups and interactive art works. Deviant resonances emerge in all systems of differentiation among presumably identified component parts: in the instruments of technology, natural organisms, energy fields, time and space, to name a few. Often, the space in which deviant resonances emerge is where the greatest interest lies, in both the making of art and in the refining of theoretical models. Often new understandings of complexity emerge from deviant resonances. As many master musicians have often noted, the real interest in music lies not in the notes, but in the spaces between the notes.

At this stage in the development of multi-agent BCI in the arts, it may be productive to approach biofeedback and related pursuits as interactions among complex systems. In propositional music, it can be useful to describe musical states as particular behavioral interactions among networks, for example among a brain/proprioceptive system and an artificially intelligent musical instrument. These musical states become differentiable interactions located along scales for comparison, just as if they were musical notes. One may compose with these states as notes (Fig. 4.10).

Fig. 4.10
figure 10

Concurrent complexities among networks as musical states

Full size image

One interesting subject high on the agenda for future development in the authors’ work is investigating ways to correlate the complexity of a stimulus environment with the complexity of EEG signals and apply the results in feedback paradigms. Already, preliminary evidence indicates that correspondences among the dimensional complexity of brain activity and the complexity of music stimuli may be observable (Birbaumer et al. 1996). Previously, Rosenboom has experimented with complexity measures applied to musical parameters in some compositions (Rosenboom 1992, 1996, 2000a). In order to make progress in this realm, more work is needed on how to refine the meaning of complexity and ways to measure it. This is a common goal in many fields now, where analysis of complexity is of interest. Also important to this is continuing work on refining paradigms for sonification of biological data in feedback paradigms, especially where the phenomena of interest are emergent (Choi 2018; Novello 2012; Rosenboom 1997, 2003, 2014).

4.5 MoodMixer

The MoodMixer project was created in 2011 by Grace Leslie and Tim Mullen and explores new possibilities for multi-agent BMCIs that respond to, and influence, the mental state of multiple participants. There are three distinct versions of the system. The first of these was presented at the 2011 New Interfaces for Musical Expression conference in Oslo, Norway, with subsequent realizations presented at various venues between 2012 and 2014. The project is described in detail in (Leslie and Mullen 2011) and (Mullen et al. 2015).

MoodMixer employs a non-contingent audiovisual BCI system which reflects, and expands on, elements of much earlier works described in this chapter, particularly Humbert’s 1974 Brainwave Etch-A-Sketch. Two normalized cognitive (e.g. focused attention, relaxation) or affective (e.g. arousal, valence) state indices are simultaneously and continuously calculated from each participant’s EEG. These define a set of coordinates within a two-dimensional mental state space. The locations of all participants in the state space determine the evolution of a music composition, either through a dynamic spatial quadrophonic mix (MoodMixer 1.0 and 3.0) or an algorithmic composition procedure reminiscent of John Adam’s Phrygian Gates piano piece (MoodMixer 2.0). In each of its three instantiations, a visual display also provided real-time feedback on the participants’ individual and/or combined states. MoodMixer explores concepts of both collaborative and competitive, as well as active and passive, approaches to real-time EEG-based music generation within a multi-user design that promotes social interaction in the experience of the installation (Fig. 4.11).

Fig. 4.11
figure 11

Architectural diagram of the MoodMixer installation and its typical dual-user quadrophonic instantiation. A two-dimensional neural state-space is explored simultaneously by two or more users. A1–A4 represent four dynamically mixed audio tracks each composed to reflect an extremum of the state space. For MoodMixer 1.0 and 2.0, the users’ positions in the state space are visually represented by a moving dot superimposed on a weighted sum of four colored spatial gradients. *For MoodMixer 3.0, this video mapping was replaced with dynamic blending of video footage from Four Stream Mind by Grace Leslie and Maxwell Citron

Full size image

4.6 Assembly Cognogenesis

Assembly Cognogenesis is a multi-agent BCI work created by Sheldon Brown and his lab at the Arthur C. Clarke Center for Human Imagination, in collaboration with Tim Mullen. Assembly Cognogenesis is a shared virtual reality environment in which two users use neural and gestural interfaces to collaborate within an artificial life world and cultivate the symbiotic relationship between imagination, engagement, and the evolving environmental system. A first version of the installation debuted in 2015 as part of the Mozart and the Mind festival in San Diego with a subsequent version shown at the Filmatic festival in 2016. A short demonstration video can be found in (Brown 2016).

Assembly Cognogenesis is based on the Assembly emergent behavior platform created by Sheldon Brown and his lab (Brown 2015). In Assembly, collections of entities evolve over time in relationship to their environment and each other, with guidance provided by one or more viewers. In Assembly Cognogenesis, both contingent and non-contingent BCI elements were added to the system. Two participants are situated within a common environment, but with each person occupying a vastly different spatial scale. EEG power spectral measures associated with attention or engagement are calculated for each participant. Hand movements and gestures are tracked, enabling gestural interaction within the virtual environment. In one instantiation, the environment consists of a unicellular amoeba-like organism inhabited by hundreds or thousands of ‘molecular’ entities, which interact and evolve according to a set of rules akin to the chemistry and physics of a real world environment. One participant resides outside the organism, at the “macro” spatial scale, and can manipulate the shape of the amoeba using their hands. An energy field surrounding the organism increases in intensity proportional to the participant’s engagement or attention measure. When this measure is sufficiently large, the participant can direct energy to specific locations within the organism. The second participant resides inside the organism, at the “micro” spatial scale, and can interact directly with the entities. When this participant’s neural measure is sufficiently large, they can use gestural interactions to channel energy available at their location into one or more entities. This causes the selected entities to reproduce and evolve at an increased rate, propagating these entities’ traits and enabling new variations to emerge. New possibilities within the environment thereby only emerge when a sufficient degree of attention or engagement is contemporaneously attained by both participants. In one variation, participants could also cooperate to evolve and guide the entities to where they could pass through the organism’s membrane and transit outside the organism, at which point the environment would reset and participants would reverse roles.

In Assembly Cognogenesis, the participants must learn to cooperate across two vastly different levels of description, by observing, and responding to the effect of each other’s actions on their shared environment. In doing so, they must maintain a common mental representation and goal structure, while also maintaining a common neural state. Contingent feedback facilitates maintenance of this common state, enabling life to evolve and new creative possibilities to emerge within the virtual world.

Assembly Cognogenesis incorporates a number of cutting-edge technologies including virtual reality, hand and gesture tracking, unobtrusive wearable EEG sensing, and (for MATM and Filmatic exhibits) real-time distributed computation of neural measures in the cloud using NeuroScale™. These afford possibilities for immersion, mobility, and scalability that would have been impossible twenty years ago. Nonetheless, one must appreciate the thematic parallels between this work and the earliest historical multi-agent BCI systems described in this chapter—for instance Rosenboom’s Alpha Checkers. Between these and many other works spanning nearly five decades, contingent feedback plays a similar and central role in establishing and coordinating interaction and cooperativity between multiple agents. From these contemporaneous interactions, interesting new emergent behaviors and perspectives may arise as individuals within the group learn to function as a cohesive unit and indeed become more than one (Fig. 4.12).

Fig. 4.12
figure 12

Photo by Tim Mullen

Image from a public exhibition of Assembly Cognogenesis in Sheldon Brown’s lab at UC San Diego’s Arthur C. Clarke Center for Human Imagination as part of the 2016 Filmatic festival. The participant on the left is occupying the macro-scale position outside the organism, while the participant on the right is occupying the micro-scale position inside the organism. The two views of the environment can be seen on the respective displays.

Full size image

5 Conclusions

The highly interdisciplinary terrain in which multi-agent BCI in the arts and multi-agent BCMI reside is populated now by a growing and wide-ranging field of practitioners, who are exploring very interesting phenomena, making stimulating works of art, and illuminating a landscape of artscience investigations. It has not been the purpose of this chapter to provide a survey of this work. Rather, by concentrating on examining the environment within which some of these ideas emerged, particularly those focused on the multi-agent concept, and describing a few examples of recent realizations, the authors hope to contribute to the growing literature guiding the evolution of this field.

The multi-agent artistic BCI and BCMI systems we have described involved relatively few agents. However, the availability of cost-effective, wearable technology for measuring brain and body activity (Liao et al. 2012), as well as the emerging use of mobile computing, and scalable cloud and fog computing for BCI (Zao 2014a, b; Intheon 2018) create new possibilities for large-scale multi-agent artistic BCI systems. At the Regen3 event in 2003, alpha-band EEG activity from 48 participants was simultaneously measured and used to control musical parameters of a jazz performance (Mann 2007). The My Virtual Dream installation at the 2013 Scotiabank Nuit Blanche arts festival in Toronto (Kovacevic et al. 2015) situated groups of 20 participants at a time (a total of 523 active participants over the duration of the 12-h event) within an 18 m geodesic dome accompanied by 360º projections of dream-like artistic visuals and soundscapes driven by the collective brain activity of all participants. In a 2017 event organized by Terra Mater Factual Studios, EEG activity from several hundred individuals in two movie theaters in Los Angeles and New York City were simultaneously measured, decomposed into spectral components and correlated in near real-time across the group using cloud computing. These measures were in turn used to drive a visual feedback display telecast to the audience members. These are but a few examples of emerging possibilities for multi-agent BCI systems as biosensing and computing technologies continue to advance.

Clearly, the idea of multi-agent BCI in the arts stimulates the imagination and suggests potentially rich paradigms for both disciplined research and imaginative exploration. Indeed, perhaps this is a place where science and art can meet in significant theoretical territory and in new avenues for materializing ideas. More generally, one can view artistic and scientific domains as complementary systems for investigating and understanding the nature of reality. Art provides a propositional “sandbox” in which one can freely explore what is possible—a realm in which new concepts and systems can be flexibly created and prototyped, without necessarily demanding a rigorous explanatory foundation, or even physical realization. Science can expand on such concepts to develop and rigorously test hypotheses, produce empirical evidence regarding what is probable, and ultimately enable the realization of some of these concepts within the physical world. Art in turn, can leverage scientific knowledge and discovery as a basis for further ideation and exploration, creating a virtuous cycle.

Thus, art and science support each other as co-evolving forms of practice and discipline. In most arenas of human exploration, balances shift among how propositional and empirical modes of speculation and verification are emphasized, a normal response to how any given era sharpens its focus on goals, aspirations, and practical needs. Decades ago, as imaginations were fueled by new waves of discovery in the fields discussed in this chapter, a spirit of futuristic optimism emerged that might be captured by this quote from Biofeedback and the Arts, “Through the use of computers as appendages of man’s brain and methods of learning with biofeedback, rates of information processing will be achieved that approach the speed of light, ergo, conception will be bound less necessarily with action, elicited or observed, and life will eventually be embodied in information-energy networks creating non-physical art; spiritual art will be revived as established networks connect us firmly.” (Rosenboom 1972).

In subsequent decades, scientific research has provided us an increasingly clearer understanding of the means to extend human cognition and communication beyond the central nervous system using neurotechnology. Advances in electromagnetic sensing and stimulation, optical physics, nanotechnology, and biocompatible materials are yielding new possibilities for measuring and modulating brain activity at far greater spatiotemporal resolution than previously possible. We have continued to increase our understanding of both the practical utility, as well as the limitations, of various forms of bio/neuro feedback and closed-loop neuromodulation. Although we have yet a great deal to learn about brain structure and function, and the neuroscience and neurotechnology fields are still relatively embryonic, artistic applications of neurotechnology provide a means for us to envision, explore, and discuss possible roles and implications for such technology within present, near future, and far future societies.