Neurodesign Live

Abstract

Neurodesign is a novel field of research, education and practice that emerges as a cross-disciplinary initiative. In 2019, the Hasso Plattner Institute (HPI) offered for the first time a neurodesign curriculum. The objective of neurodesign as we pursue it is to explore synergies at the intersection of (i) neuroscience, (ii) engineering and (iii) design thinking · creativity · collaboration · innovation. In this chapter, we share insights into the development of a curriculum that quickly became more comprehensive than we had anticipated for this initial implementation phase. Neurodesign evolves serendipitously driven by the passions of numerous protagonists who contribute their expertise, ideas and work results in a uniquely collaborative fashion. The chapter briefly summarizes input provided by neuroscientists and creative engineers from several countries and different continents, who contributed guest expert talks at the HPI to help build up a joint knowledge base. The major part of the chapter is a review of neurodesign projects that have emerged, often in collaboration with guest experts of the program. Overall, these projects indicate how intersections of neurodesign (i)–(ii)–(iii) open up cornucopias of opportunities. Especially the integration of engineering expertise has introduced many favourable dynamics. In terms of strategic reflections, this chapter shares “missions” we pursue in the development of neurodesign. These directions for further initiatives also commence a brief outlook on upcoming neurodesign developments.

In October 2019, the Hasso Plattner Institute (HPI) started to teach and practice neurodesign. It was an “experiment” and we expected to start small. Out of passionate reactions of people who got involved, the movement quickly grew much bigger than we had anticipated in the beginning.

The bodily basis of design thinking had been a side-topic in research for a while. At the HPI d.confestival 2017, a podium discussion explored the topics of Neurobiology and Design Thinking (HPI 2017). A year later, along with an HPI-Stanford design thinking research meeting, a symposium addressed the topic of Neuroscience and Physiological Perspectives on Design Thinking and Creativity (HPI 2018). These were already collaborative events, with researchers from the HPI, Stanford and other academic partners such as the Marconi Institute for Creativity from the University of Bologna involved. Yet, these events span just a couple of hours each.

In parallel, at Stanford Allan Reiss, Manish Saggar and their research teams pioneered empirical investigations into the biological basis of design thinking creativity (e.g., Saggar et al. 2016; Xie et al. 2019). At the same time, studies into the history of design thinking revealed that extensive reflections on the (neuro-) psychological processes of creativity in engineering design informed the design thinking approach since its earliest beginnings (Clancey 2016, 2019; von Thienen et al. 2017, 2021).

The movement gained impetus when Larry Leifer, founder of the Center for Design Research at Stanford and head of the Design Thinking Research Program at his faculty, coined the term “neurodesign” as a headline for promising avenues in the development of design thinking. Larry was serious about the vision of design thinking as neurodesign. He found Jan Auernhammer as a passionate companion, who became Executive Director of the Leifer Neurodesign Research Program at Stanford and soon hosted a first neurodesign symposium aligned to a Stanford-HPI design thinking research meeting in March 2019.

Thrilled by this joint passion for an otherwise rare topic combination and emerging new perspectives, at the HPI Julia von Thienen and Christoph Meinel decided to bias to action and launch a neurodesign course in what was then the upcoming winter semester. Julia wrote an extensive manuscript for a neurodesign lecture she wanted to hold, with long abstracts for each planned session and literature references. To ensure that the topic selection, course messages and references would be up-to-date, she sent her manuscript to colleagues and friends with a background in the neuroscience of creativity or collaboration, to ask for additions as well as critique. Reactions were very different from what had been expected. Instead of commenting on manuscript details, colleagues rather expressed their high level of excitement about the topic and interest in getting actively involved.

Caroline Szymanski was the first to get on-board. With a PhD in the neuroscience of collaboration and many years of involvement at the HPI D-School, she was elated that two major passions in her life, which previously seemed to require all too disparate work, might find a fertile academic home. Julia and Caroline decided to rethink the lecture. Instead of a conventional format with a single teacher, the course should bring experts of pertinent topics together, so that everyone could contribute insights from his or her special field of expertise. All colleagues to whom we reached out immediately agreed to come—from various countries and even different continents—to discuss their topics in the neurodesign lecture, even though there would be no monetary remuneration for the engagement. Soon Julia and Caroline even had to denounce favourite topics they had planned to present in the lecture series themselves, because session slots were needed for other colleagues who were ready to come.

As another HPI colleague, Joaquin Santuber quickly became engaged. He had co-taught courses on design thinking in digital engineering together with Jonathan Edelman at the HPI, where they had made favourable teaching experiences with the use of physiological sensors to augment design. In joint discussions, pedagogical and methodological aspects of neurodesign soon came into focus. What is it that HPI students can learn and contribute in this novel field of neurodesign? Why would this particular student audience be interested in the topic? HPI students are digital engineers. At the HPI—akin to Stanford—, design thinking education is embedded in engineering education. Thus, for us it makes sense to explore and develop neurodesign as a field at the intersection of three domains: engineering, neuroscience and design thinking (Fig. 1).

Fig. 1

Neurodesign fosters explorations and contributions in three domains—especially at the intersections of fields: (i) engineering, (ii) neuroscience and (iii) design thinking · creativity · collaboration · innovation

Notably, the field of neuroscience is extremely rich in (digital) engineering tasks, so that engineering skills are in very high demand at neuroscience labs. Conversely, physiological measures and research methodology common in neuroscience and related fields can be very practical tools for engineering projects as well. For instance, usability studies of engineers can benefit from physiological data captured during the tests. Thus, physiological research methods can facilitate successful design thinking in engineering. Conversely, engineering skills can facilitate the neuroscientific-physiological understanding of design thinking, creativity, collaboration, innovation, design, or any other topic neuroscientists may turn to. Yet, all this hinges on engineers thoroughly understanding neuroscientific methods.

Against this background, we soon decided to offer two neurodesign courses, not only one. Next to the lecture, there should also be a more hands-on seminar where students could learn neuroscientific and related research methodology. The seminar should cover topics of study planning, the analysis of EEG and fMRI data and practical work with peripheral physiological sensors.

In terms of teaching experiences, Julia had a record of teaching study design and other methodological topics for social-science students. Joaquin had been conducting research with, and had taught the use of, physiological sensors in engineering design. To complement them, Irene Sophia Plank joined in from the Berlin School of Mind and Brain at Humboldt-Universität zu Berlin, to contribute her expertise in EEG and fMRI methodology by co-teaching the seminar.

Would there be only engineering students in the courses? Neurodesign was envisioned as a melting pot for ideas and expertises. Opportunities in neurodesign hinge on the integration of different academic disciplines. A limitation of audience backgrounds to engineering seemed to make no sense. We reached out to the Psychology Department of Potsdam University. Within a few days, Ralf Engbert and Martin Fischer—in charge of the master program “Cognitive Science—Embodied Cognition” replied with a concrete vision of how knowledge exchange in the form of teaching could happen across university faculties. They agreed to open up the HPI neurodesign courses to their students. Possibly, in subsequent semesters there could even be courses at both faculties, at the Digital Engineering and the Psychology Department, where our students could meet, work together and earn credit points for their studies.

Thus, two courses on neurodesign—the lecture and the seminar—commenced in October 2019, which were open to students from differing backgrounds. The courses were taught by a multidisciplinary teaching team who collaboratively improvised ahead based on what seemed to work well and what seemed to be needed from one week to the next.

Once again, the reception of topics went a way beyond anything we lecturers had anticipated. Suffice it to say that the passion of participants initiated a wealth of projects apart from grading, which students and staff pursued to a large extent in their leisure time. One example is a workshop on the sonification of EEG data, which neurodesign guest lecturer Chris Chafe offered at the Technical University of Berlin (TU) several weeks after his talk at the HPI. Here, students and staff from various courses and chairs at the HPI, some of Chris’ students from the TU and the University of Arts (UdK), neurodesign guest lecturers affiliated with Humboldt-Universität zu Berlin and some colleagues from other institutions came together (Fig. 2). We exchanged data, experimented with sonification approaches and agreed on projects conducted in loosely collaborating cross-institutional teams.

Fig. 2

As a follow-up event of his HPI talk, Chris Chafe offers a hands-on workshop on data sonification in Dec 2019 at the Technical University of Berlin. It is attended by students and staff from the HPI, neurodesign guest lecturers from Humboldt-Universität zu Berlin, and colleagues from the Technical University of Berlin, the University of Art at Berlin as well as the Beuth University of Applied Sciences Berlin. Cross-institutional collaborations gain momentum (photo by Julia von Thienen)

A second example of projects emerging from the passion of students concerns neuroscientific lab assessments. Having discussed digital signal processing in fMRI, and having analysed fMRI data in class, students expressed the wish to personally conduct some fMRI assessments to understand the procedure even better. Thanks to the personal engagement of Irene, this soon became possible at the Berlin Center for Advanced Neuroimaging (BCAN) at the Charité (Fig. 3).

Fig. 3

Out of interest for the subject, HPI neurodesign course participants come together to conduct fMRI studies in their leisure time. Such assessments are kindly rendered possible by neurodesign lecturer Irene, who organizes and supervises the undertaking. As a study topic, the processing of code is addressed, so that HPI students can also be good fMRI test subjects (photo by Julia von Thienen)

Moreover, driven by further suggestions of seminar participants, fMRI scans should elucidate brain activities of digital engineers while processing code. At the HPI, already behavioural research had been conducted in the field. Christian Adriano at the chair of Holger Giese had investigated in his PhD research how programmers process and recognize “errors” in source code. At the chair of Robert Hirschfeld, a research group including Patrick Rein, Marcel Täumel and Jens Lincke had studied parallels and differences of people processing natural language vs. code. Christian and Patrick immediately gained interest in brain activities that would occur when people tackle their experimental tasks in the fMRI scanner. In rapid work, their available stimulus material was adjusted so as to become more suitable for fMRI scans. Subsequent brief pilot assessments yielded insights for potential follow-up studies.

In one part of the fMRI assessment, natural language was compared to code (Smalltalk) in a passive reading paradigm (Fig. 4). In another part of the study, participants were asked to find errors in Java code (Fig. 5). Both paradigms activated areas associated with visual processing. In upcoming neurodesign courses of the next semester, the topic of designing stimulus material specifically for neuroscientific tests has been included in the curriculum, so that possibly the same research questions can be addressed again in subsequent fMRI or EEG assessments with iterated stimulus material.

Fig. 4

Areas activated more strongly during passively reading Smalltalk code compared to a description of the same content in English. The increased activation of visual areas when reading code is probably due to the stimulus material used in this preliminary assessment, where code stimuli tended to be longer and more complex than the content formulated in English

Fig. 5

Areas activated in the first 10 s of attempting to detect errors in complex Java code. The activation of visual areas is probably due to the complex visual nature of the code, which spanned several lines

As another unexpected development, at the HPI a third class began to work on neurodesign topics. Matthias Bauer and Christoph Meinel jointly offered a web-programming class. Here, students could choose between several projects they wanted to work on over the semester. Julia had submitted one project invitation to this class bearing on the development of an online test platform for creativity tests. This project call complemented several other project invitations, amongst which the students could choose. Serendipitously for us, all students ended up voting in favour of the neurodesign project, so that the whole class came to pioneer web-programming of a neurodesign online test platform. Matthias and Julia were very impressed by the large-scale and impactful results the students engendered, working to a large extent self-organized on this project (cf. Sect. 2).

Beyond student projects, also the core academic research and teaching team evolved quickly in the course of just one semester.

First of all, Theresa Weinstein discovered neurodesign as a promising area of in-depth work, where she could bring together her formerly separate fields of study. She had completed the basic and advanced track at the D-School and had worked as a design thinking coach; thus she knew design thinking very well from practical perspectives. She had also worked as a student assistant in design thinking research with Martin Schwemmle for more than two years, reflecting her interest in academic studies on creativity, collaboration and innovation. Beyond that, she had studied social and cognitive neuroscience at the Berlin School of Mind and Brain, Humboldt-Universität zu Berlin. Theresa became the first PhD candidate in neurodesign at the HPI.

Babajide Owoyele, educated as an engineer at the Technical University of Berlin, who is a PhD candidate in design thinking research with Jonathan Edelman, also found neurodesign a fruitful area of work, for which he has unique visions. In particular, Babajide gives thought to the uniqueness of the (digital) engineering environment where neurodesign evolves. He notes how creativity and collaboration tests are often designed to be relatively domain-general, whereas ideally we should develop domain-specific assessments as well, especially for the context of digital engineering. As an analysis approach that invokes an embodied cognition perspective, he pioneers gesture studies in creative engineering teams, exploring the use of contemporary technology such as the Microsoft Kinect to capture and understand anonymized posture and gesture-speech data. This is a line of research that certainly can gain further impetus as neurodesign develops towards radically making gesture and speech research accessible to creativity and design research communities.

Shama Rahman, too, gained great interest when she learned about neurodesign, as she was invited to contribute a guest expert talk in the lecture and seminar series. Shama had lived in London for two decades, where she had pursued a variety of passions in seemingly disparate domains. She had studied molecular biology and later obtained a PhD in complex systems mathematics and physics by studying neuro-psycho-physiological processes of artists in a creative “Flow” mental state using EEG. Further interests in making positive impacts in peoples’ everyday lives yielded entrepreneurial initiatives. Here, her passion for digital engineering obtained a key role as well. In particular, Shama is CEO of NeuroCreate, a start-up that invokes Artificial Intelligence to facilitate human creativity. NeuroCreate coined the term of ‘Augmented Intelligence,’ referring to symbiotic designs where AI and humans interact in ways that are well-informed by the underlying neuroscience of creativity. In terms of topics, this engagement brings together works in the fields of deep learning, sensor technology, software development, user experience, creative and collaborative productivity, mental health and wellbeing studies as well as economical concerns. Beyond this already wide spectrum of activities, Shama also pursues artistic passions, such as playing and composing music, acting, staging storytelling and other forms of immersive experiences, theatre, games, installations and salons. Shama immediately noted that all her seemingly so variegated activities fell into the fields (i)–(ii)–(iii) of neurodesign. Shama recognized: She was a neurodesigner, and she had been a neurodesigner for decades.

In the course of the semester, Shama came not only for a single lecture, but she came for a whole week. Her work topics were so interesting to the HPI audience, that even two student teams, not only one, came to work with her intensely in the course of semester projects (Fig. 6). Everyone involved worked so thoroughly, with a time investment much beyond what is common in classes, that both student projects came to be presented at an international creativity conference (McKee et al. 2020a; Adnan et al. 2020). Seeing this large potential of neurodesign to bring about highly novel and worthwhile solutions in the areas (i)–(ii)–(iii) of neurodesign, Shama is now ready to move from London to Potsdam/Berlin to deepen her engagement at the HPI.

Fig. 6

In a D-Flect talk at the HPI, Shama discusses the potential of Artificial Intelligence to facilitate human creativity and collaboration (photo by Stefanie Schwerdtfeger). Together with Felix Grzelka and Holly McKee (both HPI), she invites the audience to take part in a study where this can be probed. Participants will conduct AI-assisted brainstorming sessions while their EEG is recorded with consumer-grade technology, as with a wearable EEG-sensing headband Shama is wearing in the image

In what follows, we will briefly summarise some content of neurodesign education, presented by various lecturers in the last semester. Thus, we want to help readers gain an impression of topics that were discussed intensely over the past months. They contribute to a core evolving knowledge base of neurodesign (Sect. 1). We will then review neurodesign projects that have been conducted so far (Sect. 2). Most of them were based at the HPI. Yet some others already emerged at different institutions in the wider area of Berlin-Potsdam, reflecting a great collaborative spirit in this geographical region at the time. Indeed, the multiple universities concentrated in this area, next to pulsating artistic developments, seem to provide a rich and fertile ground for neurodesign. Section 3 reviews strategic missions we pursue in further developments of the field at the HPI. They also inform our outlook on upcoming developments (Sect. 4).

1 Neurodesign Education

Here, we provide a brief review of content and insights contributed by neurodesign lecturers in the course of the last semester. The overview begins with talks in the neurodesign lecture and then turns to teaching content of the seminar. Beyond the brief review shared in this chapter, all lecture talks are available in full length online at www.tele-task.de as part of the series “Neurodesign Lecture—Physiological Perspectives on Engineering Design, Creativity, Collaboration and Innovation (WT 2019/20).” There interested readers can find further details including literature references.

1.1 Introduction to Neurodesign

In the beginning lecture, Julia von Thienen (HPI) emphasises the importance of boldness and creativity in the exploration of new knowledge domains. To discover possibly fruitful connections across diverse and formerly separated fields of work, it is often important to hypothesize and dream up wildly novel contributions in the beginning. This initial activity can feel like following hunches and speculating wildly. Such bold explorations are, however, an important aspect of discovering and enculturating new work domains. More rigorous tests and criticism become important later in the process, when the objective is to see which connections are robust, and which work avenues seem most worthwhile. Together with the audience, Julia probes some potential points of convergence between the works of several neurodesign guest lecturers who come to present later in the semester. Altogether, her talk explores potential connections between six different topics in design thinking creativity research. These topics can also be looked at as “fragments” or “puzzle pieces.” One goal of neurodesign is to understand innovation beyond fragmentation (cf. neurodesign missions). This means to probe for connections and a bigger picture across multiple study topics in creativity and innovation research.

Julia begins with questions to the audience as to how children versus adults differ in their creative activities. She shares observations about children at a kindergarten age who spend many hours per day on creative pursuits: They indulge in imaginative play, make spontaneous creations in large numbers, curiously explore novel subjects and learn on the fly. At the same time, as creativity expert Giovanni Corazza from the University of Bologna notes, it seems that in the history of science and technology all the major steps taken by humanity were made by adults. How can these observations about children versus adults excelling in creativity be reconciled? Julia emphasises that clarifications are specifically important for design thinking, because this approach to innovation intentionally fosters and combines both, childlike as well as typical adult approaches, in creative projects. She also refers to an upcoming talk of Julia Rodríguez Buritica, where the audience will be able to learn more about information processing in children versus adults.

As topic (II), Julia reviews design thinking theory, which has evolved over multiple decades. Coherently, design thinking pioneers have described “two roads to creation,” one approach predictive of creative leaps, the other predictive of less novel, but highly sophisticated, polished and perfected outcomes. Descriptions of both approaches are condensed in the Sense Focus Model of Creative Mastery. When working in a sense-mode, people explore new ways of seeing, hearing, feeling and experiencing. They let loose, do what feels right, act spontaneously, humorously and playfully. They use unstructured approaches, follow their intuitions, impulses and curiosities. This approach is said to facilitate heightened creativity, including creative leaps. By contrast, in a focus-mode people engage in rational planning; based on domain-specific knowledge and skills they reflect, analyse and synthesize. They exert meta-cognitive control and meta-rationality; they follow structured approaches. Activity in a focus-mode is said to engender technically refined, sophisticated solutions. Julia highlights how work in a sense mode often occurs at low levels of cognitive control, whereas work in a focus mode typically invokes high levels of cognitive control. Thus, an upcoming talk of Mathias Benedek on the role of cognitive control in creative pursuits will be pertinent for a better understanding of contributions that can be expected from sense- versus focus-mode activity in creative pursuits. Julia also highlights how children often engage with the world in a sense mode, whereas adults more typically invoke a focus-mode.

As topic (III) Julia discusses a creative process model set forth by Tim Brennan. She highlights how this model can be seen as an overlay of two motion patterns, one being a straight path from A to B, the other being a yarn-ball motion path of walking around in an exploratory or search mode (Fig. 7). She continues by discussing differences in prototypical motion paths of children versus adults. Adults who want to go from A to B typically pre-plan the path and opt for the shortest distance, which is energy efficient. Reliably, adults reach the destination B. By tendency, children at a kindergarten age choose their walking paths more spontaneously. They engage in joyful explorations and make frequent “detours.” If left to themselves, young children may forget about the initial goal of reaching point B and thus never get there. Tentative patterns emerge—as hunches: Children, child-like approaches in creative pursuits, activity in a “sense mode” and free walking of a yarn-ball type appear to predict heightened creativity. By contrast, adults, typical grownup approaches in creative pursuits, activities in a “focus mode” and walking paths of straight rigid lines could be predictive of reliable goal attainment and less radically novel, but more sophisticated solutions. In terms of empirical studies, Julia reviews experiments indicating that free walking of a yarn-ball type does indeed engender heightened creativity, compared to walking along rigid lines. She also invites the audience to listen closely when Caroline Szymanski talks about interbrain synchrony during collaboration, as links between motion and collaboration will be discussed in her talk.

Fig. 7

Metaphorically, the creative process can be understood as combining two prototypical motion paths: the goal directed shortest distance path often taken by adults, and an exploratory path often taken by children

As topic (IV) Julia invokes the headline of “orientation.” She emphasizes that people need an initial cognitive map—a mental representation of an area—in order to pre-plan the shortest path from A to B, at least when B cannot be seen immediately. Such a cognitive map can also help to re-plan the path in case of obstacles, e.g. when a road is closed. Moreover, the cognitive map allows people to recognize what stimuli lie “off-track” and can therefore be ignored as irrelevant. By contrast, free walking of a yarn-ball type does not require an initial cognitive map regarding the terrain. People can approach anything that elicits interest and can try things out. Yet, people will build up a cognitive map gradually based on personal experiences in the field. In creativity theory, a concept akin to cognitive maps is prominently discussed, namely “conceptual spaces.” They are not maps of real geographical areas, but abstract maps of “work areas,” including “steps one can take” in the field. A hallmark of radical innovation is that it does not only add elements in established conceptual spaces, but rather entails a complete restructuring of earlier conceptual spaces. Julia hypothesizes that conceptual spaces might be encoded as cognitive maps in the brain, likely in the hippocampal formation and related regions. She moves on to review the role of emotions in the orientation process. Here, she highlights differences between pre-planned A-to-B paths compared to free walking. In a group of people, all individuals might have different emotions; still they are likely to select the very same A-to-B path based on a common cognitive map and efficiency optimization. By contrast, in the case of free walking, emotions likely play a key guiding role. Based on individual interests, people choose unique paths in the open terrain. In terms of upcoming lecture talks, Julia highlights that Laura Kaltwasser and Sergio Agnoli will also discuss the role of emotions in human behaviour.

Fragment (V) is headlined “attention.” Julia points out that motion paths straight from A to B can easily be carried out with a narrow breadth of attention and the focus rigidly attuned towards goal B. Other information can often be discarded as irrelevant (e.g., what plants there are in a park at the side). By contrast, free walking of a yarn-ball type likely thrives upon a wide breadth of attention, leaky filters, or altogether more flexible and varying attention: Anything in the environment can capture the person’s interest. Yet, from the perspective of goal-attainment (reaching B), the person may seem all too easily distracted as she attends to this and that without necessarily moving closer to B. Julia then reviews empirical research results. People with a wide breadth of attention and leaky filters show increased creative performance. In addition, attention is more selective in adults than in children. In further detail Julia also overviews research conducted by Axel Menning and colleagues at the HPI as well as Sergio Agnoli and colleagues at the University of Bologna. While methodologically very different, both lines of research suggest the importance of attending to seemingly irrelevant information in creative pursuits. All in all, a narrow and rigid focus of attention seems to hamper creativity. Such a rigid attentional focus is found in adults more commonly than in children, not least because a narrow and long-enduring focus of attention is enabled by cognitive control systems in the frontal lobe, which are not yet biologically matured in children.

As topic (VI), Julia reviews environments for creative work. She emphasizes that design thinking spaces are designed to facilitate free motion during work hours. An experiment on the impact of places has found that the design thinking environment including free-walking opportunities boosts creative performance compared to traditional seminar rooms.

1.2 Social Neuroscience and Teamwork

Caroline Szymanski (Max Planck Institute for Human Development & HPI D-School) introduces the topic of social neuroscience and discusses its relevance for neurodesign. In particular, social neuroscience examines how the brain mediates social processes and behaviour. As a discipline, social neuroscience dates back to the “Social Neuroscience Bulletin,” which was published quarterly between 1988 and 1994; it is thus a relatively novel field in itself. Social neuroscience research covers topics of cognitive science, neuroscience, psychology, cognitive neuroscience, behavioural neuroscience, affective neuroscience, behavioural genetics, psychophysics, philosophy, artificial intelligence, computational neuroscience and more. The main neuroscientific methodologies used are EEG and fMRI. Especially interesting for neurodesign is the social brain hypothesis, set forth by Dunbar and reformulated by Tomasello.

Human beings are able to pool their cognitive resources in ways that other species are not […] made possible by a single very special form of social cognition, namely, the ability of individual organisms to understand conspecifics as beings like themselves who have intentional and mental lives like their own. (Tomasello 1999, p. 5)

From this understanding, Tomasello developed the concept of ‘shared intentionality,’ which denotes the unique human ability of doing something together for the sake of doing it together. Shared intentionality occurs when several individuals attend to the same thing, and understand the situation on a meta-level. Shared intentionality therefore goes beyond joint attention in the important aspect that all individuals (know they) share the same intention. From a developmental perspective, this is the basis of people feeling as a “group member” and developing a “team-feeling.” Four- to nine-year olds already understand the concept of team membership as based on shared intentions, in contrast to “arbitrary” non-intention-based group membership, such as persons being grouped together based on gender.

Social neuroscience can help elucidate the physiological underpinnings of design thinking (i.e. collaborative creativity and innovation) by clarifying the neural basis of concepts such as team membership and collaboration. In terms of research outcomes, major ‘social brain networks’ discovered so far are associated with the frontal cortex. They include in particular: (A) the reward & motivation network, primarily located in prefrontal brain regions, namely the orbitofrontal cortex, (B) the cognitive control network, mainly located in dorsolateral prefrontal cortex and (C) the social perception & attribution network, predominantly located in medial prefrontal cortex and parietal cortex.

In her lecture on social neuroscience, Caroline offers a variety of examples how literature concerning social-brain networks can help elucidate and facilitate design thinking teamwork. (A) The reward and motivation networks are especially important for collaborative creativity, insofar as social reward is a main causal factor that drives people’s engagement in collaboration. At the neural level, social reward has been shown to be at least as effective as monetary rewards. By contrast, social exclusion has been demonstrated to be processed by the brain like physical pain. Thus, the awe of social exclusion can even be countered with painkillers developed to antagonize physical pain, such as ibuprofen. (B) Cognitive control research also has specific implications for collaborative creativity. E.g., impulsive people tend to act less socially appropriate, but they are more creative when not restricted in their impulsivity. (C) The neural networks processing social perception information known so far can be mainly classified into empathy and theory of mind networks. They are all part of the so-called Default Mode Network (DMN), which is said to be concerned with feelings regarding oneself together with others and thinking about others. From a design thinking perspective, it is important to realize that understanding others’ intentions allows humans to (co)-experience how others feel, i.e. to empathically understand what it is like to walk in someone else’s shoes. Moreover, for collaborative teamwork it is helpful to realize that our brains can usually only do one thing at a time (i) focus on empathy and social cues (default mode network), versus (ii) focus on a specific task (task-positive network). Most commonly, only one of these two neural networks can be activated at any given moment. That is clearly an important insight with methodological implications for design thinking, which pursues major aims of empathic understanding.

All in all, social neuroscience and neurodesign are relevant to each other in two major ways: First, social neuroscience provides research insights on collaborative creativity that facilitate mindful design thinking/creative engineering practices. Second, knowledge concerning the biological basis of social interactions hinges on our technical capabilities of tracking relevant phenomena, and digital engineers can make important contributions in this domain.

1.3 Interbrain-Synchrony During Collaboration

In a second lecture, Caroline Szymanski reviews the subject of interbrain-synchrony during collaboration, which was her dedicated PhD research topic some years ago. There is converging evidence from numerous research studies that social interaction is characterized by synchronized brain activities amongst those persons who interact. This so-called “interbrain synchronization” has been found to be modulated by social context: Interbrain synchrony is more pronounced during cooperation than during competition. It is stronger in mother-child interaction compared to child-stranger interaction. Similarly, interbrain synchronization is stronger among lovers than between friends; least synchrony is found among strangers. Furthermore, interbrain synchronization has been reported to vary as a function of social dynamics: Lead-follow behaviour is markedly associated with differences in interbrain synchronization. A “natural leader” has the ability to take on other people’s perspectives, which is then reflected in increased interbrain synchronization during their interaction, which highlights a possible role for interbrain synchronization as a neural mechanism underlying team formation. In detail, Caroline talks about studies that find team performance to be associated with changes in interbrain synchronization. There is evidence that teams who “synchronize their brains more” also perform better as a team. Caroline closes by suggesting how inter-brain synchrony mediates between bodily synchronization and cognitive synchronization, and how this ‘triplet’ (body/brain/cognition synchronization) can be used to facilitate teamwork. For instance, joint motion helps brains synchronize, which in turn facilitates joint task attention. This, in turn, can improve team performance. Making explicit use of such mechanisms—(a) bodily synchronization to drive neural synchronization, to foster cognitive team convergence vs. (b) bodily desynchronization to reduce neural synchronization, to foster cognitive divergence—can well be used to facilitate design thinking or other forms of creative collaboration, by means of dedicated coaching interventions.

1.4 Sonification of Brain Data for Seizure Detection

Chris Chafe (Director of the Center for Computer Research in Music and Acoustics at Stanford University) speaks about the sonification of brain data. In digital engineering, often information is represented visually, e.g. with texts, graphs or other images. However, information can also be represented in other formats, such as acoustically. As Chris shows with a number of examples, the approach of data sonification is auspicious for different reasons. First, it creates a novel way for people to experience data. For instance, listeners can gain further insight into dynamics of climate change as driven by CO2 levels in the atmosphere. Here, much like background music in films, acoustically represented data has a unique potential to resonate with listeners on emotional levels, beyond cognitive understandings of relevant patterns. Second, body data sonification can be pursued for diagnostic ends. One example is the sonification of EEG data, which allows medical novices to detect silent seizures in patients with a high degree of accuracy. Third, there can also be an artistic use of data sonification, as exemplified by art installations based on brain data. All these examples are described in further detail below, as they have stimulated a multiplicity of neurodesign project work (Sect. 2).

1.5 Shared Responsibility in Collective Decisions

The talk of Marwa El Zein (Institute of Cognitive Neuroscience, University College London) explores the topic of collective work and in particular collective decision-making as opposed to single-person work and single-person decisions. This topic is highly relevant for design thinking, where leading experts often emphasise advantages of teamwork over single-person pursuits. Marwa’s review explores an even bigger picture, beyond individuals versus teams. Her perspective covers individuals, teams and also crowds of people so large that individuals do not even know each other. A key question is whether decisions and work results are better when people work individually versus collectively. It turns out: Who performs best—individuals or collectives—is highly context-dependant. This observation is already reflected in seemingly contradictory sayings, such as “two heads are better than one” (collectives get better results) versus “too many cooks spoil the broth” (individuals get better results). Thus, it is important to better understand the parameters when collectives outperform individuals. Marwa introduces the Wisdom of the Crowd Theory and the Jury Theorem, which already spell out several important conditions for groups to outperform individuals: Decisions to be taken are categorical and they have an objectively correct answer; all answers are formulated independently; moreover individuals opting for a decision (“voters”) must perform better than at chance level. Marwa moves on to discuss the importance of “competence similarity,” a factor that has been repeatedly found to determine if teams perform better or worse than individuals. Individuals with similar competence levels perform better when teaming up. However, teams where members possess different competence levels perform strikingly worse compared to their best team member alone. Marwa points out that this phenomenon is often interrelated with yet another factor: “confidence.” Teams almost universally weigh an opinion stronger when it is communicated by a team member who expresses a high degree of confidence in his stance, compared to an opinion communicated in an unconfident tone. Thus, methodologically, it seems key for teamwork that each individual is aware of his or her competence level, and communicates confidence accordingly.

Marwa further discusses different available methods for determining group decisions based on individual votes, and how these methods do not seem equally suited for delivering best overall outcomes. In the studies she discusses, best outcomes to problems with objectively correct answers are achieved when individuals (experts) are pooled into small independent groups; each small group gives their consensus decision and the average of these consensus decisions is taken as a final answer. This method of “small group voting” outperforms methods of individual voting and voting in a large crowd. It can be seen as particularly encouraging evidence for the “coopetition” approach often taken in Design Thinking, where several teams get to work in parallel on the same challenge.

Another key topic Marwa explores is the motivation of individuals to work in teams. While most research literature focuses on performance benefits of teams, Marwa explores in her research another beneficial side of teamwork, independent of performance, to which she refers as “shared responsibility.” She discusses in detail some of her studies, showing that individuals also join teams to share and thus minimize regret and disappointment in the case choices have negative consequences. Moreover, when working in teams, people become less influenced by anticipated regret and disappointment. This is another finding that can be highly relevant for teams working on innovation challenges. After all, throwbacks and resistance are common phenomena in innovation endeavours, and teams seem naturally more resilient to them then individuals.

1.6 Psychology of Design: Evolution of the Intersection of Two Inseparable Fields

Jan Auernhammer (Executive Director of the Leifer Neurodesign Research Program at Stanford University) explores the role and meaning of “neurodesign” at Stanford University. Stanford’s Design Group (formally known as the Design Division), including the Center for Design Research founded by Larry Leifer in 1984, has had a strong legacy in developing and researching design practices around the globe. Jan reviews the history of design thinking at Stanford, beginning with seminal works of John E. Arnold, Robert H. McKim and James Adams in the late 1950s and early 1960s. There have been strong lines of continuity at Stanford’s Design Group ever since. The Design Division and the Joint Product Design Program started by Arnold and McKim continuously explored synergies at the intersection of art, engineering, psychology, and business management. In terms of psychology, there has been an enduring concern for “human needs” and “human values.” Moreover, dedicated research has investigated creative abilities of people, and practices of engineering design teams over decades. As Jan points out, such interests in human needs, creative abilities and thinking in design have advanced content provided by psychologists like Joy Paul Guilford (a psychometrician), Abraham Maslow and Carl Rogers (both representatives of humanistic psychology). In these fields, nowadays, methods of neuroscience have the potential to add important understandings of details. In this sense, activities at Stanford Design have always focused on the intersection of thinking (psychology) and design (practice). Notably, John Arnold was a psychologist and an engineer. He brought psychological insights to design practice. Larry Leifer (head of the Design Thinking Research Program at Stanford) was one of his successors at the department. In his PhD, submitted in 1969, Larry worked on a topic at the intersection of Neurology (under supervisor Leon Cohen), Electric Engineering (with supervisor James Bliss) and Biological Science (with supervisor Donald Wilson). Thus, Larry is one of the very first “Neuro-Designers.” Based on this long tradition of work at the intersection of thinking and design, the Leifer Neurodesign Research Program aims to bring together Engineering Design, Neuroscience, Psychology, Cognitive Science, and other fields to advance human practices and abilities in design, as well as artificial intelligence through design. Jan emphasizes how collaborations with neuroscientists like Manish Saggar (Brain Dynamics Lab), Allan Reiss (Center for Interdisciplinary Brain Sciences Research) as well as other cognitive scientists and psychologists are important to research, in particular to advance an ever better understanding of thinking in design. Research needs to cover more than predefined cognitive tasks. It needs to elucidate open tasks of real-world complexity in design. It also needs to apply appropriate neurocognitive research techniques in real-world settings. For this endeavour of neurodesign research, funding, sponsors, and active partners will be key.

In this volume, Jan—together with his Stanford colleagues Neeraj Sonalkar and Manish Saggar—also explores the topic “NeuroDesign: From Neuroscience Research to Design Thinking Practice.” Here, they explore several gaps between the disciplines of Neuroscience and Design (Thinking). They also suggest a research agenda to bridge the gaps.

1.7 Attentional Mechanisms in the Creative Thinking Process: Insights from Psychophysiology

Sergio Agnoli (Marconi Institute for Creativity, University of Bologna) introduces the audience to basics of creativity research. How is creativity defined? What dimensions are important in the measurement of creative performance? Which factors are known to impact the dynamics of creative projects by facilitating or hindering creative performance? His further talk is guided by an “energy metaphor” in the study of human creativity. First of all, he discusses the architecture of the human brain, which yields a computing capacity around 38 petaflops (a quadrillion of floating point operations per second). By comparison, present-day supercomputers such as the Marconi Supercomputer (MS) have a processing capacity of ca. 20 petaflops. Remarkably, the human brain has an average energy consumption of merely 15 W per day, whereas the MS requires 3000 W per day. Thus, the human brain is an incredibly energy-efficient structure. One energy-saving strategy hardwired in the human brain is to develop “heuristics” for the interpretation of reality. Sergio shows a number of examples where study participants fail to notice odd events in the environment, because people only perceive what they expect based on prior knowledge. Overall, when people get trained on a task, the energy needed by the brain decreases rapidly, while the person’s task performance increases as rapidly. Sergio discusses a number of mechanisms how the brain achieves this kind of energy-efficiency and performance-curve. He explains how current research at the Marconi Institute is directed towards elucidating the energy expenditure of the brain in the creative process. For instance, in an EEG study participants were asked to generate alternative uses for everyday objects. Analysing time dynamics, the authors find initial answers of participants to be mere recalls from memory of already existing object usages; this is associated with low levels of energy expenditure. Afterwards, as participants think up more novel uses, the brain’s energy expenditure is increased; physiologically, synchronization in the EEG alpha frequency band is observed across brain areas. The causal impact of neural synchronization (in the alpha and beta frequency band) on creative performance is then established in a neurofeedback paradigm. Moreover, there is an “energy saving strategy” people can use by paying attention to a single element only, while disregarding surroundings. In another experiment, the authors presented everyday objects on a computer screen: one in the middle and many other objects ordered in a circle around it. Participants were asked to think up uncommon uses for the object in the middle. By means of eye tracking, the authors observed whether participants indeed only attended to the object in the middle or also screened objects around. Those participants showed increased creative performance who dedicated attention not only to the stimulus in the middle, but the surroundings as well. To complement these insights concerning “mechanisms” of creative thinking, Sergio discusses the role of peoples’ personality traits, motivations and emotions in channelling energy expenditure.

As part of his talk, Sergio introduces the Marconi Institute for Creativity, which indeed resembles the Hasso Plattner Institute and the Digital Engineering Faculty at Potsdam University in several notable regards. First, the Marconi Institute was co-founded by the University of Bologna and a non-for profit organisation, namely the Guglielmo Marconi Foundation. Similarly, the Digital Engineering Faculty was co-founded by the University of Potsdam and the HPI, which in turn is financed by the non-for-profit Hasso Plattner Foundation. Both Guglielmo Marconi and Hasso Plattner are successful, creative engineers whose non-for-profit foundations have a strong concern for the topic of innovation. Academically, the Marconi Institute for Creativity (MIC) resides also within a laboratory (the MIC Lab) in the Department of Electrical, Electronic and Information Engineering at the University of Bologna—thus at an engineering institute akin to the HPI at Potsdam or Stanford Engineering with its design thinking activities. The Marconi Institute pursues scientific research on the mechanisms underlying creativity. Similarly, at Potsdam and Stanford the Hasso Plattner Design Thinking Research Program sets out to investigate why and how design thinking · creativity · innovation works. The Marconi Institute offers creativity education at the university, and sometimes at schools, to multidisciplinary audiences. Similarly, the HPI and Stanford Engineering offer design thinking education at the university, and sometimes at schools, to multidisciplinary audiences. Finally, the Marconi Institute offers consulting services for companies in the field of innovation, like the HPI Academy at Potsdam. This multiplicity of activities is advanced by institute directors whose personal engagements radiate in numerous academic disciplines beyond single university faculties: Giovanni E. Corazza at the Marconi Institute (who holds a chair on Telecommunications at the University of Bologna), Christoph Meinel at the HPI (holding the chair of Internet Technologies and Systems), Uli Weinberg (professor at the HPI School of Design Thinking), Larry Leifer, Bernie Roth and David Kelley at Stanford (all three of them professors in Mechanical Engineering). As a minor difference in foci of work, the Marconi Institute is a bit more concerned with creative thinking processes of individuals, whereas the HPI and Stanford tend to emphasise and explore collaborative creativity.

1.8 We Feel Therefore We Are? About Emotions and Cooperation

Laura Kaltwasser (Berlin School of Mind and Brain, Humboldt-Universität zu Berlin) combines her talk with an on-site experiment. At the beginning of her lecture, volunteers put on Empatica E4 wristbands, to capture skin conductance and heart-rate measures over time. Laura announces that “something will be happening in class.” She will give a signal to the audience, then volunteers shall press a button on the wristband and later the physiological data will be analysed live in class.

Laura’s talk focuses on the role of emotion in social decision-making. The subtitle of her talk is “We feel therefore we are,” as an iteration of Descartes’ famous statement “I think therefore I am.” Laura says she is convinced that emotions play a fundamental role in our self-conception and also in actions we take. To convey the tight interrelation of both topics—emotions and social decision-making—she refers to a phenomenon called “negative reciprocity.” Here, people punish unfair behaviours of others at the risk of high personal cost. This typically occurs based on strong emotions. An example is the behaviour of soccer player Zinedine Zidane in 2006 during the world cup final. A player from the opposite team had insulted Zidane’s mother. Zidane punished this unfair behaviour by bumping his head against the other player’s chest. This was obviously an action at the risk of high personal cost. Zidane was shown a “red card.” His team, having one player less, lost the championship. It was the last official match Zidane ever played. The tendency of people to engage in such negative reciprocity has been found to vary strongly across cultures; it has been found least expressed in South America and more expressed in Eastern cultures, where it is often called ‘a culture of honour.’ A complementary concept used in research is “positive reciprocity,” which addresses altruism and pro-social behaviour. Again, studies find strong cultural variations as to how much people are inclined to show positive reciprocity. Thus, such patterns of social interaction seem to be strongly impacted by social norms, which differ from culture to culture. At the same time, patterns of interaction are clearly also depending on biological factors. Emotions, for instance, impact social behaviours and they do have marked biological foundations. For instance, people who are blind and people with average eye-sight show similar facial expressions in the case of joy, across all age groups. Thus, the bodily expression of joy seems to have strong biological, culture-independent components. Overall, emotions with their strong biological manifestations seem to play an important role in social decision-making. Laura refers to Antonio Damasio and his “somatic marker hypothesis,” according to which the whole body helps us take decisions. Subsequently, Laura discusses how emotion-related parameters are often assessed in research, based on pictures of faces depicting different emotions. She shows some examples. One slide depicts—in a large format—a fearful face. In parallel, a loud screaming sound shatters the lecture hall. Laura gives a signal and volunteers in the audience wearing an Empatica E4 wristband press the button, so that this moment can later be identified in the data analysis. Laura moves on to discuss brain processes that typically obtain when people see stimuli such as fearful faces. Having discussed the measurement of emotion-related parameters in research, she moves on to review measurement approaches for the study of social decision-making, as conducted by means of socio-economic games. To introduce her own research in the field, she reviews earlier studies. Empathy is often measured in terms of people’s accuracy in detecting other people’s emotions based on their facial expressions. Moreover, research has found that people who show more facial expressions themselves, especially more positive emotions, are also more cooperative. Thus, expressiveness in facial gestures has been discussed as a social signal to indicate cooperativeness. In detail, Laura reviews two of her own studies. She finds that people who show more pro-social behaviour in socio-economic games are specifically better at recognizing fearful faces of others. They also tend to express more emotions by means of facial gestures during social interactions, compared to less cooperative persons. Design thinkers can note how it may be important for teamwork to recognize emotions not just as something that concerns the individual team member and how they feel about something, but as crucial social signals that should not be missed, if the cooperative work is to be successful and effective. In another study, Laura and her colleagues find that pro-sociality and assertiveness are predictors of people engaging in negative reciprocity. This hints at different motives people can have for punishing unfair behaviour at the risk of personal cost. It can be an altruistic undertaking (society should not be unfair, I need to help establish fairness even if this has negative consequences for me). Negative reciprocity can also have more “egoistic” motives related to assertiveness. In her summary, Laura reviews several ways in which studies of emotion and social-decision-making could be improved. She invites audience members to think how respective parameters might be interesting to include even in Digital Engineering master theses. She also shows openly accessible databases, where videos and sounds labelled with regard to emotions can be found, so that they are available as stimulus materials for further studies.

After Laura’s talk, Joaquin Santuber shows samples of the data captured with Empatica wristbands during the session. The data of one volunteer is looked at more closely. This person showed a strong increase in electrodermal activity (a stress indicator) in response to the fearful face and scream presented in class. It takes several minutes before the value of electrodermal activity after the scream resumes the low level it had before the scream.

1.9 Creativity and Cognitive Control

In his talk, Mathias Benedek (Karl-Franzens-Universität Graz) addresses the subject of cognitive control in creativity. This is a subject of high relevance for design thinking, as design thinking theory has covered the subject intensely since the 1950s. Based on marked theoretical assumptions, design thinking has been developed as a practice where both processes with high levels and low levels of cognitive control are invoked, sometimes one after the other, but always in close interconnection. Examples of phases with high levels of cognitive control include the synthesis phase in creative processes, where structured methods such as analyses in 2 × 2 matrices are invoked. Examples of phases with low levels of cognitive control include experiential approaches, where design thinkers seek immersive experiences in a field to court spontaneous and intuitive insights. Design thinking theory concerning cognitive control is also discussed in greater depth in the chapter “Theoretical Foundations of Design Thinking, Part III: Robert H. McKim’s Visual Thinking Theories” in this volume.

Mathias begins his discussion of the subject with a review of the lives and works of famous creators. These provide evidence indicating that both—low levels and high levels of cognitive control—can be important for creative breakthroughs. For instance, the chemist August Kekulé developed an insight regarding the chemical structure of the benzene molecule in a dreamlike state, i.e. in a moment of low cognitive control. At the same time, analyses of the daily schedules of famous creators reveal that many of them followed rigorous agendas, with relatively fixed time windows of intentional and concentrated work: examples of creative activity with high levels of cognitive control. In order to clarify the concept of cognitive control, Mathias refers to the works of Daniel Kahneman and distinguishes between two kinds of cognitive processes. System 1 processes are automatic, unconscious, fast, undemanding, associative, undirected and spontaneous; they occur at low levels of cognitive control. By contrast, system 2 processes instantiate high levels of cognitive control; they are deliberate, conscious, slow, effortful, analytical, goal-directed and—obviously—controlled.

Mathias discusses how cognitive processes including creative thinking can be reconstructed in terms of (a) attention processes, (b) memory processes and (c) cognitive control processes. Regarding (a) attention, one major question has been whether it is more favourable to have attention constantly “on task,” or whether phases of devoting attention away from the task might also be favourable, e.g. to court incubation (the development of insights by means of non-conscious processing). Research indicates that short tasks are better solved with attention “on task;” here mind wandering does not seem to be helpful. Yet, more complicated problems can benefit from attention “off task,” i.e. from break times where the brain can continue to process the problem in non-conscious ways. From methodological perspectives, it seems important to engage in undemanding tasks during breaks, as these appear to impact chances of solving the problem in most beneficial ways. Another key distinction concerns “internally” versus “externally” directed attention. Studies suggest that internally directed intention facilitates mental simulation, such as imaginations, which are crucial for high levels of creative performance. In the EEG, this internally directed attention is indicated by increased alpha activity. Regarding (b) memory, Mathias discusses how semantic network analyses can be used to elucidate commonalities or differences between the memory structures of highly creative versus less creative individuals. Generally, when seeking novel ideas, people seem to begin with common ideas in the field and gradually come to think about more remote associations; highly creative individuals do this more quickly than less creative individuals. In addition, neuroscientific research suggests that recalling past events, imagining future events and creatively imagining novel events recruit relatively similar brain areas, underpinning the importance of memory for imagining novel solutions. Regarding (c) cognitive control, studies regularly find that high levels of intelligence including executive functioning (which allow people to act at high levels of cognitive control) predict people’s creative abilities and also their creative achievements. This argues in favour of cognitive control abilities being beneficial for creative performance. On the other hand, there is a lot of anecdotal evidence of people producing highly creative works in circumstances of reduced cognitive control, e.g. intoxicated with alcohol. Around 70% of the American Nobel Prize winners of the early last century had documented alcohol problems. This suggests a correlation, which does not yet answer questions of causality. Since then, empirical research has sought to clarify effects of alcohol on creative performance. Study findings have been variegated so far. Consistently, higher levels of alcohol intoxication cause reduced levels of cognitive control, especially in terms of reduced working memory functioning. Creative abilities do not seem to be affected consistently; methodologically careful double-blind studies suggest neither an increase nor a notable decrease of creative abilities under an intoxication of ca. 0.6 per mill. However, studies suggest that people find their own outcomes more creative when they believe to be intoxicated, regardless of whether people actually did receive an alcoholic or a non-alcoholic drink in the study. Another important line of research investigates contributions of two cognitive networks during problem solving. The Default Mode Network (DMN) conveys spontaneous, self-generated thoughts, such as mind wandering or episodic remembering (often occurring at low levels of cognitive control). The Executive Control Network (ECN) conveys goal-directed thought, including working memory functions and task-switching. Typically, these two networks show strongly anti-correlated patterns: When activity in one network increases, activity in the other network decreases. However, during many creative tasks the two networks show increased coupling. This may reflect a fruitful interplay of generative and evaluative cognitive processes, as required for effective creative work.

All in all, both information processing at high levels and low levels of cognitive control can facilitate creative breakthroughs. System 2 processes (of high cognitive control) help creators stay focused on a task; they allow people to overcome “dominant” ideas, which are merely uncreative recalls of already known solutions. System 2 processes also allow people to implement effective, goal-directed problem solving strategies. Thus, system 2 is highly relevant for active creative thinking. By contrast, spontaneous thoughts produced by system 1 (at low levels of cognitive control) allow people to process personally meaningful problems even when attention is off-task. In that case, automatic processing continues in non-conscious ways. Moreover, the working memory capacity of system 2 is highly limited; the sheer information processing capacity of system 1 is huge. Finally, goal-directed thoughts produced by system 2 can run into fixation. System 1 acts largely undirected, in an associative manner, and can produce highly surprising “out of the box” solutions: creative leaps.

1.10 Normative Aspects in Creativity, Collaboration and Culture Development

In her lecture on the role of norms in the context of creativity, Julia von Thienen begins with a remark concerning the overall topic selection in class. All talks in the first neurodesign lecture series discuss creativity and collaboration based on research with humans. This makes sense, in so far as there is a central concern for design thinking in class, which is a human practice. At the same time, creativity and collaboration can also be observed in other species. Often it helps to study a full range of phenomena including extreme cases to gain a better overview and understanding of peculiarities. In design thinking, the “power of ten” method asks for a consideration of phenomena at varying degrees of magnitude. This is also the approach Julia takes in the lecture as she explores creative capacities from miniature creative performances to most celebrated achievements, in a cross-species comparison. Fruit flies, for instance, do not react in deterministic ways to stimuli such as light. Most commonly they fly towards it, but in some instances they fly away—a diverging behaviour. This can be considered a miniature creative capacity: the ability to diverge. It is a biological capacity of the individual. Then, on the level of groups or populations, the phenomenon of “culture development” can be observed. Here, individuals need the capacity to build on creative ideas of others. For instance, songbirds learn melodies from neighbouring birds and even teach these to offspring. From a human perspective, however, culture development per se is not yet the greatest creative achievement. There needs to be a trajectory in culture development towards ever better, ever more sophisticated solutions. In the literature, this is called “cumulative culture” or a “ratchet effect.” Julia distinguishes three ways in which ratcheting can occur. First, there can be an increasing differentiation of solutions in a domain. For instance, in the prehistory of humanity, in the domain of stone tools, an increasingly differentiated palette of solutions was developed, from tools for hunting and eating to equipment including needles and figurines. Second, the sheer number of solutions in a domain can increase. According to research, this often happens at an exponential growth rate in human culture. E.g., from 1200 to 1900 in many different disciplines—such as philosophy, geology, medicine etc.—the number of publications increased exponentially. Third, solutions can show an increasing performance on a dimension of interest. An example is the top speed achieved with automobiles, which increased markedly from the late 1800s to the present day. Ratcheting yields most celebrated creative achievements in human culture, such as “science as a whole.” Notably, even the brightest human cannot invent science as a whole alone, in a single lifetime. For instance, Einstein is praised for outstanding creative achievements in science. Yet, he did not start from scratch. Rather, he built substantially on inventions made by others. Einstein used, but did not invent, numbers—and generally symbols—, mathematics, writing and books. He also thrived upon general supplies he did not invent himself, such as cooked food, medicine, furniture, pen and paper etc. Creative humans develop solutions that are far more sophisticated than what they could possibly invent all by themselves. Marked creative achievements emerge on a cultural level when creative solutions of previous generations and contemporaries become interconnected in a smart, productive way. Cross-species comparisons indicate that humans excel at the development of cumulative culture. This leads to key questions: In terms of biology and culture organization, what enables a smart interconnection of creative activities, even across countless generations, so as to yield cumulative culture and ratcheting? How can we model (reconstruct and predict) phenomena of culture ratcheting? Julia highlights how the answer may begin with the standard definition of creativity: A creative achievement obtains when a solution is produced that is novel and effective. Julia draws attention to the effectiveness dimension. She says that norm systems specify effectiveness criteria. For instance, in the realm of art people long invoked the norm system that paintings must depict scenes in detailed and naturalistic ways in order to be effective (“good”). Painters practiced and experimented in order to get better and better at these ends. When early modernists entered the stage, they changed the norm system. They rejected the effectiveness standards of detailed and naturalistic depictions. Instead, they introduced novel criteria: Paintings should depict scenes abstractly, based on simple shapes and patterns. Now a novel direction was provided in which artists could practice their skills, and could be creative, in order to produce ever more effective (“good”) paintings. As in this example, Julia says, it is possible to model cases of incremental versus radical innovation based on norm systems. She shows further sample reconstructions from the fields of automobile design, physics, sports, psychology, philosophy, mathematics and music. In all reconstructions, radical innovation builds on changes in the norm system. Julia lays out that in each work domain a paradigm is defined by effectiveness standards that originators follow in their creative pursuits. Effectiveness standards specify tasks: What to do, and how to do it. Changing effectiveness standards changes the paradigm: new rules, new game. In-paradigm-creativity means to advance creative (novel and effective) solutions based on established effectiveness standards. This approach engenders cumulative growth and progress in a field, i.e. incremental innovation. Out-of-paradigm-creativity means to advance (i) new effectiveness standards and (ii) creative solutions based on the new standards. This approach yields radically new solutions, i.e. radical innovation. While design thinkers typically celebrate radical innovation, Julia also cautions the audience. Paradigms are creativity engines. They drive cumulative culture growth and ratcheting. If radical innovation, i.e. paradigm shifts occurred too often, no paradigm would ever become sophisticated. Thus, for the development of culture at large it is important that forces of innovation favouring radical change and forces of innovation resistance that protect existing paradigms balance each other. In the final part of her talk, Julia looks at the biological basis that may allow humans to engage in creativity, coordinated over countless generations by means of cultural paradigms. What biological endowment allows humans to build up, protect and change norm systems, thus enabling cumulative culture and ratcheting next to the invention of novel paradigms? Julia reviews studies into the evolution of human creative capacities. These suggest characteristically human forms of creative performance were rendered possible by the development of theory of mind, sophisticated language and the use of material culture, a.k.a. the increased production of artefacts. In addition, Julia says, people’s abilities of adhering to norm systems, and the ability to change rules, are essential. In terms of neuropsychological theory, they are addressed as “executive functions.” Julia introduces some tests that are used to assess executive functioning. A volunteer tries the Wisconsin Card Sorting task live in class. Beyond that, norm systems can also be ingrained in cognitive maps concerning work domains. In biological terms, these are likely encoded in the hippocampal formation and related regions. Lastly, Julia addresses the challenge that norm systems define social groups. People who adhere to “your norm systems” belong to “your in-group.” People who frustrate your normative expectations are “not like you,” they are “strangers.” E.g., when a person comes and eats an animal that “is not for eating”—this person comes from a different culture, he/she is “not like you.” Julia reviews neuroscientific studies showing that our brains process information differently depending on whether a behaviour is shown by an in-group or an out-group member. Consistently, the same behaviour is processed more negatively when shown by an out-group member. Julia warns that radical innovation changes norm systems. Therefore, radical innovators run the risk of being perceived as an out-group member by everyone in society, leading to a biased and more negative assessment of whatever the radical innovator does or suggests. This can be one reason why it is helpful to develop radical innovation in teams. When the objective is to foster radical innovation, ideally decision-takers develop personal bonds to the innovation team, so that emerging proposals of radical change will not get processed as coming from “out-group members.”

In a second part of the lecture, Joaquin Santuber (HPI) addresses normativity in the context of collaboration from an experimental point of view. A key question is how normative-social processes, such as collaboration, can be related to neuro-physiological dynamics in the individual. Using systems theory and an embodied-enactive cognition perspective, he elaborates on how the synchrony of physiological signals can be seen as the coupling mechanisms of people engaged in collaboration. To show the merit of this approach, he presents an experiment conducted at the HPI on physiological synchrony between participants performing three tasks. In all cases, participants get to work in pairs of two persons on varying, wooden dinosaur puzzles. In the first task, participants get to work on a dinosaur puzzle without any specific instruction. In the second task, the two participants obtain different instructions. In the third task, both participants obtain the same instruction. After every task, a self-report questionnaire on perceived team cohesion is completed by the participants (Perceived Team Cohesion Questionnaire, PTC). To study changes in the level of interpersonal synchrony across tasks, for each dyad the following data is gathered and analysed: electrodermal activity (skin conductance), heart rate variability, and automated video analyses of facial expressions as well as head pose. Among all data collected, facial expressions provide clearest results and seem best suited to distinguish between the three social conditions. High level of synchrony of positive facial expressions are found during task 1 and 3. By contrast, interpersonal synchrony of negative facial expressions is higher in task 2 than in task 1 and 3. These results are consistent with questionnaire data; perceived team cohesion correlates with synchrony in positive facial expressions (r = 0.44). Joaquin also discusses methodological aspects and challenges of data collection and analysis when feasible and accessible methods are used. The study of synchrony between social and physiological dynamics needs to account for emergent properties of collaboration, as well as the self-organizing neuro-physiological dynamics of each participant engaged in skilled action. In this sense, it is important to complement quantitative data of physiological signals with qualitative data regarding the social process.

1.11 The Use of Artificial Intelligence to Facilitate Human Creativity: Entrepreneurial and Artistic Approaches

Shama Rahman (Complexity Group, Institute of Mathematics and Physics, Department of Condensed State Matter, Imperial College London; Centre for Cognition, Computation & Culture, Goldsmiths University of London; Royal College of Music and CEO of NeuroCreate) reviews the field of Artificial Intelligence and creativity. The topic is already widely discussed, most frequently centering on questions as to whether Artificial Intelligence might itself become creative. Shama, however, invokes a different point of view: (how) could Artificial Intelligence help enhance human creativity? She discusses neuroscientific underpinnings of creative performance, such as creative work in a Flow state. In her PhD and subsequently in her startup NeuroCreate, she has been working on the measurement of Flow via EEG. It can now be detected by proprietary deep learning models.

From philosophical and practical perspectives, Shama explores the potential of a complementary symbiosis between Artificial Intelligence (informed by neuroscientific knowledge) and human creators. This neurodesign avenue of work would lead to ‘augmented creative intelligence.’ Shama discusses how we might all benefit from such an approach.

In a live demonstration, Shama introduces the software FlowCreate™ Innovator developed by NeuroCreate. She discusses how the design is informed by theories and research findings concerning creative processes. This includes Shama’s own PhD work, where she studied creative performances of musicians, recorded along with EEGs of the performing artists.

Together with the audience, Shama uses FlowCreate™ Innovator during a brainstorming task. Participants witness how Artificial Intelligence is invoked to stimulate ideation. The approach allows humans to think about solution directions they would not normally consider, without the tool. Artificial Intelligence also helps humans be more systematic in the exploration of potential solution spaces during brainstorming. This includes a discovery of personal unconscious biases and blind spots: areas one should consider, but they don’t come to mind—unless Artificial Intelligence points them out.

Further insights into Shama’s works are provided in Sect. 2, in a discussion of two neurodesign projects conducted under Shama’s supervision.

1.12 Examining Social Influences on Brain and Behaviour Across Development

In her talk, Julia Rodríguez Buritica (Biological Psychology & Cognitive Neuroscience, Free University of Berlin) examines social influences on brain and behaviour across development. A key question in this research domain is how individuals make choices. This can be choices of consumers to buy or not buy a software package, choices of digital engineers to use this or that programming language, choices of students in class where they need to select an answer in a multiple choice test, or choices of any one of us in everyday life. Julia discusses the important factor of social influences for human decisions. The paradigm in which she investigates the phenomenon is called “reinforcement learning.” Here, each choice alternative has a specific outcome, which is good or bad. The question is how experiences of good vs. bad consequences after deciding on an option impact people’s learning, i.e., their future choices. Of course, humans learn from their own experiences. When a choice has negative consequences, the person is less likely to decide on this option again. E.g., when a selected answer in a multiple choice test turns out to be wrong, the student will likely pick a different answer when asked the question anew. Similarly, students learn from the outcomes of others. When a peer selects an answer in class, which is then labelled as incorrect, other students are less likely to opt for that answer henceforth. Notably, humans use roughly the same brain regions when they learn from their own outcomes compared to when they learn from the outcomes of others. Relevant brain regions include the medial prefrontal cortex, which is part of the frontal lobe that matures late in life. It develops until the age of 20 or even beyond. Thus, children vs. adolescents vs. adults differ in how they learn from outcome observations. For children it is specifically difficult to learn from negative consequences. Indeed, this learning involves complex representations in the medial prefrontal cortex, e.g. to inhibit a previously preferred choice option, while the pertinent brain structure is not fully matured in children yet. As a pedagogical consequence, it has been recommended to use less “red markings of incorrect answers” in primary school; emphasising what the kids did well might be a more effective teaching strategy. All in all, the field of reinforcement learning has developed a high level of sophistication, which allows an accurate modelling of how people learn and decide based on (i) their own experiences, (ii) observations of others and their choice-outcomes, (iii) explicit advice given by others, (iv) the social role of advisors, such as the person being a peer vs. not a peer and (v) inclinations of the individual to explore the field irrespective of social information. Julia highlights the great potential of further collaborations between neuroscientific lab research and practical projects especially in educational contexts. Here, digital engineering solutions could provide a crucial bridge between the fields, and they could advance innovative solutions. Apps could easily track choices made by individual students, could model the influence of others and provide age- or otherwise adaptive learning feedback, e.g., to account for the difficulty that children have in learning from negative outcomes.

1.13 Seminar Topics

In the neurodesign seminar, Joaquin Santuber discusses three specific opportunities for neurodesign: (i) feasibility (off-the-shelf technology), (ii) accessibility (open-source and open-science initiatives) and (iii) HPI expertise (using Digital Engineering knowledge for creative purposes). Thanks to advances in hardware technology, computer vision and data processing, new digitally-enabled methods are available to a broader group of researchers and practitioners. Thus, working with physiological data is not a reserve of experts who harness high-end devices in research labs any more. Nowadays, widely available tools can be combined with open-source and open-access data processing libraries. Data analysis becomes more widely accessible both to researchers and practitioners in a resource-effective manner. For instance, the coding of facial expressions used to be a labour-intensive task that required special training and expertise. By now, automated approaches for the coding of facial expressions are freely available. At the same time, the outstanding expertise of the HPI in Digital Engineering renders this place a fertile environment for explorations into the full potential of such digitally-enabled feasible and accessible methods. Since many of those methods are explained in more detail later in this chapter (Neurodesign Projects), here some brief examples shall suffice. During the seminar, participants come to try out automated analyses of gestures and body postures using computer vision. This method can provide a rich account of non-verbal communication and embodied aspects of creativity (OpenPose). Related insights can be gained by the study of facial expressions and head pose using automated video analyses (OpenFace). Such approaches help shed light on the affective and emotional states of users. Eye-gaze analysis (eye tracking) facilitates studies on attention and patterns of visual perception. Data on heart rate variability and electrodermal activity can be gathered with off-the-shelf technology like smartwatches and fitness trackers. It can augment studies regarding topics such as cognitive load, stress and exertion.

Joaquin’s discussion of three specific opportunities for neurodesign at the beginning of the semester—feasibility, accessibility and digital engineering expertise—turns out to be an accurate preface for subsequent works. Almost all neurodesign projects that emerge in the course of the semester tap the full spectrum of these opportunities. Projects typically harness consumer-grade products (instead of cost-intensive neuro-medical lab equipment). Many of the projects yield open-source publications, software and hardware. All projects build on digital engineering expertise to deliver novel and effective solutions. At the same time, the specific area of work differs from one neurodesign project to the other; contributions are developed for the fields of neuroscience, design thinking research and/or creative engineering.

Julia von Thienen discusses bodily perspectives on creativity and collaboration. These have played a key role in design thinking theory and practice for decades. Julia elaborates on the importance of morphology for creative behaviour options. She refers to cartoons such as “Maya the bee,” where bugs necessarily are depicted with a changed, more human morphology. Otherwise, stories could not be told about the protagonists encountering problems and finding creative solutions together. Julia invites the audience to reflect on morphological differences between the cartoon characters and real insects, such as bees and dung beetles. E.g., the cartoon characters have hands to handle artefacts. They communicate emotions and intentions by means of facial expressions and body postures. They have frontally positioned eyes with round pupils, allowing them to focus attention, and to focus attention jointly. Overall, Julia emphasises, creativity is a full-body phenomenon; it is not only about “the brain.” Another key topic is the well-researched impact of posture and motion on creative and collaborative achievements. Across multiple studies, fluid, bilateral and relaxed (non-strenuous) motion has been found to facilitate creativity. Here, experimental findings regarding the impact of motion are consistent with biographical research, in which people report on the situational circumstances when they found creative breakthrough insights. While it has long been noted how creative insights regularly emerge in situations where people have their attention “off-task,” e.g., they take a break, and how relaxation (non-strenuous activities) seems to be important, it increasingly becomes clear that highly conductive situations for creative insights also often involve motion. Typical examples are people going for a walk or taking a shower. These observations are congruent with neuroscientific findings, which suggest an involvement of the cerebellum in high levels of creative performance. The cerebellum is traditionally known for facilitating fluid motion bilaterally. Julia also discusses the role of the environment in stimulating or inhibiting favourable postures and motions, such as the impact of different table or chair arrangements. Notably, design thinking environments designed to facilitate creativity and collaboration encourage motion during work hours.

In other sessions, Julia provides introductions to the basics of measurement theory and study design. She reviews different scale levels of data (nominal, ordinal, interval and rational), quality criteria for measurements (objectivity, reliability and validity) and different ways to assess them, such as re-test versus parallel-test reliability. Another methodological topic is study design, where Julia discusses different theories of causation (Aristotelian, Regularity, Nomological, Probabilistic, Transference, Counterfactual and Interventionist), with the Interventionist Theory of causation being most frequently used as a basis for social science study design. Different sampling methods are discussed, such as convenience, stratified versus cluster sampling approaches. In class, based on study design templates, participants learn to plan, conduct, analyse, criticize and improve randomized experiments.

In terms of work presentations, course participants of both the seminar and the lecture learn to follow common scientific formats when presenting their own work in talks and on scientific posters (introduction—aim—theoretical background—pertinent literature; methods; results; conclusion—discussion—limitations; references; acknowledgements). In the introductory part of project presentations, course participants also learn to invoke the design thinking approach of pointing towards important, unmet needs, which shall be addressed by means of novel (neurodesign) solutions. Again in line with design thinking approaches, course participants learn to develop convincing, captivating visions of where their projects can and should be leading.

Irene Sophia Plank (Berlin School of Mind and Brain, Humboldt-Universität zu Berlin) teaches research methodology for neuroscience in several sessions. Her lectures cover null hypothesis significance testing, including p-value calculation and interpretation, sampling distributions, common misconceptions about p-values, type-1 and type-2 errors in null hypothesis testing, deficiencies of null hypothesis testing (such as widespread p-hacking by means of multiple tests) and parameter estimations. Irene further covers several statistical tests and their prerequisites, such as t-tests, ANOVAS, linear models and linear mixed-effect models. In class, tests are calculated in R. In homework submissions, student teams can choose between R and Python. In further sessions, Irene covers topics of digital signal processing in electroencephalography (EEG) and magnetic resonance imaging (MRI). She discusses the spatial and temporal resolution of both methods and the neural origins of signals on macroscopic, mesoscopic and microscopic levels. Regarding EEG assessments, Irene discusses resting membrane potentials, action potentials, synapse activities, extracellular field potentials, event-related potentials, visually evoked potentials, mismatch negativity, P300, language-related ERPs, the readiness potential, EEG frequency analysis by means of Fourier analysis, time-frequency analysis and the reconstruction of EEG signal sources in the case of few or many dipoles. Regarding MRI assessments, Irene covers physical and biological foundations of the approach, signal generation, the magnetisation of spin systems, excitations via radiofrequency pulses, the relaxation of the MR signal, T1, T2 and T2^* contrasts, neuronal energy consumption, the cerebral vascular system, haemodynamic responses and the haemodynamic response function, neuronal correlates of the BOLD signal, spatial scales and spatial resolution, the partial volume effect, the temporal resolution based on sampling rates and biological factors, the linear transform model, scaling and superposition, as well as studies indicating shortcomings of present-day fMRI methods. In terms of practical data analysis, regarding the EEG Irene teaches how to conduct data preprocessing, how to calculate event-related potentials and cluster-based permutation analysis. Regarding the fMRI, she teaches data preprocessing, how to conduct a 1st level analysis on single subjects, and how to calculate a 2nd level analysis that integrates data of multiple subjects. Calculations are conducted in class with Matlab, SPM and Fieldtrip.

This rather comprehensive input of Irene on behalf of neuroscientific data processing is also reflected in a relatively large number of neurodesign projects that are specifically concerned with neuroscientific data (pre-)processing.

2 Neurodesign Projects

In this section, we review a selection of neurodesign projects that have emerged in the last semester. Most of them will be continued in follow-up projects in the next semester. The abstracts are intended to provide rather self-standing introductions to projects, so that readers can also read selectively about endeavours of particular interest.

2.1 Neurodesign Tests

This project is specifically concerned with the measurement of creativity and collaboration. Up to today, creativity assessments are often conducted with tests in a pen-and-paper format. This approach has a number of severe limitations. First, test-taking with analogue test material is tedious for researchers. Test responses of participants need to be manually re-cast into a digital format to allow for statistical processing; this procedure is time-consuming and error-prone. Second, the analogue test approach is barely scalable; only a limited number of participants can be included in studies with pen-and-paper tests. Third, the rating of standard test responses often requires subjective assessments of “expert raters.” Here, the analysis of test responses is resource-intensive and the reliability of test results is not ideal.

On the internet, a few creativity test compilations and platforms are already available. They are typically hosted by engaged creativity scholars who provide a web-service as a “pet project” alongside demanding regular jobs in fields such as psychology and/or neuroscience; the design of internet services is not their primary field of expertise.

In a one-semester project, Nina Ihde, Jannis Rosenbaum, Martin Michaelis, Katharina Blaß, Florian Papsdorf, Philip Weidenfeller, Arne Zerndt, Ahmad AlAbbud and Florian Fregien (2020) have developed a web-based platform for researchers to share and access standardised creativity tests (Fig. 8). The platform is designed to offer state of the art usability experiences. Tests can easily be created, used for study purposes and can be shared with colleagues. The web-platform also offers an immediate connection to Amazon Mechanical Turk (MTurk), so that researchers can administer their tests to huge numbers of study participants. All test results are straightforwardly available in a digital format. Since data is stored safely on HPI servers, confidentiality of research data can be ensured for scholars who use the service.

Fig. 8

The platform “Neurodesign Tests” was developed by Nina Ihde, Ahmad AlAbbud, Florian Papsdorf, Jannis Rosenbaum, Philip Weidenfeller, Arne Zerndt, Katharina Blaß, Martin Michaelis and Florian Fregien. It renders standardized creativity and collaboration tests accessible for researchers around the globe for digitally-facilitated testing (photo by Kay Herschelmann)

This project was supervised by Matthias Bauer and Julia von Thienen at the HPI. It benefitted from major input by Mathias Benedek (Universität Graz) and Laura Kaltwasser (Humboldt-Universität zu Berlin). The project team is also grateful to Adam Royalty and Grace Hawthorne (both Stanford University) for sharing and discussing pertinent test material.

The platform Neurodesign Tests is accessible at https://hpi.de/neurodesign/tests.

2.2 Measuring Creativity with an Online Game

Up to the present day, creativity tests are often administered in pen and paper versions—an approach that induces severe limitations. Most notably, only relatively few study participants can be tested, as the processing of handwritten data is labour-intensive. Furthermore, a number of standard creativity tests involve self-reports, which are not necessarily reliable indicators of either creative potential or actual creative behaviour. Other standard creativity tasks ask participants to engage in rather “unnatural” behaviours, such as naming uncommon uses for everyday objects.

As an alternative to more traditional creativity tests, Corinna Jaschek and Eva Krebs explore opportunities for creativity-measurements via a web-based computer game (Fig. 9). In the game, participants get to work on the goal of protecting a living organism against various attackers. This can be achieved in the game by placing and upgrading objects on the game grid. The approach is realized as a tower defence game, a common subgenre of strategy video games. It is implemented via the open-source game engine Godot.

Fig. 9

Eva Krebs and Corinna Jaschek develop a web-based computer game for creativity assessments (photo by Kay Herschelmann)

All game interactions taken by the user—such as placing objects, upgrading objects or halting enemies—are automatically tracked and stored. These events can then be analysed automatically, without the need for a human expert who manually judges the creativity of the player’s action. All test results are immediately available in a digital format. Conclusions can be drawn regarding a single participant, but also across all players. For instance, how novel/original is an action taken in the game by a single player, in light of all actions that have ever been taken by players of the online game? Moreover, the effectiveness of actions taken in the game can be assessed objectively, as effective actions stop attackers and help the organism at stake maintain a good health status.

Due to the web-based application, the testing routine can achieve a high level of standardisation in studies across the globe. Moreover, studies can be hosted via Amazon Mechanical Turk (MTurk) or similar platforms, so that it becomes feasible to work with huge numbers of study participants.

People’s complex gaming behaviours permit the calculation of an array of measures, which can serve as indicators of different facets of the creativity construct. Which of the available game-measures best serve to assess the participant’s “fluency,” “originality,” “flexibility,” “problem-sensitivity” etc. is currently elucidated in validation studies, which combine the game approach with standard creativity tests on behalf of various creativity facets.

The game is developed based on a design idea by Corinna Jaschek, Tom Beckmann, Kim Borchart and Christian Flach. The project is supervised by Julia von Thienen at the HPI and Oren Kolodny at the Hebrew University of Jerusalem.

2.3 Neurodesign Cards

Why, how and when does design thinking work or fail? Many aspects of design thinking can be understood in depth based on a thorough understanding of the human body: how humans become creative and collaborative—or fail to do so. The field of neurodesign brings together expert knowledge from different strands of methodologically rigorous empirical research. This knowledge is gathered, for instance, in neurodesign lectures at the HPI, with contributions from leading experts of internationally recognized labs and research centres.

In the form of a simple to use card set, Julia von Thienen, Caroline Szymanski and Theresa Weinstein make key research insights available to design thinking practitioners. The cards overview design-thinking-relevant empirical research findings and discuss implications for practice. Thus, design thinking coaches and teams can learn to deploy interventions even more mindfully and purposefully.

In this project, the card set editors develop the overall framework and contribute a basic stock of cards. In addition, further cards can be authored by interested neurodesign experts who review the implications of their own research findings for design thinking · creativity · collaboration practice.

The card set is tested and iterated in collaboration with Annie Kerguenne and Miriam Steckl from the HPI Academy. First prototypes have been discussed and used at the Connect & Do Day organized by the HPI Academy on February 14, 2020.

2.4 Sonification of Brain Data

Brain data is often visualized either through colourful fMRI images or lines of EEG plots. However, much of what goes on in the brain cannot be intuitively understood or analysed by looking at static images only. There is rhythm in the brain. Slow rhythms dominate brain activities when people relax and go to sleep. Creative fluency is also often associated with a rather relaxed rhythm of brain activation. When people concentrate intensely, faster rhythms become prominent. During extreme stress, excessively fast rhythms can take over. There is also the importance of “geography.” Does an activation pattern begin in the back of the brain, driven by visual information processing? Or does a burst of activation suddenly emerge in the front of the brain, reflecting conscious control and planning? Is mostly the left hemisphere activated, indicating verbal or symbolic processing? What is going on in the upper-middle zone of the brain, revealing body-related information processing? Such differences of rhythm and directions of activation-spreading could be heard much easier than they can be seen in a picture.

As the pioneering works of neurodesign guest lecturer Chris Chafe show, the sonification of brain data is feasible nowadays and it renders possible very intuitive analyses of what happens in the brain. With 3d sonification models for headphones or even multiple speakers prepared in a room, brain data can be rendered meaningful for listeners.

Based on novel sonification algorithms, questions such as the following can be addressed:

• How does the brain activity of highly creative persons sound, compared to the activity of less creative persons who work on the same task?

• How does the rhythm of brain activity change when people experience creative flow, frustration, high-vs.-low levels of attentiveness etc.?

• Is it possible to discern differences between patient groups and healthy participants acoustically, by means of listening to peoples’ brain activities?

• (How) can brain data sonification aid education? E.g., does it help students understand brain functioning when they can walk through a huge 3d brain, where different kinds of brain activity are represented acoustically and analysed together with the lecturer?

• (How) can we help creators better regulate their own work processes by providing acoustic feedback on their own brain activity?

• (How) can we generate captivating art experiences by means of sonified brain data?

Several neurodesign projects explore opportunities in the field of brain data sonification for diagnostic, applied and artistic ends. Currently, they concentrate on the sonification of EEG data (the area where Chris Chafe has already pioneered solutions). In the future, extensions towards fMRI data, or also towards a sonification of full-body-data, are likely next steps.

2.5 Brainwave Sonification Toolbox

One major challenge for brain data sonification emerges right after the phase of data collection. Raw EEG data is full of artifacts and noise, such as muscle movements and eye blinks that induce strong signals in the recordings. If the measured EEG curve was represented acoustically straightaway, the rhythm of eye blinks would dominate the sound, making it hard to discern dynamics in brain activity proper. Moreover, EEG raw data often comes with technological artifacts, such as “electrode drifts” where the measured voltage of an electrode steadily increases or decreases over time. In a labour-intensive work phase of data preprocessing, neuroscientists remove artifacts from their data sets before analysing and creating EEG plots. Similarly, sonification algorithms can provide best and clearest acoustic representations of brain activity when they are applied to datasets of brain activity that contain as few artifacts as possible.

Another challenge of applying sonification algorithms to EEG data emerges from the multiplicity of recording formats and software packages that prevail in research. Sonification algorithms could be used routinely if written for a standard data format used across labs. However, a variety of different file formats are used in research. Moreover, many EEG data files can only be opened with lab software that is not freely available.

In this project, Leon Papke, Carla Terboven, Philipp Trenz and Simon Witzke (2019, 2020a, b) envision EEG preprocessing for sonification purposes as a (free-of-charge) service. Users can upload EEG raw data and receive preprocessing support by a software package designed for good user experiences. With this service, (i) brain data preprocessing can happen rapidly, (ii) neuroscience expert knowledge is not required, (iii) expensive lab software is not required, (iv) EEG raw data can be uploaded in many different file formats and (v) users can obtain download files in different data formats; the .csv format is suggested as a standard for sonification algorithms.

A first prototype of the Brainwave Sonification Toolbox is available for testing and initial usage. It is realized as a Docker container. Thus, the application can run locally on a computer, but it can also be offered as an intra-net or cloud service. In terms of architecture, the frontend of the service is realized via Angular and the backend via Django. Currently, brain data can be uploaded in the following formats: European Data Format (EDF), BioSemi (BDF) and the file format of BrainVision (.eeg). Users can then define the desired preprocessing pipeline. Common preprocessing steps are represented visually as drag-and-drop boxes with simple explanations. For instance, users can apply high-pass, low-pass or band-pass filters with self-selected values. The application of filters on EEG data is realized internally via MNE (Gramfort et al. 2013a, b), which is an open-source python-based platform for the preprocessing of brain data. Its relevance for professional neuroscientific practice is already reflected by a large number of academic citations. In contrast to the Brainwave Sonification Toolbox, MNE is designed to be operated by neuroscience expert users who are ready to dedicate time and effort to data preprocessing.

This sonification project supports open science initiatives. The code of the Brainwave Sonification Toolbox is published on https://github.com/1p4pk/brainwave-sonfication-toolbox. It is available for free and can easily be extended.

2.6 Real-Time EEG Sonification with the BITalino Platform

In the area of EEG sonification, a number of pioneering works have been conducted in recent decades (Väljamäe et al. 2013). Often, projects reside somewhere in between science, engineering and art, where authors vary in the purpose they emphasise most.

An example of a project where primarily artistic aims are pursued is the Gnosisong brain installation by Chris Chafe and Greg Niemeyer at the Centro de Cultura Digital in Mexico City from August 28 to September 24 in 2015 (Berkley Center for New Media 2020; Gnosisong 2020). Here, visitors experience an abstract three-dimensional depiction of a brain, where medical-quality EEG data is conveyed by means of sounds and visuals. Thus, viewers shall be stimulated to experience and reflect on the “mystery of thought.” They witness rhythmic patterns emerge or dissolve, the speed of signals increases or slows down, different nodes of the installation (i.e. EEG channels) begin to signal in synchrony and then acquire individual dynamics again.

An example of a project with marked artistic aspirations next to very practical use cases is that of Miranda (2006). He presents a brain-computer interface system that allows users to compose and perform music on a mechanical acoustic piano regulated by their brain activity. Technically, the system tracks activity in different EEG frequency bands of the user, which in turn activates generative rules for the production of original music pieces. While in this project music is understood as an artistic expression of the player, the practical use case Miranda has in mind is a therapeutic treatment for persons with physical disabilities, who cannot play instruments in more traditional terms.

In yet other cases, purposes predominantly reside in the area of (applied) science. Again, a project by neurodesign guest expert Chris Chafe provides a good example. In collaboration with Josef Parvizi and colleagues (Parvizi et al. 2018), a sonification tool was developed to facilitate the diagnosis of “silent seizures.” Here, patients enter medical conditions without easy-to-observe bodily convulsions. In a test study, the authors find that untrained personnel using the sonification tool can diagnose silent seizures even more reliably than trained personnel, who reviews the same EEG data by means of visually inspecting EEG graphs. While this project is primarily directed towards goals of applied science, artistic concerns also play a role. In order to create interpretable sounds that highlight brain dynamics of interest, Chris “composed” an acoustic signal in the range of a male voice. Seizure patterns in the EEG data are sonified by means of a screaming or moaning sound that humans can easily produce, imitate and “understand.”

In his sonification project, Noel Danz (2019, 2020) brings together elements of previous sonification work from artistic and applied-scientific contexts. He explores opportunities in the area of three-dimensional EEG data sonification—akin to the Gnosisong brain installation—, but pursued for applied rather than artistic ends. In addition, Noel seeks to advance an “ultra-feasible” neurodesign method: (i) EEG data sonification shall be possible in real time. (ii) The method shall be feasible for many persons, due to low budget solutions across the whole production chain, from EEG data acquisition, over sonification software, up to three-dimensional sound experiences. (iii) The sonification algorithms shall help elucidate EEG data of any kind—beyond specific, predefined diagnostic ends.

Earlier works in this direction had been conducted by Baier et al. (2007), who provide a model for real-time three-dimensional EEG data sonification in the service of diagnostic goals. They develop a multivariate event-based sonification approach, which displays salient rhythms, modulates pitch and provides spatial information. The test case of the authors—akin to Parvizi et al. (2018)—is the diagnosis of seizures. In contrast to the project of Noel, the work by Baier et al. (2007) is not necessarily directed towards creating an ultra-feasible method. The EEG data they sonify was acquired in professional medical labs. Thus, the question of how each and every interested user might obtain EEG data for sonification trials is not of concern in their project. Moreover, Baier et al. invoke one very specific test case, i.e. seizure detection. The great bandwidth of questions that might be addressed by means of 3D brain data sonification—including topics of potentially unique brain dynamics in highly creative and/or collaborative persons—are at most a brief side-topic in their discussion. Finally, there are various technical differences between the solution of Baier et al. (2007) versus that of Noel. Most notably, Baier et al. communicate information about the physical location of an EEG signal by means of different pitch levels in the sound. Noel develops a solution where listeners can hear the location of a sound, i.e. people can hear where the EEG signal comes from.

To allow for ultra-feasible EEG data acquisition, Noel opts for the BITalino platform. The BITalino is a low cost, open source, single-board computer. It is designed for purposes of education, prototype development and biomedical research—a solution that won the European Commission Innovation Radar Prize 2017 in the category “Industrial & Enabling Tech.” It has a well-document API and offers bindings for many programming environments. In his project, Noel works with the BITalino Plugged Dual kit with additional EEG sensors, yielding overall costs of ca. 300 €. This is cheaper in orders of magnitude compared to regular EEG equipment used in medical or scientific labs. Moreover, compared to other consumer-grade EEG solutions like the Muse 2, which records with a fixed number of four EEG channels in the front of the head, the BITalino platform is more flexible and allows users to self-select locations on the skull where brain activities shall be captured.

As a technical approach to 3D brain data sonification, Noel chooses the paradigm of binaural sound localisation. This approach builds on the human ability to localise sounds in space based on how sound signals reach the ears (Stern et al. 2006). E.g., a sound signal that comes from our left side reaches the left ear first; the sound signal also has a slightly higher decibel-level when reaching the left ear compared to the right.

In terms of design, Noel creates a spatial representation of the brain (Fig.10). Users can upload EEG data or make their own recordings with the BITalino platform. To explore EEG data acoustically, the user can position a microphone by means of drag-and-drop actions somewhere on the brain map. Now the microphone indicates the position of the listener in the brain. In a next step, the user can select one or more EEG channels for sonification. If a selected EEG channel resides on the left side of the microphone, the sound is emulated, so that the listener experiences it as coming from the left direction. EEG channels being closer versus more distant from the microphone also get represented in different ways. Again, the sound is emulated, so that listeners experience the sound as coming from more distant versus closer sources. Technically, this is realised by means of an open source WebAudioApi “Panner3D,” which emulates the ways in which sounds reach human ears based on the position of the audio source in relation to the listener. Presently, the sonification algorithm is implemented to signal acoustically relevant changes in the EEG plot.

Fig. 10

Based on binaural sound localisation, listeners can hear EEG data captured at different positions of the skull [image reprinted with permission from Danz (2020)]

Noel’s sonification solution is available here: https://hv10.github.io/neurodesign_sonification.

2.7 From EEG Data to 3D Sound Spatialization

The aim of a project by Lukas Hartmann, Tim Strauch, Philipp Steigerwald and Luca Hilbrich (from the Technical University of Berlin) is to use EEG data to create a 3D sound installation, which renders distinct rhythms and topographies of brain activity in such a way that it can be easily interpreted, even without extensive prior training in neuroscience.

After preliminary testing with PyLive and Ableton as well as SuperCollider, the team is currently developing a spatialization system that enables users to distribute audio live, across a speaker array (Fig. 11) using EEG data as a control input. A first realization of the system is planned in lecture hall 3 at the HPI, and/or in the Design Thinking Research Lab of the HPI main building on the 3rd floor.

Fig. 11

A multichannel sonification array set-up is planned for brain data sonification

To control audio parameters, the team tracks energy in different EEG frequency bands over time (alpha, low beta, high beta etc.). The distribution of audio channels mirrors EEG channel locations on the skull: EEG activity captured at frontal positions triggers speakers in the front of the room, while, EEG activity from medial positions on the skull is represented by speakers in the middle of the room and EEG signals at occipital positions (the back of the head) by speakers in the back of the room.

This platform is intended to host a multitude of projects in the field of brain data sonification. With the audio input left open-ended, a wide variety of purposes can be addressed.

Notably, sounds need to be designed mindfully for each purpose. Simple and concise sounds are likely to be most serviceable for diagnostic and educational applications. Ideally, even untrained persons can discern constituent frequencies and their relative ‘geography’ in sonified EEG signals. Listeners shall obtain an easy and effective way to interpret brain activities with just a little help of experts, who provide short introductions to the meaning of sounds that can be heard from different locations of the room. In other applications, it may be desirable to use more complex inputs. For instance, based on real-time measurements of EEG activity, live audio—such as instruments—could be used and distributed by the platform to stage live performances.

Ultimately, the project aim is to engage listeners with brain data, to help audiences understand brain dynamics, and to foster critical design thinking around the human brain.

2.8 Brainwave Sonic Instrument for Sound, Multi-channel Sound Installation and Live Performance

Nico Daleman (from the University of Arts at Berlin) explores solutions for brainwave sonication in the form of an open source virtual instrument compatible with sound installations and multichannel systems. Here, the aim is to create immersive sonic experiences and musical performances. In a current solution co-developed with colleagues from the Technical University of Berlin, six speakers are distributed in an architectural space. They convey brain activities captured at frontal, medial and occipital regions of the brain (cf. project described above, Fig. 11).

For his project, Nico decided to work with .csv files that are processed through the open source software Supercollider. Naturally, multichannel EEG measurements—with data recorded from various locations on the skull—translate to a multichannel audio environment and multitrack audio recordings. Indeed, neuroscientific brain measurements by EEG or fMRI include crucial information about spatial locations. For the interpretation of signals it is key to know where each signal was measured, e.g. in the front or the back of the brain. Thus, recorded brain data could not be adequately sonified with traditional stereo sound solutions, where listeners could only distinguish signals coming from the left versus the right side of the brain. Surround sound is needed to convey precisely whether brain activity signals originate more from frontal or occipital regions, from the left or the right hemisphere. In this endeavour, multichannel audio spatialization techniques, particularly Object-based audio (Tsingos 2018) and Binaural rendering (Roginska 2018) can be very serviceable to create a unique sonic experience of the recorded brain data.

Unlike Chris Chafe’s seizure-detection project, which aims for an overall sonic rendering of EEG data (but more in line with his composition Gnosisong), Nico’s approach is designed to deliver a novel sonic immersive experience, in which the complexity of brainwaves can be perceived spatially. A first prototype (based on Bovermann et al. 2011) has been set up for eight different EEG channels, but could be expanded to 16 or more channels, depending on the input data provided. Technically, the channels figure as “objects” in the framework of Object based audio techniques, using the Ambisonic Toolkit (Anderson and Parmenter 2012). Based on energy in the different EEG frequency bands over time (alpha, low beta, high beta and theta), different audio frequencies are generated. Due to the low frequency characteristics of the original signals, these are also used as Low Frequency Oscillators (LFOs) to modulate between audio output frequencies, and to control or modify different parameters of the piece, including filter cut off frequencies, spatial position and amplitude. These parameters can be controlled in a live musical performance via a MIDI controller such as the Novation Launch Control XL, which allows the performer to improvise and react in real-time to the outcome of pre-recorded brain data. Using LFOs to modulate audio outputs also allows further experiments outside the digital domain. Alternatively, EEG signals captured with devices such as the 16-channel-EEG cap OpenBCI can be used in real-time to regulate, for instance, CV controlled devices such as modular synthesizers. This opens up new collaborative frontiers between neurodesign, sonification practices, electronic musicians and sound artists. Thanks to the flexibility of Object based audio techniques, the original eight-channel solution can also be experienced in a six-channel speaker setup, or alternatively as a binaural rendering. A first prototype of the instrument and the sound results can be found at https://github.com/nicodaleman/brainwave-sonic-instrument.

2.9 Measuring Creative Flow in Real-Time Using Consumer-Grade EEG and a Neural Network

When an individual works in a mental state of creative Flow, she feels fully immersed in her activity. She is highly concentrated, works productively, enjoys the activity and experiences self-fulfilment. The concept of Flow was introduced by Csíkszentmihályi (1975, 1990). According to his model, individuals experience Flow when they find their work task challenging in a positive sense, and when they have the necessary skills to tackle the objective well.

Flow states are desired for several reasons: They are prized for yielding best work outcomes. Subjectively they feel joyous and individuals experience a sense of fulfilment. ‘Innovation-training’ in particular, which naturally enables more creative Flow states, has even been found to increase brain health (Chapman et al. 2017).

With her start-up NeuroCreate, Shama Rahman wants to bring research findings to the world of everyday applications. The company’s platform FlowCreate™ Innovator uses Artificial Intelligence to facilitate human brainstorming processes. Presently, biofeedback regarding Flow states is being added to the system. NeuroCreate’s proprietary deep learning model applied to EEG datasets has been used to identify the physiological signature of high versus low levels of Flow, such that real-time classification is possible. Respective data had been gathered with state-of-the-art EEG lab equipment (64 active electrodes in the 10–20 Biosemi configuration), by Shama in the course of her PhD research. Now for the public to enjoy biofeedback regarding Flow levels, data from consumer-grade wearables or other sensors will have to suffice as information input.

An experiment by Holly McKee, Felix Grzelka, Laurenz Seidel and Pawel Glöckner (2019, 2020b, c) explores physiological signatures of Flow states during brainstorming, identifiable with consumer-grade wearables and webcam recordings. According to Csíkszentmihályi (1990), Flow states will only be achieved regarding brainstorming when people have high skill levels in brainstorming and experience the given brainstorming tasks as challenging. Thus, in terms of sampling methodology, only persons experienced in brainstorming should participate in the study. A talk at the HPI D-school with an audience of design thinkers was used to acquire test participants; only volunteers with at least half a year of design thinking experiences were included in the study. Data collection was supposed to occur in subsequent weeks at the HPI, but got delayed due to the COVID crisis.

The study is designed as an experiment with repeated measures. Participants get to work on brainstorming tasks in two-person-teams. Each team conducts two brainstorming rounds, on different but structurally similar brainstorming tasks A and B. As an experimental manipulation, work on one of the tasks gets disturbed, e.g. by “technical difficulties” that constantly disrupt brainstorming activities, so that study participants cannot achieve a Flow state. There are four different study conditions, to which pairs of participants are randomly assigned. (1) Participants work on task A first, then on task B; activities in task A get disturbed. (2) Participants work on task A first, then on task B; activities in task B get disturbed. (3) Participants work on task B first, then on task A; activities in task A get disturbed. (4) Participants work on task B first, then on task A; activities in task B get disturbed.

In this experiment, physiological data is acquired with the consumer-grade 4 channel EEG headband Muse 2 (which captures EEG, heart rate and motion), the wristband Empatica E4 (which captures electrodermal activity, heart rate and motion) as well as webcams to record faces (later analysed in terms of facial action units determined by the software OpenFace).

For each pair of participants, the study begins with equipment being put on. Then, baseline measures are obtained: 1 min with open eyes, 1 min with closed eyes (to elucidate eye-related artifacts in the EEG signal); followed by 5 min of talking on ‘mundane’ un-creative non-task related topics. Participants then obtain an introduction to the platform FlowCreate™ Innovator, where they will jointly conduct brainstorming. This screen-based brainstorming routine serves to reduce motion artifacts in the data, compared to standard design thinking brainstorming routines where people move around quite a bit; at present motion tends to induce high levels of artifacts especially in EEG recordings. The chosen brainstorming scenario with FlowCreate also resembles intended later use cases for biofeedback as envisioned by the project partner, e.g. as amendments to existing NeuroCreate platform software. In terms of experimental procedure, participants work in teams of two on brainstorming tasks according to one of the study conditions (1)–(4). They also fill out questionnaires that capture demographic data, subjective reports of Flow states in the course of the study and subjective reports of how challenged people felt by tasks A and B.

The data is analysed to identify patterns indicative of high vs. low Flow states in people. Research questions include: Can we identify a “physiological fingerprint” of Flow using consumer wearable technologies? Can NeuroCreate’s deep learning model, which has resulted from an analysis of sophisticated EEG lab data, also predict Flow states based on data acquired with consumer-grade hardware? Considering multivariate data from different sensors, such as EEG, motion, heart rate, EDA and webcam recordings of faces: which data and means of data analysis provide most reliable and valid indicators of Flow states, or would this rather be a combination of all?

A subsequent study by Samik Real and Sami Adnan (2020) compares the performance of three different deep learning approaches in determining Flow states based on EEG data. Two different datasets serve as input material. First, the team can work with a proprietary dataset of NeuroCreate, where 64-channel-EEG-recordings have been conducted while professional musicians improvised pieces of music. Later, expert judges reviewed audio of the musical performances and rated whether or not musicians were in Flow over time, alongside self-assessments of the participants themselves. This dataset showed consistent EEG patterns related to expert-judged classifications, and thus provides a very well-labelled dataset for training deep learning models. Secondly, data from the study of Holly McKee, Felix Grzelka, Laurenz Seidel and Pawel Glöckner can be analysed, where EEG data should be captured during brainstorming with the Muse 2 as a consumer-grade sensor technology. Here, the data is labelled once in terms of study conditions (Flow, uninterrupted work versus no-Flow, interrupted work) and also by means of self-reported Flow experiences of participants captured via questionnaires. All these EEG recordings deliver time-series data of voltage fluctuations measured on the skull, at the position of respective electrodes. An EEG power spectrogram can also be computed.

In this project, three different deep learning approaches are implemented and compared, to see which performs best in classifying Flow states. The first is a Recurrent Neural Network. This approach is currently state-of-the-art in assessing time-series data and/or language, or in the analysis of other datasets where judgements depend on “time of occurrence” and “sequence.” Models work with a memory of previous calculations in order to analyse subsequent inputs. However, a lot of computational power is needed to parallelize different analysis chains, because previous calculation results must be stored for subsequent analyses in each case. The second model to try is a Convolutional Neural Network (CNN). It uses images (such as power spectrograms or topographical images) as input. Filters are used to extract features/patterns automatically. This approach is currently state-of-the-art in the field of computer vision. To render the approach applicable in the realm of EEG assessments, it can be applied to EEG spectrograms. The third model to be tried is a Temporal Convolutionary Network (TCN). This approach is similar to CNN, but it works with one-dimensional data (like EEG raw ‘wave-form’ data of voltage changes), not with two-dimensional data (such as power spectrograms or topographical images). Thus, in contrast to the former CNN model, with TCN no initial data-conversion is needed. Moreover, in contrast to RNN, calculations can easily be parallelized.

Regarding the technical realization, Pandas, NumPy and ScikitLearn are used for the preprocessing of data. Models are created with TensorFlow in the backend and Keras in the frontend.

Major goals of the project are (i) to classify Flow states based on power spectra or wavelength patterns, (ii) to compare the performance of different deep learning models and (iii) to deploy models for real-time classification.

Since this project processes EEG data recorded by the end of the winter semester, it will be continued and finalized in the summer semester.

Ultimately, the vision of both projects is to help people achieve peak performance Flow mental states by providing real-time biofeedback on low versus high levels of Flow. This can improve people’s self-regulation and can also help people take informed decisions (e.g., when best to take a break from work). Artificial Intelligence could also make suggestions to users, or could automatically make adaptations in the environment (e.g., on the screen), that allow people to achieve Flow states more regularly. This would have direct implications for future ‘gamified’ designs of the FlowCreate™ Innovator.

2.10 Deep Learning on EEG Data of Team Collaboration

Electroencephalography (EEG) is one of the most common methods to record brain activity. It plays an important role in neuroscientific studies of brain functioning. It is also a central diagnostic tool in medical assessments regarding brain activity or brain-mediated functions, such as sleep.

The processing of EEG data is currently a time-intensive task handled by human experts. EEG data is noisy in several respects, and even in carefully conducted EEG lab assessments the data is full of artifacts. The mere act of eye-blinking in subjects induces strong artifact-signals in the recordings. Body motion and sweating in subjects also disturbs the signal. Moreover, there are so-called “electrode-drifts,” where the measured voltage of an electrode constantly increases or decreases over time. In addition, EEG recordings vary from one person to another; individual differences need to be accounted for in the data analysis. All in all, the preprocessing and analysis of EEG data is currently a resource intensive undertaking that can only be conducted by topic experts.

Deep learning is an approach of machine learning that invokes artificial neural networks. Pre-defined deep learning models are supplied with data, so that they can “learn.” Deep learning approaches may be very fruitful to facilitate the preprocessing and also the analysis of EEG data. One major advantage of deep learning is that it can be applied to EEG raw data—no prior filtering is needed. Indeed, the model itself can learn to recognise artifacts, and remove them if desired. Moreover, models can learn to detect any feature of interest (such as indicators of sleep disorders or EEG-patterns indicative of good creative performance), while at the same time learning to disregard uninformative artifacts. Results of deep learning models might even be better than outcomes of human assessors, because the machine can detect relevant patterns in the data that humans may be unaware of.

Tobias Bredow and Emanuel Metzenthin (2019, 2020) explore the potential of deep learning in one sample domain of EEG research that is specifically relevant to design thinking. Here the question is whether or not members of a team collaborate well together. Indicators of good vs. bad team collaboration shall be found in people’s EEG recordings.

In order for deep learning models to identify suitable indicators of team performance, well-labelled EEG data is needed. It must be clear what EEG data stems from well-collaborating teams and which data stems from not-well collaborating groups.

The data for this deep learning project is provided by Caroline Szymanski. The original neuroscientific study is discussed by Szymanski et al. (2017). In the experiment, 42 participants perform a visual search task first alone and then in teams of two. The EEG is recorded with 64 electrodes per person, each sampled at a rate of 5000 Hz, later reduced to 1000 Hz by a bandpass filter. Teams are classified as collaborating well together when they perform better together than any individual team member performed alone. The difference between individual performance versus team performance also provides a metric of how good the team works together. Each search round endures for an average of 7 s. In the original neuroscientific study, it is possible to determine good versus bad team collaboration by analysing the first 1.5 s of EEG recordings from each search round. In the analysis, brainwave-synchronization across team members is identified as a predictor of good team collaboration. By using statistical regression models that include brainwave-synchrony and other parameters, Szymanski et al. can explain 74% of the variance in team collaboration scores. (If regression models explained 100% of team collaboration scores, parameters in the regression model would suffice to determine exactly how well teammates work together in each search round.)

To train their deep learning model, Emanuel Metzenthin and Tobias Bredow first dichotomise team collaboration scores, so that each EEG recording is labelled as either indicating good or bad teamwork. Available EEG recordings up to 10 s in each search round are included in the study. Then, ten samples out of each 1 s (altogether 1000 data points per search task) are extracted for the training of deep learning models.

This project compares the performance of four different deep learning approaches: (1) A Convolutional Neural Net (CNN) applied to EEG raw data, (2) A Convolutional Neural Net (CNN) applied to pre-processed EEG data, (3) a Long-Short-Term-Memory model (LSTM) applied to EEG raw data and (4) a Long-Short-Term-Memory model (LSTM) applied to pre-processed EEG data. In cases (1) and (2), the CCN is realized with four convolutional max-pooling layers. It includes batch normalization. There is one dense layer with 100 neurons; the output layer has one neuron. In cases (3) and (4), the LSTM is a single unit long; it has one dense layer and one output layer.

The EEG recordings available to this project include raw data only. To obtain data for trials (2) and (4), the authors apply a high-pass filter at 0.5 Hz. This is a common EEG preprocessing step, though it has also been critically reviewed as impoverishing data quality (Tanner et al. 2015).

All deep learning models are validated on a set of 961 data samples not used for training before. This validation sample includes roughly equal amounts of “good team collaboration” (54% of the cases) versus “bad team collaboration” (46% of the cases). Results are reported in Fig. 12.

Fig. 12

The performance of four different deep learning models to predict good (1) versus bad (0) team collaboration based on EEG-recordings of team members [image reprinted with permission from Bredow and Metzenthin (2020)]

The CNN applied to EEG raw data performs best. It achieves an accuracy of 99%. This indicates a strong increase in prediction accuracy by means of deep learning compared to the best performing regression model reported in the original neuroscientific study.

Notably, applying deep learning on preprocessed data reduces the model performance in this test. The prediction accuracy drops from 99 to 92% with CNN models. While this result may be due to a non-ideal preprocessing of EEG data in this study, or to a CNN design that is not ideally suited for this kind of data, the neuroscientific community is also well advised to pay close attention to the finding. One century ago, breakthroughs were achieved in the field of statistics when Ronald A. Fisher (1925) began to analyse data that had previously firmed under the label of “measurement errors” and was thus disregarded. A quick recollection: While in statistical analyses, initially only the arithmetic mean compared across study conditions was considered informative, and deviations from the mean were disregarded as “errors,” Fisher pioneered thinking about these presumed errors. Might valuable information be extracted from them? This line of thinking led Fisher to invent the concept of data “variance” and statistical approaches to analyse it. Such procedures nowadays underlie most statistical computations. Similarly, neuroscientists may be encouraged to think about valuable information that might potentially be extracted from data that is regularly removed in the phase of data preprocessing. Even if humans do not presently know what to make of the information that is readily removed, deep learning models can potentially benefit from it already today.

The project by Emanuel Metzenthin and Tobias Bredow also has a practical “product” outcome. The authors have packaged a solution, so that interested neuroscientists can easily try out CNN or LSTM machine learning approaches on their own neuroscientific datasets. The package is available for free at https://pypi.org/project/deepeeg/0.1/. It supports (i) the training of models with chosen data, (ii) the application of high-pass, low-pass or bandpass-filters on EEG data and (iii) outcome reports to evaluate model performance. This package can be used by neuroscientists who have basic Python skills, but have no deep learning experiences and thus could not set up CNN or LSTM models themselves. By trying out the models on datasets, neuroscientists can explore whether such deep learning approaches may provide fruitful solutions in a work area of interest.

2.11 The Impact of Remote vs. Face-to-Face Collaboration on Team Performance

In the industry, remote collaboration becomes ever more popular (Bitkom 2020). Increasingly often, companies provide opportunities for employees to work in “home office” or at other self-selected locations. This has not only become an option for employees who work on rather self-contained tasks, but just as much for employees who are supposed to collaborate with colleagues.

In a noteworthy contrast to industry trends, research on team performance suggests that collaboration tends to be more effective in teams that work face-to-face as opposed to teams that collaborate remotely (Fletcher and Major 2006; Szymanski 2019). Yet, research in this field has only just begun. Fletcher and Major analyse self-report data. How about measures of behavioural performance or physiological assessments? Szymanski draws attention to tendencies in an array of neuroscientific studies that seem to imply disadvantages of remote collaboration; however, the studies themselves have not been specifically designed to elucidate the contrast of face-to-face versus remote teamwork.

This situation provides the background of a study by Justus Hildebrand, Kim Borchart and Hendrik Rätz (2019, 2020a, b), which pursues two ends. First, a dedicated experiment shall be conducted where face-to-face versus remote team collaboration is compared directly. It shall yield quantitative measures regarding behaviour and physiology. Second, the study seeks to draw attention to different contexts, which can impact dynamics of remote collaboration. In some contexts, remote collaboration may be very difficult, whereas in other contexts it might work well. This study explores a context where remote collaboration is likely to be rather successful—and thus might also provide a model for successful remote collaboration elsewhere. In particular, in this study (i) team members share a clear and emotionally engaging vision: They do not get to work on arbitrary, tedious work objectives, but instead play a captivating online game that they want to win together. Furthermore, (ii) team members know each other personally very well before engaging in remote collaboration. Factors (i) and (ii) have been discussed by Szymanski (2019) as likely means to facilitate remote teamwork. In addition, (iii) there is an engaging online game environment where collaborators meet each other digitally, which could provide a substitute for the experience of face-to-face interactions in the real world.

In this pilot study, a pair of participants (friends) get to play the game Counter Strike in wing-man mode. They play 20 rounds altogether. First, participants play 10 rounds in a remote-collaboration scenario where verbal communication occurs via voice chat through Discord. Then, on another day, 10 rounds are played with participants co-located in the same room, allowing for face-to-face communication. All games are played on the same map (“Inferno”) and participants use the same hardware in both settings. During all games, participant faces are filmed with mobile phones positioned at a fixed location in front of each player. Video data is later analysed via the software OpenFace, which extracts “action units” related to different parts of the face, such as eyebrows or corners of the mouth. Thus, facial gestures indicative of emotions can be analysed automatically. In addition, objective game data is captured, such as round length, matches won versus matches lost etc. All in all, this study covers ca. 230 min of gameplay. It yields about 500 min of video material (47 GB), and 208 MB of OpenFace data.

In the data analysis, team synchrony is used as an indicator of successful team-building and good team performance (cf. Szymanski 2019). The study specifically considers emotional synchrony. For each peak of one player on an analysed emotional dimension, such as happiness, the time frame of 1 second is screened in the other player, to assess whether this person also produces an emotional peak on that emotional dimension. Thus, dichotomous results are obtained: Regarding each emotional peak of one player, emotional synchrony is either attributed or not to the other player, depending on whether or not this person produces a corresponding emotional peak in the 1-second-frame of analysis. Overall, the assessment covers emotional dimensions labelled as “happiness” (OpenFace action units 6 + 12), “sadness” (action units 1 + 4 + 15) and “anger” (action units 4 + 5 + 7 + 23).

In terms of findings, it is first of all noteworthy that significantly more emotions get expressed in the face-to-face situation (p = 0.006), as Hildebrand et al. (2020c) report. However, more emotional synchrony is found in the remote-collaboration condition (p = 0.023), as depicted in Fig. 13.

Fig. 13

More emotional synchrony is found in the remote compared to the co-located setting [image reprinted with permission from Hildebrand et al. (2020c)]

As an interpretation, emotional expressiveness might generally be greater in face-to-face interactions, allowing for multi-facetted communication. In remote collaboration, people might only express emotions when the subjective sentiment was strong; thus, in well-collaborating teams a lot of synchrony could likely be found. This may hold even more when team members pursue a joint goal and react to the same events in their digital environment. Thus, team synchrony in such remote scenarios could be due more to joint immersive experiences and joint goals than reactions to each other.

Overall this study finds quantitative indicators of collaboration working fairly well in remote teams, i.e., teams producing rather high levels of emotional synchrony. This positive finding could be rendered possible in this particular study by context parameters such as (i) team members knowing each other very well prior to remote collaboration, (ii) a captivating joint goal of winning a game together and (iii) an engaging virtual environment where teammates collaborate, substituting for face-to-face meetings in the real world. On the other hand, emotional expressiveness was found to be much reduced in the remote collaboration condition. That means teammates only obtain impoverished information from each other. Possibly information transfer can be optimised for good team performance regarding clearly defined goals. Yet, in many work contexts the rich information content of numerous, emotionally expressive facial gestures might provide benefits that need to be elucidated in further studies.

2.12 Healthcare as a Domain of Neurodesign

Neurodesign has been introduced at the HPI as a field at the intersection of (i) neuroscience, (ii) engineering and (iii) design thinking · creativity · collaboration · innovation. In their project, Manisha Manaswini and Maroua Filali rethink what neurodesign can be and what it can do. They emphasise how the field of healthcare has a lot of overlap with neurodesign objectives, even though descriptions of neurodesign have not yet mentioned the keyword “healthcare” explicitly so far.

Why and how is healthcare relevant to neurodesign?

First, neurodesign is concerned with the physiological underpinnings of creativity and collaboration. The dimension of health-to-illness is relevant to people’s inclinations and abilities of being creative or collaborative. For instance, neuroscientific research has found that physiological damages at certain brain regions impact people’s creativity (Mayseless et al. 2014) and collaboration (Flanagan et al. 1995). Moreover, linkages between mental health and creativity (Maslow 1962; Kaufman 2014) or linkages between mental health and people’s ability to engage with others constructively (Widiger 2012; Livesley and Larstone 2018) are long-studied topics.

Second, “neuroscience” must be understood in a broad sense in the realm of neurodesign. It is not only about the brain, it is about the whole body. Many “neurodesign assessments” that have been endeavoured in recent neurodesign projects track body motion, heart rate, facial gestures, electrodermal activity etc. (cf. neurodesign missions and other neurodesign projects). Thus, under the headline of “neuroscience” the community is interested in all body-related data. Information on health versus illness is body-related data, and therefore relevant to include in neurodesign studies.

Third, some novel healthcare solutions are examples of innovative engineering. Ideally, they also have a design thinking focus on human needs, i.e., they are well-designed to deliver exactly what the users need.

Fourth, there can be a fruitful interplay between healthcare solutions and solutions for the analysis and facilitation of creativity or collaboration. This is what the project of Manisha Manaswini and Maroua Filali (2020) sets out to elaborate, based on suggestions for a technology transfer from known healthcare applications to possibly fruitful, novel applications in neuro-design-thinking contexts.

One example is the capturing of sound-levels via microphones that has already been pioneered in health studies (e.g., Goelzer et al. 2001). Design thinking, or creative collaboration more generally, typically involve varying sound levels as well. Microphones can be used to extract only loudness information in a room. Thus, no personal information regarding conversation content or speakers is captured and the data is strictly anonymous. Such loudness information cannot only be used to study impact on health, but also to elucidate the impact on creative or collaborative performance. Research shows that high sound levels in the environment tend to impair concentration and communication, though dynamics seem to be complex (Dalton and Behm 2007; Keller et al. 2017). In design thinking, music is often used to guide the mood of creative teams. Sometimes, music is played rather loudly to prevent discussions—conversations get difficult when the sound level in the room is high—, to encourage instead the use of hands and a bias to action during prototyping. Also apart from music, the open environment of design thinking spaces filled with many actively working teams tend to create high levels of background noise. This is often perceived by design thinking participants as “activating.” However, there are also anecdotal reports of people feeling exhausted after “noisy” design thinking days (Meinel et al. 2017). Thus, a better understanding of the role of sound levels on creative performance and recreation needs after active design thinking would seem highly desirable. The same holds for investigations into the impact of different sound-types (e.g., conversations vs. music) on design thinking performance in varying process phases.

In healthcare, motion has been tracked with devices such as the Microsoft Kinect, for purposes such as monitoring the risk of falling in elderly people (Parajuli et al. 2012) or to track respiration (Ernst and Saß 2015). Creativity research indicates a causal impact of posture and body motion on creative performance (Leung et al. 2012; Slepian and Ambady 2012; Oppezzo and Schwartz 2014; Andolfi et al. 2017). Again, a re-use of sensor technology that has been successfully deployed in health studies, now for neuro-design-thinking research purposes, seems highly promising. In addition, communication and collaboration are clearly impacted by the way in which people move, e.g. the gestures they make. Thus, sensor technology to track motion—especially when there is also a potential of recognising gestures—would seem highly serviceable for design thinking creativity and collaboration research: It allows more systematic analyses of how motion relates to creative and collaborative performance. Once clear patterns have been established, feedback could be given to individuals or teams based on motion assessments.

Another area where neurodesign can benefit from technology already deployed in health studies concerns recreation. Sleep is a relevant subject for design thinking research. The role of sleep, relaxation and dreaming for creative performance has long been emphasized in the community (McKim 1972). Experimental evidence underpins the belief that sleep is indeed important for people to perform well on creative tasks (Wimmer et al. 1992; Marguilho et al. 2015). To allow for a contact-less and therefore convenient ways of sleep-tracking, the tool Sleepiz can be suggested. It allows everyone to spend nights without the disturbance of wearables, and yet ample data is generated that permits a grand scale analysis of how sleep and creative performance interrelate. With a clearer understanding of this relationship, advice could also be given adaptively to design thinkers, as to when some rest would likely increase people’s prospects of finding good, creative solutions.

These are just some examples of how sensors that have been successfully deployed in healthcare contexts could be re-used to facilitate design thinking—creativity—collaboration research. Readers are invited to think up further, possibly fruitful areas of application. All in all, the field of healthcare has many resources to offer that can be highly pertinent for the emerging field of neurodesign.

2.13 Neurogaze: Exploring the Potential of Eye Tracking in Digital Engineering

The standard way to interact with computers—by mouse clicks—was invented more than half a century ago. This traditional solution is currently being reconsidered. Might there be other or additional ways to interact with computers that would seem highly fruitful? Computers or other digital solutions might react to human emotions. What if people could direct computers with the power of thought, i.e., by means of recorded brain activities? Moreover, heart rate data or other physiological parameters could provide relevant input for digital applications. The present study by Philipp Bode and Christian Warmuth (2020) explores yet another sensor-based approach for novel solutions in human-computer-interaction: eye tracking.

The human eyes have long been considered a “window into the mind.” Both historically and recently, interest in the eye and gaze tracking is great, so that sophisticated theoretical understandings have already been achieved in this field (Radach et al. 2003). Eye tracking has been discovered as a field of great potential—but also as a field of some technical challenges—by scholars of human-computer interaction and usability studies (Jacob and Karn 2003; Poole and Ball 2006; Majaranta and Bulling 2014; Zhang et al. 2017).

In their project, Philipp Bode and Christian Warmuth (2020) take a close look at this promising field of research and make three major contributions. (1) They have developed the tool “Neurogaze,” which supports the assessment and interpretation of eye tracking data. (2) They have conducted pilot studies on the use of eye tracking in varying application contexts. (3) They sketch out a research agenda regarding the use of eye tracking to determine the “cognitive load” in a user, so as to facilitate the design of adaptive systems.

In their project, Christian and Philipp work with the Tobii Pro Nano Hardware Package. This technical equipment supports eye tracking in one person at a stationary position; it is designed to record eye activities of a user who looks at a computer screen. The sampling rate of this device is 60 Hz, thus it yields 60 data points per second. As a professional eye tracking device, it produces highly reliable and valid eye tracking raw data.

To complement the sensor hardware, Christian and Philipp have programmed a python-based tool, Neurogaze, that supports the gathering and interpretation of eye tracking raw data. The tool has two major parts.

First, there is the Application Tracker. It documents over time which application is open (e.g., Firefox), which Tab is open (e.g., Google search), click-events in each window, how windows are positioned on the screen etc. Thus, eye tracking data can later be used to indicate where a person was looking at a certain time, e.g. on a particular website. Since eye tracking hardware delivers many data points per second, large amounts of data accumulate quickly. Thus, in the software design care has been taken to enable efficient data storage, by using binary representations to reduce storage overhead.

Second, the Gaze Tracker stores information provided by the eye tracking device. It collects 3-dimensional information of how the eyes are positioned in front of the sensor bar. Thus, one can tell, for instance, how far a person sits away from the screen and how she moves over time. In addition, the Gaze Tracker collects information about pupil diameters and about x-y coordinates of gaze on the screen (i.e. where the person is looking from moment to moment). Moreover, the Gaze Tracker supports an accumulated display of gaze points for a chosen time frame. Thus, when the user looks at a horizontal chat-input-bar for one minute and rarely looks elsewhere, the display of accumulated gaze points in the one-minute-timeframe shows large numbers of data points forming a horizontal “bar” exactly at the position where the chat input occurs on the screen.

Neurogaze is available at: https://github.com/christianwarmuth/neurogaze.

In pilot studies, Christian and Philipp recorded their own eye activities in front of their computers over a couple of days, using the Tobii Pro Nano and Neurogaze. With this pilot data, several interesting applications can be pursued. First, by accumulating gaze point data over longer times, “activity profiles” can be displayed regarding applications of interest. Thus, it can become obvious how a user is strongly focused on a “message input bar” in a program, how the person occasionally reads messages from others that appear at a certain location on the screen, and otherwise disregards large portions of the window (Fig. 14).

Fig. 14

Accumulated gaze points on the x-y coordinates of the screen indicate that the person has mostly looked at the message input bar in the application of interest [image reprinted with permission from Bode and Warmuth (2020)]

A second interesting analysis is to consider gaze motion across the x-y coordinates on the screen separately. For instance, it can be tracked how gaze moves between left versus right on the screen, by analysing gaze data on x-coordinates only. Plotting this data yields a characteristic graph, e.g., when the person is reading. During that activity, her gaze moves regularly between left versus right (Fig. 15). By analysing derivations of this curve, velocity and acceleration data can be obtained. These provide a good basis for activity classification. For instance, in reading there is usually a relatively low level of gaze motion velocity as it takes people time to read bit by bit until the end of a line. Yet, at regularly occurring intervals, gaze motion speeds up drastically as the gaze shifts rapidly towards the beginning of a new line (Fig. 16). This pattern is significantly different from eye-motion during other activities, such as programming or drawing.

Fig. 15

Eye motion during reading: Plotting gaze-motion on the x-dimension of the screen yields a characteristic pattern in the case of reading [image reprinted with permission from Bode and Warmuth (2020)]

Fig. 16

Eye motion during reading: Derivations of the graph in Fig. 15 yield a graph depicting gaze motion velocity that can be used for activity classification [image reprinted with permission from Bode and Warmuth (2020)]

Another interesting application is to look at moments in time where the eye tracking sensor does not record data. Losses of signal for a duration of roughly 100–400 ms most likely indicate eye blinks of the user. Analysing their own data, the authors find a clear pattern in their eye activities, with thus determined eye-blinks: Over the course of a workday the number of eye blinks increases significantly. This may reflect increasing tiredness of the eyes, increasing tiredness of the person, or both.

Based on adaptive threshold values (Dar et al. 2020), Christian and Philipp also classify eye activities. They distinguish between fixation (steady gaze on a point), pursuit (such as following a moving object or reading a line) and saccades (the rapid change of gaze points).

Third, Christian and Philipp envision a research agenda for the use of eye tracking to determine cognitive load and thus support the development of adaptive systems. “Cognitive load” is a concept known to the public especially in the variant of “cognitive overload,” meaning a situation where the awake person cannot process novel information productively any more. In science, Cognitive Load Theory (Sweller 2011) explores phenomena regarding human processing abilities based on characteristics of the human biological information processing “architecture.”

Cognitive load is a relevant topic for system design. Ideally, systems deliver information to the user in such a way that he or she can easily process the content. If for instance a car driver is experiencing cognitive overload, this situation can pose serious risks for the safety of the driver and others. A well-designed adaptive system might help to reduce the driver’s cognitive load by down-regulating information complexity; moreover, safety-relevant information may have to be emphasized for the driver.

As Christian and Philipp submit, eye tracking could potentially provide reliable and valid indicators of cognitive load. One possible indicator they suggest for further study is the rate of eye blinks. In their pilot studies, they had found increased rates of eye blinking in the afternoons compared to morning hours; this might correlate to people approaching levels of cognitive overload in the afternoon while people enjoy fully available processing capacities in morning hours (after recreative sleep). A second eye tracking parameter that could indicate cognitive load might be microsaccades, as already suggested by Krejtz et al. (2018).

To elucidate the psychometric properties of eye blink rates and microsaccades as measures of cognitive load, Christian and Philipp suggest a repeated-measures-study according to the “n-back” paradigm (described and applied in Bedford et al. 2009). Here, participants are presented with digits one after another and need to recall elements that came earlier. For instance, participants can be presented—one after the other—the following digits: P, X, T, S, with “S” being the most recent stimulus. Asked about previous digits, it is relatively easy for participants to move one step backward and report on the stimulus right before the present one (T). Cognitive load increases as participants are asked to move further steps backward, e.g. to report on the digit that came three steps before the present stimulus (P). In the data analysis, both the reliability and validity of eye blink rates and microsaccades as indicators of cognitive load (“numbers of steps backward”) can be explored.

3 Neurodesign Missions

As a third resource, we share missions and aims that have guided the development of neurodesign at the HPI in the first implementation phase. As results have been found fruitful, these goals will continue to inform developments of the curriculum and related research programmes.

3.1 Understanding Innovation Beyond Fragmentation

Innovation is a basic phenomenon of culture development—in humans and potentially also in other species. Studies into this ubiquitous phenomenon necessarily take place on multiple levels of observation, from macro- to micro-levels (Fig. 17).

Fig. 17

Design thinking—creativity—innovation research explores multiple facets in a complex realm of phenomena. Research tends to focus on fragments and puzzle pieces. For an understanding of creativity and innovation at large, knowledge sharing and holistic views are important as well [graphic inspired by Batey (2012)]

In design thinking innovation research, different facets of the phenomenon have been elucidated in detail. Studies have addressed characteristics of creative people, creative processes, the impact of places, creative products, and other topics.

Obviously, innovation studies also benefit from a great variety of research methodologies. These methods include objective assessments (such as performance metrics in the assessment of products, or EEG-recordings in human research), self-ratings (e.g., people finding their own work outcome more versus less creative) and other-ratings (for instance, experts judging the originality of participants’ test responses).

Innovation researchers develop deep expertise by concentrating on some pieces of the overall puzzle. This fragmentation is a helpful and maybe necessary manoeuvre to advance sophistication in specific directions. Yet, to understand innovation as a whole, it is important to move beyond fragmentation.

3.2 Professional Bridges Between Theory and Practice

Much research on the biological basis of creativity and collaboration is conducted in neuroscientific labs, far away from the studios where creativity, collaboration and innovation are taught and realized on a daily basis. The gap between research and practice is often so severe that only popular-writing authors with limited pre-experiences in at least one of the domains endeavour publications that span both fields. We believe that regular exchange between research and practice is important for the prosperity of both. Ideally, this exchange is a priority shared by leading experts from both sides and is promoted at high levels of professionalism. We believe this exchange between research and practice can happen best in teams or larger groups, where members of varying backgrounds make joint experiences and collaborate (Fig. 18).

Fig. 18

Neurodesign seeks to bring research and practice closer to one another, by facilitating personal exchange among experts of both, and by facilitating joint projects

3.3 Inspiration from Nature for Digital Engineering

Gaps between the analogue versus the digital—or the natural versus the artificial—are often large. For instance, living creatures including humans naturally spend their days moving about, engaged in manifold physical activities. By contrast, today for many persons around the globe digitally organised work enforces relatively static body positions over multiple hours per day, such as sitting in front of a computer with limited motion opportunities. How might digital engineering solutions of the future facilitate natural human behaviours rather than imposing non-natural behaviours onto people?

Humans naturally possess many senses and use them to navigate the world, such as the senses of vision, hearing, smell, touch, taste, of body position, temperature and so forth. Present-day digital solutions commonly address very few of the human senses. Information is usually provided visually on a computer screen, commands are typically entered through the banal motion act of pressing buttons.

In our digital age, many people need “mindfulness trainings” where they learn by means of time-intensive training how to better attend to the full set of human senses, including smell, touch, body position etc., in order to achieve psychological wellbeing. By contrast, in many hours of human-to-computer or human-to-screen interaction per day, humans learn the opposite: how to disregard and not use these senses. Indeed, senses of smell, body position etc. would ground people in the here-and-now of their physical environment—often a context that is very different from things going on at a screen (where people indulge in work objectives, game or film scenarios). Digital solutions that lead people to unlearn the natural use of their full set of senses are unhealthy; they call for a re-design.

How might digital engineering solutions of the future help people use and enjoy all of their senses?

Human-to-human communication is highly multifaceted and takes place on many levels. There is exchange by means of language and tone. Much information is conveyed visually and sometimes audibly by means of gestures, postures and motion. People touch each other. There is also the huge domain of communicating by action: doing something that conveys intentionality—revealing what the person is trying to do; making physical arrangements in space, such as creating physical prototypes; doing something meaningful, or doing something unintentionally that still conveys important information about the acting person. Given this richness in human-to-human communication, it is surprising how limited communication channels are in human-to-machine interactions. For instance, the programming of a digital engineering solution is usually language based, and barely any representation system other than language can be used. How might the programming of machines take place in different representation systems, beyond language? How might the greatest possible variety of representation systems be used in human-to-machine interactions?

Also, human-to-human interactions are characterised by role-flexibility. Even a simple conversation often means to have changing leads and initiatives in a single minute, such as person A making a contribution and then person B making another independent, sensible contribution. When humans program digital engineering solutions, the roles of a human versus the machine tend to be static. How might we enable humans and machines to flexibly adapt their roles during interactions?

In general, living organisms are an effervescent source of inspiration for artificial designs. Nature has created solutions that are fascinating and highly effective in many different regards. In which ways do we want artificial solutions to become better? What examples from nature could inspire novel and more effective technical solutions?

3.4 More Digital Engineering in Neuroscience

We want to leverage more digital engineering expertise in the field of neuroscience. This aim is based on observations about traditional career paths in neuroscience. Many neuroscientists are originally trained in fields such as psychology, medicine or biology, some also in physics. Yet, the day-to-day tasks of neuroscientists are to a large extent digital engineering tasks, such as data processing, data modelling or data visualisation. Furthermore, new research designs that take neuroscientific experiments outside of the lab to study human behaviour in a “natural” environment are often requested. However, these settings typically require different and new technological solutions and analysis procedures, compared to traditional lab-based approaches. Therefore, we believe that the creativity and expertise of digital engineers is very fruitful when brought to play in the field of neuroscience on a much larger scale.

3.5 An “Embodied Cognition Perspective”

Design thinking has a long tradition of attending the whole body, both in research and practice, to promote good design and radical innovation. The approach builds on concepts such as visual thinking, which elucidates the role of the senses in creative design, and ambidextrous thinking, which emphasizes the importance of bilateral body engagement for creativity and innovation. In more recent terminology, neurodesign invokes an “embodied cognition” perspective on creativity and collaboration. With this outlook, it helps to underpin design thinking in two major respects. First, neurodesign embeds prevailing design thinking interventions like bodystorming in theoretical frameworks. Thus, it enables innovation practitioners to make use of such techniques in more goal-directed, efficient ways. Secondly, neurodesign guides the development of novel interventions based on concepts of embodied cognition. Altogether, we encourage physiological research beyond the brain. Also in creative design practice, we track and facilitate full-body engagement.

3.6 Novel Career Models for Neuroscientists

We want to create novel job opportunities for neuroscientists and support a greater variety of life models in the field. This concern is based on observations about career choices that many well-trained neuroscientists need to make during or after their PhD. To progress on the academic career path, neuroscientists are often expected to be equally active in fields such as study planning, data acquisition, data processing etc., leading to a weekly workload considerably beyond 40 h with little flexibility to adapt life models on demand. Also, work beyond academic research—even when concerned with the same phenomenon otherwise studied in the lab—is rarely appreciated; it rather tends to impact people’s career prospects negatively. While for example in clinical neuroscience the exchange between lab research (e.g., on depression) and application (e.g., clinical depression treatment) is lively and well-integrated into many research programs, this is rarely the case in social neuroscience. Here, neuroscientists that research phenomena such as collaboration, creativity or social (team) interaction are expected to dedicate all their time to lab research and their career development is based on scientific publications only. In neurodesign we decidedly promote both: exchange between researchers and practitioners, as well as giving individuals the career option to distribute their time between neuroscientific research and applied work in their domain of expertise.

3.7 Merging Science, Engineering, Philosophy and Art

Ever since its earliest beginnings, design thinking has a strong tradition of merging science, engineering, philosophy and art. This is evident both in terms of collaborations across disciplines and in terms of people’s individual biographies. When John E. Arnold lay important groundworks for design thinking at Stanford University, he did so in close collaboration with guest experts who personally came to lecture in the courses; these guest experts had varying academic backgrounds covering science, engineering, philosophy and art. Moreover, in terms of personal biographies, many design thinking protagonists unite two or more disciplines in deep ways. John Arnold was a social scientist and engineer. Buckminster Fuller and Robert H. McKim as two other protagonists who shaped the emerging concept of design thinking in early days both drew from engineering and art alike, with regard to their public works and personal lives. Beyond that, institutional co-operations reflect the bold intention of merging disciplines. Notably, the Joint Program in Design at Stanford University was offered collectively by the Mechanical Engineering Department and the Art Department. Neurodesign continues in this legacy. We wish to include experts from all fields in the neurodesign curriculum, who address creativity, collaboration and innovation from varying academic perspectives. Moreover, we encourage students to work across the borders of traditional disciplines. This can happen either in interdisciplinary teams or also in single-person work. Participants in neurodesign courses are invited to invoke artistic freedom, and to include artistic aspirations, in the creation of innovative engineering solutions for neuroscience and beyond.

4 Outlook

Based on stirring experiences in the first semester of neurodesign education at the HPI, it is clear that this area of research, teaching and practice is extraordinarily fruitful and shall be further advanced. The next step is to establish a more comprehensive curriculum at the HPI, which covers neurodesign-relevant topics in an increasingly systematic way.

The format of seminars has been found highly practical to convey relevant methodological knowledge. It also yields sufficient creative freedom for student teams to think up innovative project ideas and iterate towards amazing solutions.

A greater number of different seminars shall be developed in the near future to provide even more in-depth knowledge regarding methodological topics and the devising of scientific publications. Here, the curriculum shall become increasingly well-orchestrated with courses developed by colleagues at the institute. As a notable example, Falk Übernickel (holding the chair of design thinking and innovation research at the HPI) offers courses on social science research methodology and scientific writing skills for PhD students, which can inspire much fruitful exchange.

Another area where further explorations shall be endeavoured concerns ultra-feasibly methods for the assessment of body data. Already in the last semester, the repertoire of tools increased rapidly. At first, parameters such as skin conductance, heart rate and arm motion were measured with Empatica E4 wristbands, automated emotion-analyses were conducted based on webcam-recordings, and the Kinect was used to analyse body-motion. Soon the available equipment came to include four different technical solutions for EEG assessments, eye-tracking technology and various further body sensors. This repertoire shall be further extended and opportunities as well as limitations of such ultra-feasible methods shall be explored in further detail.

In the upcoming teaching term, the seminar Ambidextrous Thinking (2020) at the HPI will provide an opportunity for students to acquire or deepen knowledge in theoretical, methodological and technological neurodesign domains. The course is partially an iteration and partially a continuation of the Neurodesign Seminar (2019/20). In subsequent teaching terms, a larger number of seminars might be offered at the HPI to allow for more in-depth probing regarding specific topics of interest.

One specific area where selective seminars can be expected in the near future concerns data sonification. Marisol Jiménez joins the neurodesign teaching staff at the HPI for upcoming courses. She completed her Doctor of Musical Arts at Stanford University in 2011, where Chris Chafe was one of her teachers. Marisol’s works include composing numerous electroacoustic music works, sound and mixed media installations as well as the prototyping of instruments that permit experiments with the tactile process of creating sound—possibly a new field for human-machine interaction. Marisol will co-teach in upcoming HPI courses of the summer semester 2020 and might subsequently offer unique ambidextrous sonification seminars.

Further in-depth courses regarding neuroscientific topics shall be developed. Here, more intense collaborations with the Psychology Department of Potsdam University, specifically the master program “Cognitive Science—Embodied Cognition,” can be a fruitful avenue for further advancements of the curriculum.

Beyond the seminar, the format of a lecture series with guest expert talks has been found highly effective to collocate up-to-date knowledge regarding various work domains relevant to neurodesign. In such a lecture series, students hear about specific topics from those experts who have been researching the particular field intensely over many years, so that best-informed overviews are provided and the latest state of knowledge is conveyed.

In the past semester, by means of invited guest experts, great emphasis was placed on the neuroscience of creativity and collaboration in human research. While these topics are certainly central for neurodesign, they are not at all exhaustive. In subsequent years, lecture series might be hosted to advance the neurodesign cross-disciplinary knowledge base in further directions. For instance, there could be a neurodesign lecture series that is more directed towards digital engineering than neuroscience. Another lecture series might be dedicated to comparative creativity and innovation studies across species; possibly, it could also cover “artificial species” such as particular digital engineering systems. Moreover, the expanding HPI advances novel work domains such as Digital Energy, which can also inspire courses. The energy efficiency of the human brain was already discussed in some neurodesign inputs (Agnoli 2019; Plank 2019). It could be an interesting comparative topic for further classes, where topics such as the following might be explored: (i) Roughly, human brains work in a low-energy mode when the person utilizes status quo knowledge. By contrast, learning something new and changing one’s cognitive system as required for radical innovation is energy-intensive. (ii) Political and bureaucratic systems regulate levels of continuity versus change in states partially by regulating energy-demands of different endeavours. Most commonly, states facilitate continuity with the past, while making change difficult: Inhabitants or organizations need to invest much more energy to obtain permissions for endeavouring change than to obtain permissions for leaving things as they are. E.g. inhabitants don’t need to dedicate energy to bureaucratic activities in order to leave their houses as they are; however trying to build a new house can be a very energy-demanding undertaking measured by the bureaucracy people need to manage. (iii) There is a multiplicity of digital engineering solutions designed to be energy-efficient. These can be compared to energy-efficient solutions in the field of biology. Overall, somewhat varying core topics can be addressed in upcoming neurodesign lectures from year to year, where ideally experts from varying backgrounds contribute up-to-date knowledge in each semester.

Beyond such a collocation of expertise from different disciplines and institutions by means of guest expert talks, there is also the objective of systematically amending in-house knowledge concerning the theoretical basis of design thinking. In previous years, empirical studies and theories on why and how design thinking works have been conveyed in the lecture Design Thinking for Digital Engineering. So far, this course was mostly concerned with research findings engendered with social-science methods, such as psychological experiments or survey studies. Clearly, this knowledge stock needs to be amended with insights regarding the physiological basis of design thinking phenomena. This is endeavoured in the upcoming semester in a lecture now called (Neuro-) Design Thinking for Digital Engineering.

Finally, neurodesign thrives as a cross-disciplinary and cross-institutional initiative. It draws its power from mutual curiosity and passionate collaboration. We do our best to facilitate further fruitful developments beyond single locations. Institutions that wish to build up neurodesign themselves are more than welcome to get in touch and we will support initiatives as we can. We also very much appreciate personal exchange, such as persons coming from one institution to spend some time at another. In addition, on the neurodesign homepage we publish project calls, where experts from either domain—(i) neuroscience, (ii) engineering or (iii) design thinking · creativity · collaboration · innovation—can invite projects at the intersection of fields. These can be endeavoured in the form of bachelor, master or PhD projects. Also students with expertise in one or more of the fields are strongly encouraged to unfold their full creativity and explore what they can render possible in the novel realm of neurodesign.