Keywords

1 Introduction

Design contexts for Human-Computer Interaction (HCI) are becoming increasingly complex. While HCI designers understand the traditional tools of the HCI trade (research and design methods, e.g. [2]), contexts in which they work, and their complexities, may be less familiar. Research through Design (RtD) [3,4,5,6,7,8,9] is gaining traction in HCI as a way to explicitly engage in combining design studies, design experimentation and design practice. The approach offers, through the design of an artefact and its use, an opportunity to gain a deeper understanding of the context, as well as the fit between the context, the designed artefact and its use.

The objective of this paper is to critically reflect on how research through design inquiry can be used to gain new knowledge within a novel design context, i.e. context that is not familiar to HCI designers. We make use of the interaction design triangle presented by Fallman [1] to structure and describe our process of repeated drifts between design studies, design practice and design exploration.

The backdrop for the practical part of our project was a large European research and innovation project, The People’s Smart Sculpture. It focuses on effects of technology and digitalization on urban living. In particular, the evolution of the urban cultural sphere was of interest. As a consequence, in the context of real-life, project based RtD course, we were challenged to explore the use of technology in dance. The timeframe for the project was only five weeks. Dance and technology span a challenging design space, with some beautiful examples of work [10,11,12,13,14] yet not too many HCI researchers and designers are working in this space. One of the reasons for this situation is that dancers often needed to wear sensors that would limit their movements or were required to pay attention and relate to different technologies while dancing [10, 13], something that may be disrupting the flow of the movements.

A dance performance, as an artistic and cultural form, engages both visual and auditory senses. The music, in a usual workflow of a dance performance set up, is chosen prior to work on choreography. Thus, music leads the choreography, dance and the performance and contributes to building a coherent visual and auditory experience [15].

With the inquiry as broad as the one we started with, we initially questioned how the technology can be used to enhance, or bring something new to a dance performance, coining the term addhance to describe our aim. However, by engaging in design studies and HCI literature around possibilities to addhance the performance, we have discovered an opportunity to disrupt the common workflow by making dancers movements the primary driver of the experience, thus producing music as a result of performing movements.

In what follows we describe how the explorations were carried out, and what insights were gained. We challenged ourselves to think big, to think how digital design and designed artefacts could truly revolutionize the dancing experience. The outcome of our design practice, i.e. the artefact we made, shows a much more constrained set of opportunities than what our vision is. However, the designed artefact, Musical Moves, is a research product [16] that enables further exploration of the design space and possible relations between the dance and the technology. Musical Moves is based on a movement capture using the Microsoft Kinect. Watching numerous exemplars of dance performances and carefully analyzing how the body parts move, we have developed an initial language that translates specific movements into sounds. Thus, the movements captured by Kinect are translated into a series of harmonious sounds and provide a new opportunity for finding joy in dance movements, also for others than dancers, e.g., children.

2 Methodology

We adopted a research through design (RtD) approach to carry out a series of exploratory activities to deepen our understanding of the relation between dance and technology. The term “research through design” comes from Frayling [17] when he presents a descriptive framework for research in arts. However, the term was later expanded, and here, we use it as defined in ([6], p. 3):

[…] a type of research practice focused on improving the world by making new things that disrupt, complicate or transform the current state of the world. […]

This definition of RtD helps us argue that the chosen methodology is suitable for our work. In his paper on research through design [5] emphasizes the importance of conceptual articulation in leading design practice. In [6], by using Koskinen et al.’s [18] framework, is described how critical design approach can play an important role in RtD project cases. Led by a critical approach, researchers make provocative artefacts that drive people to think and reconsider some aspects of the world.

In our case, we focus not just on design explorations and practice, but we engage in reflections and critical thinking toward a conceptual formulation and maturation [5, 6]. We took a concept driven critical design approach, where theory and concept, lead our design decisions. The intention of our project was to change the current workflow of making a choreography. Through this, we want to provoke a new way of thinking and a different perspective on the dance performance and consequently stimulate discussion and make new knowledge contributions [19, 20].

RtD represents a combination of research and making. Differently from the design practice where the intention is to design commercially successful products, in RtD artefacts are designed to be carefully crafted questions, questioning “what it might be” rather than “what it is” [5, 19]. Also, by making things and placing them into the world, RtD can change the current state, creating new situations and new practices for anthropologist and researchers to investigate [6]. Additionally, Gaver’s approach on RtD [5] is more ludic, where the designed artefact is more explorative and open. Rather than trying to solve a problem, or do the right thing [17], the design researcher tries to explore towards some goal, such as what supports pleasurable experience with the artifact. This approach is pertinent to our case as we aim to addhance the dancing experience.

2.1 Fallman’s Triangle

Fallman’s model, a triangle, aims to integrate the theoretical reflections with design practice and explorations as well as to shape the interaction design research discipline and give researchers a useful tool to control and keep up with their project efforts, see Fig. 1.

Fig. 1.
figure 1

Interaction design research triangle [1]

Full size image

Design practice – this relates to the kind of activities that the interaction design re-searcher would undertake if he was practicing interaction design outside of the academia, such as producing commercially successful products. He joins the design team and engages in design practice as a designer but does so with a design question or inquiry in mind, which may or may not be related to the overall scope of the project.

Design explorations – has similarities with design practice in its synthetic and provocative approach to designing a new solution. However, it has a different perspective: Questioning “what if” takes precedence and the main intention of the de-sign is to provoke, criticize and to be proactive and societal in its expression.

Design studies – Other activities related to RtD involve the intellectual tradition within the discipline and the contribution that the researcher will bring in the ongoing discussions. Design studies are more focused on seeking general knowledge and to describe and understand the design process.

Fallman argues that interaction design research happens when design studies, explorations and practice are integrated into a project. To enable discussion on tensions and movements in the model describes the following terms: Trajectories - sought moves between two or more activity areas in the model, Loops - the ability to freely move between theory and practice, and Dimensions –subsets of the model and refer to discussions and tensions about the relevance of the activities presented in the extremities of the Fallman’s triangle.

3 Applying RtD to Dance

At the outset of our project, we were encouraged to explore the field of dance and technology freely. Below we will describe how we applied RtD in the context of a dance and technology project. Moreover, we will make use of Fallman’s framework to reflect on our project effort at a time.

3.1 Defining a Design Concept

Fallman states that projects can start in different extremities of the triangle. This is influenced by the perspective of the project and the tradition in which your research is rooted. Led by the choice of our methodology and the fact that we designed within a field of which we didn’t have deep knowledge, we started the project with design explorations, searching exemplars on how technology had been integrated in dance performances or described in the HCI literature.

The first search was broad and helped us get an overview of existing work within this specific context. Also, it helped us to identify new opportunities for design. Figure 2 shows some of the initial exemplars that caught our interest.

Fig. 2.
figure 2

Annotated exemplars

Full size image

We were fortunate enough to have Bill Gaver as a guest lecturer at our university and share his view on RtD. Annotated portfolios were presented as a way of communicating design research and generating new knowledge within HCI. [21] defines annotations as “textual accounts of artefacts, including any theoretical pronouncements about them…annotated portfolios may serve an even more valuable role as an alternative to more formalized theory in conceptual development and practical guidance for design”. Referring to Gaver’s annotated portfolio, we can argue that through searching for exemplars, we created a portfolio of artefacts designed within dance and technology. Annotating these exemplars helped us gain insights into the field, and these, in turn, guided our design process. Although this way of working was not really in tune with Gaver´s use of annotated portfolios, it was helpful, perhaps because the design space was an unfamiliar one and it provided some structure for discussing ideas and concepts.

We used Bowers’ [8] features of annotation as a frame to annotate our portfolio of exemplars. He highlights that portfolios can be annotated in different ways based on the purpose and interest. In the beginning, we started with a limited number of exemplars and the purpose of annotating was to explore which role the technology was playing in the dance performance.

After that, we went back to collect even more exemplars. In total, we gathered more than 60 exemplars (see Fig. 2) of technologies used for adding or enhancing a dance performance. In the second round, the purpose of annotations changed. At this point, they reflected how the technology was used in each case and its role in the dance performance.

We experienced this annotation process as creative, and helpful in defining, and later refining, our conceptual inquiry. For example, amongst our annotations, one of the categories, that we called “added senses”, stood out to us. Deciding to further explore this category, we discussed which senses could be involved in a dance performance, and whether they could add to, or enhance, the performance. The discussion about whether we could add or enhance lead us to create a new term, addhance, that harmoniously combines the two.

In parallel with collecting and annotating exemplars, we worked towards theoretical and conceptual explorations of the design space. Thus, once addhance comes into focus, this exploration entailed asking, why, how and whose (dancers, audience) senses could be affected during a performance. Of relevance was the “Eclipse” project [22] which presents an interactive dance costume that creates its own visual and sonic environment based on the wearer´s movements. This project inspired questions as: What if dancers could wear sound? Or, what if they generated sound? Would this addhance the performance? After several brainstorming sessions, we found the concept of making dancers identity audible to be of great interest, and it defined the direction for further inquiry.

To challenge the concept, we moved in Fallman’s triangle from design explorations toward design practice and prototyping. Three low fidelity prototypes were made. Experimenting with material and technological aspects of these prototypes, discussing them with others, we concluded that the concept of identity was too ambiguous and could create confusion. As we were not able to achieve conceptual maturation, we drifted back into design exploration.

We started looking into ways to addhance a performance using sound in a different, possibly disruptive way. We had numerous brainstorming sessions and reading hours to related concepts and theories we could find in the literature with the annotated exemplars and insights gained. The inquiry was always accompanied by the “what if” question. The intention was to bring something that would create a change or would boost a discussion within the dance and technology context and maybe lead a change in the society at large.

While investigating in the relationship between music and choreography, we found an interesting case which inspired and guided our conceptual thinking. It exhibits distinctive movement vocabularies: Errand into the Maze, a ballet with the music scare by Menotti. The first choreography was by M. Graham. The movement, framework and visual imagery of her choreography were created before Menotti’s music was added [15]. His score was then ‘shaped to fit the demands of Graham’s movement scenario’ [23]. The second choreographer, G. Bodenwieser, had a more traditional approach to choreography. She “laid the movement phrases on top of the sound, shaping them in time and style to the music” [23]. While Bodenwieser’s represent the common workflow of making a choreography we were fascinated by Graham’s experience. She was giving expression to movements by making the individual movements audible in a harmonious way. This led to our idea of turning the tables in regard to the way music is played during a dance performance. The article “Gesture ≈ Sound Experiment: Process and mapping” [10] was also important for our work. The paper describes the development of a system which supports kinaesthetic-auditory synchresis, where human body motion is mapped with sound in such a way that sound production becomes an inherent and unavoidable consequence of moving the body” [10]. We built on the concept of kinaesthetic-auditory synchresis presented in that paper. We wanted to give expressive sonic capabilities to the whole body in motion, with the aim of producing music.

Thus, we finalized our design concept: Producing music through body movements – Musicify movements to include music as a dimension of choreography, rather than the means for leading it. We define ‘producing music’ as bringing into existence, making or providing vocal or instrumental sounds combined in a harmonious way. Also, we define ‘movement’ as the act, process or result of moving or not moving/the absence of movement.

3.2 Musical Moves

Once we had a well-defined concept, we moved again into technological explorations and further design practice and prototyping. The first discussion was related to the kind of technology we could utilize to implement our concept. The questions raised were: what technological tools can be used for motion detection and what technological tools can be used to produce sound. In motion detection, the technologies discussed were wearables and optical motion sensors. While wearables may be more accurate in detecting motion, they can as well restrict the dancer movements [13]. Similarly, to [10] work, we seek to create performances which engage the whole body, and hence avoid dependence on hand-based sensor input or ‘interface artefacts’ which draw attention (of both performer and audience) away from the body towards the artefact. Hence, they chose the Kinect motion sensors to implement the kinaesthetic-auditory synchresis concept. Another case where the motion sensors have been implemented is in the SoundEffects project [24]. The author explores how dynamically adapting musical beat and rhythm can be used to stimulate and motivate physical activity in older adults. The prototype developed used an optical motion sensor (Microsoft Kinect), to detect motion and Max MSP, as a sound synthesizer [25]. We decided to apply the same technology, thus using Kinect as a motion sensor for detecting body movement and Max for producing music, as the two main pillars of our solution. However, we are aware that the Kinect system has weaknesses, and we have taken these into consideration when implementing our solution.

Exploring Mappings Between Sounds and Movements

We once more started prototyping to further explore and challenge our concept. Technological explorations went hand in hand with our attempt of creating a map between dance moves and sounds, which when performed by a dancer would produce harmonious music. Continuous explorations of possible mappings encouraged the iterative implementation of our prototype.

The journey of implementing our solution started with exploring possible mappings. The design practice and the prototyping were always influenced by the “what if” questions that we raised during our design explorations. Every improvement in the solution came after long hour hours of brainstorming in conceptual terms and then translating those discussions into the prototype.

To compile a first mapping, we relied on the dancing expertise of one of our team members, supplemented with numerous dance performance observations, body movement analyzations and music element relations. These explorations provided useful information for the technical solution. This helped in deciding on what data was relevant to read through Kinect for recognizing specific movements. Inspired by previous research projects that have used Kinect as part of their installations [10, 24] and as well testing it ourselves and with colleagues, we could build a better understanding of what data the Kinect could reliably read and modified our code to better recognize movements based on these data streams.

We mapped the set of movements previously mentioned to selected instruments and sound files diverse enough to produce multiple styles of music. After long hours of reviewing the literature, brainstorming and going through dance examples we concluded on a first mapping.

The initial tests and adjustments produced inharmonious sounds. The dancer had control of a digital orchestra, but the moves were not orchestrating harmonious and pleasant music. To address this, we iteratively looped between design practice and design explorations. We experimented with adjusting pitch, volume, type of instrument, sound file seed, or the ease of triggering sound, and tested the system ourselves. We paid attention to what degree the music was, in fact, harmonious, as well as the coherence of music and movements. Unfortunately, the results were unsatisfactory. Hence, we brought in two experts on music and technology. We presented our concept and the ongoing process, demonstrated the system, and gave them the opportunity to test themselves. The suggestions were to map the dance moves with notes produced by only one instrument. The piano became our instrument of choice. We were also advised to employ chords as they are, by definition, a harmonic set of pitches made up by multiple notes. Moreover, we were advised to utilized compatible chords to produce a harmonious music. Thus, considering these guidelines, we went back to refine our mapping.

We took a top-down approach in planning out the mapping. We now rooted our choices in music-, and body movement theory. The exploration phase helped us map a set of common dance movements, which we further integrated into a short choreography. The choreography was made up of a set of four different movements. We planned each of the movements in the choreography as a set of three body parts moving simultaneously. This, to be compatible with the music theory and the experts’ recommendations about utilizing chords as a way for achieving a harmonious music. Chords are typically made out of three or four notes. We focused on three notes chord.

Mapping technique: Each dance move was mapped to a chord. Thus, each body part movement of the dance move was mapped with a piano note. The aim was to achieve a system where one body movement would produce a single note, hence, a dance move would produce a chord, while multiple dance moves would produce music. We decided to utilize a “pop-music-heuristic” in striving for harmonious music production: “4 chord songs” [26]. As demonstrated by the band, Axis of Awesome, these 4 chords are all made up of the C-major scale (see Fig. 3, left) and can be combined rather freely to produce harmonious music. In our system, this would translate to freedom in combining dance moves, maintaining a harmonious musical outcome. The mapping process was accompanied with brainstorming and rooted in principles of mapping [27].

Fig. 3.
figure 3

Illustrates the thought process and the final mapping

Full size image

The theoretical and technological explorations led to the final vocabulary (see Fig. 3, right), where each dance move was mapped with a piano note, as presented in the picture. After implementing the mapping, we began testing the prototype ourselves. The results immediately seemed more harmonious to us. We then fine-tuned the parameters as well as the ease of triggering for each note, before moving on to testing different styles of dance with the system. We perceived a greater degree of movement-music style coherence with this implementation.

3.3 Testing

After several rounds of testing the prototype ourselves and making use of the dance experience of one of our team members, we had the opportunity to test it with professional dancers. No prior instructions were given to them and both girls were encouraged to freely explore the system and improvise choreographies. We produced a video scenario to show their experience with our systemFootnote 1. In Fig. 4 on the left is shown one of the dancers testing the system, with a mirroring of the computer screen in the corner and on the right is the MAX code running in the background, to produce harmonious music.

Fig. 4.
figure 4

Dancer testing the system (left) and what is happening in MAX (right)

Full size image

Their feedback was positive, pointing at the freedom that the system gave to the dancer and the enjoyment of learning how to compose harmonious music by exploring different combinations of moves. Testing with professional dancers gave us confidence in the concept of our design. Moreover, it was a lesson learned for further refining the technological solution.

4 Drifting in the Fallman’s Triangle

The process described above was conducted with the intention of learning how to apply Research through Design. We will, as mentioned, use Fallman’s triangle [1] as a framework to discuss our case. In this section, we will elaborate on how we moved and drifted within the different parts of the model as a mean on reflecting on our process.

Looping between design practice and design explorations - While in design explorations we worked toward conceptual maturation by formulating an inquiry and suggesting new design alternatives that would provoke discussion, in our design practice we focused on making our idea feasible and bring change through the design of an ultimate particular [1]. An example of this loop was the concept of: making identity audible. While design explorations led us to the concept of making identity audible, the design practice made us aware that there were weaknesses in the concept, consequently not leading in a conceptual maturity. This was the first loop that we went through during our four weeks of learning and applying RtD. The second loop happened once we had defined the idea of wanting to make music through body movement (Sect. 3.1) and move on into technological explorations (Sect. 3.2), leading to conceptual maturation. Technological explorations and further prototyping are other examples of these loops.

The activities in each extremity and the looping between design exploration and practice was orchestrated by design studies. Using the concept of trajectories, we argue that the theories and previous work in the design field and as well in dance and technology, influenced our choices during the project. [21] annotated portfolio led our exemplars exploration. Concept driven design influenced our work and helped us maintain focus in the formulation of a final concept, which would later be further implemented as a design artefact. Moreover, RtD principle on designing artefacts which enhance a critical thinking and boost discussion, is noticeable in our final solution.

We did not use the concept of dimension as described in [1], as the intention of this paper is to use the triangle as a framework for reflecting on our case and not discussing on the subsets of the model itself.

The reflection above shows our RtD process prompted in the Fallman’s model. We have included what kind of activities has happened in each extremity and as well the kind of movements that we took among these extremities. Furthermore, we can argue that critically reflecting on design practices and explorations can bring new knowledge in design studies. This is consistent with RtD. On the one hand the theories feed and trigger the design of new provocative solutions and on the other hand, the artefacts designed will serve as further explorations of desired changes in society. The artefact itself serves as a mean for furthering critical discussions both in design studies and in the new context for which that is designed. Similarly, our system intends to further discussion in the context of dance and technology, by questing the origin of the music in a dance performance. Positioning the artefact in dance schools can, for example, trigger the discussion on how music and dance are integrated in a choreography. Moreover, touching on the subject of annotated portfolio [21], putting our prototype in the same portfolio along with [10, 22, 24] etc. prototypes, could contribute to further knowledge in the context of dance and technology design as well as in novel interactions and forms of participation. We hope that this paper may serve for furthering knowledge on how to apply RtD in simple, time constrained innovative projects. We can argue that Fallman’s model is very helpful for the researcher to keep track of its design efforts and as Fallman states the interaction design research in possible only when the three extremities of the triangle model are integrated with each other in the project.

5 Contributions and Further Explorations

Sharing our design process through this paper our aim is to contribute in the field of RtD in two dimensions: knowledge about learning and applying RtD, and knowledge of applying RtD in a novel context. We hope that our approach will motivate others to explore how to apply RtD. We believe that the field might benefit from what [28] calls “the method stories”, in which designers share their stories and the challenges that they have faced in design projects. Moreover, our contribution is in applying RtD in a new and interesting context, that is, use of technology and dance as a means for contributing in culture. Music and dance are considered art and consequently an important part of cultural heritage. Musical Moves’ interactivity is important to its function, but the concept on which it was designed is more important, producing music through body movement. Although we initially designed the system with dancers as future users thinking that this experience would support them in dance classes, the tool is flexible. Thus, it can be applied in a different context and enhance a whole new pleasurable experience.

We were invited in an exhibit for children where we had the chance to present and test our new design as well as its flexibility to adapt to different contexts and user group. Some technical, system, modifications were thought to be necessary to be able to test the flexibility of the concept with a new user group. We modified the system to estimate the distance between the shoulders of users, and scale the minimum distance required to produce sound proportionally. The exhibit took place at Sentralen (Oslo), on the 13th and 14th of January, as a part of an initiative called CityKids (see Fig. 5).

Fig. 5.
figure 5

Moments from the CityKids experience

Full size image

We noticed that children had a playful experience while interacting with the system. We used observations and kept notes about all the children visiting our room. They seemed to be attracted by the music and the Kinect skeletons who mirrored them, on the screen. The element of silence seemed essential to the users perceiving their control of the music, which in turn seemingly contributed to them grasping the concept. We observed users stopping or starting movement once noticing that the music followed their lead. Similar cases occurred on multiple occasions, more frequently when the system was solely being used by one person. Another interesting aspect that we noticed was that although initially hesitant to test the system, once the children movements started producing music that increased the motivation to continue dancing. Moreover, it seemed difficult for children to understand the fact that the more rhythm they incorporated in their movements, the more pleasurable and harmonious the music would be.

Our experience with City Kids sparked the interest in exploring the adaptation of our system to design for a more ludic experience, where exploration, playfulness, reflectivity and openness become the leading design concepts [29]. Leaving the dancer to explore for himself and find meaning in this system, designed to musicalise moves, seems very interesting and intriguing. This demonstrates a strength of applying RtD, where the designs are often generative. One makes a generic prototype that can be tweaked to particular use situations. We aim to further expand our vocabulary of moves and music hope our research would inspire further research in the use of technology in dance.