2006: Mobile Music Technology: Report on an Emerging Community

Abstract

The new field of mobile music emerges at the intersection of ubiquitous computing, portable audio technology and NIME. We have held a series of international workshop on this topic with leading projects and speakers, in order to establish a community and stimulate the development of the field. In this report, we define mobile music, and map out the field by reporting on the workshop series and accounting for the state-of-the-art.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Mobile music is a new field concerned with musical interaction in mobile settings, using portable technology. It goes beyond today’s portable music players to include mobile music making, sharing and mixing. The core themes of NIME—interaction, interfaces and music, can today be deployed on mobile electronics. While NIME projects have mostly been concerned with stationary concert performance or installations, mobility allows NIME concepts to occupy exterior urban space, and exploit people’s movements through it, as well as the heterogeneous space and social dynamics found in those environments. The International Workshops on Mobile Music Technology are the first events that focus on this new field. They have played a key role in the development of mobile music since 2004 and can be regarded as one direction for expansion of the NIME community. This report establishes a definition of mobile music, describes the workshop series, and accounts for the state-of-the-art.

2 Mobile Music Technology

A number of recent technological advances have pushed the envelope of possible human-computer interactions, giving rise to new fields such as locative media and pervasive gaming. Mobile computing enables systems to be used anywhere and on the move. Coupled with context-aware computing and global positioning technology, mobile devices can respond to the user’s surroundings and location, situating interaction within everyday settings. Augmented and mixed-reality technologies merge the digital and physical realm, making them cohabit in the same environment. Ad hoc, peer-to-peer and distributed networking allow groups of users—ranging from co-located strangers to dislocated friends—to collaborate spontaneously, across distances and without the need for centralised supervision. Meanwhile, the miniaturisation of consumer electronics and improvements in high-capacity digital storage have given rise to powerful portable mp3 players that could easily contain one’s complete music collection. Moreover, mobile phones have brought us ubiquitous network connectivity. Mobile music emerges at the crossroads of these technological advances and their resulting new practices, joining the worlds of ubiquitous computing and locative arts, with mobile consumer electronics, and the sensing and interaction tradition of NIME.

2.1 Mobile Music: Beyond Portability

Mobile music as a term covers any musical activity using portable devices that are not tethered to a specific stationary locale; in particular those where the activity dynamically follows users and takes advantage of the mobile setting, thereby leveraging novel forms of musical experience. Mobile music devices might possess properties such as context awareness, ad hoc or distributed network connectivity, or location sensing, sometimes combined with technology embedded in the physical environment. Therefore, they can be used anywhere and on the move, and take advantage of people’s displacements, location, and of the changes of social and geographical context that mobility implies. Examples of mobile music activities include pushing music to people nearby (Jacobsson et al. 2005), sonifying local Wi-Fi coverage while riding a bike (McCallum 2005), or remixing music tracks with remote friends across peer-to-peer networks (Tanaka et al. 2005). Mobile music goes beyond the iconic Walkman™, and does not need to imply individual use, headphones or passive music listening. It spreads over a large spectrum of musical interactions, ranging from consumption to creation, and with mobility increasingly blurring this distinction (D’Arcangelo 2005). Mobile music resonates with practices of both NIME musicians and everyday users of consumer audio products.

2.2 Reconsidering Musical Interaction with Mobility

Mobile music creates a tension between music and place as well as new relationships between musician, listener, and music. For electronic musicians, the mobile environment offers more than just a new place to transplant NIME techniques. Rather, mobility encompasses specificities that encourage us to reconsider the basic tenets of musical interaction. Mobility implies outdoor environments where space and place become tangible parameters, and also implies always-on itinerant devices: location can become a “sensor” input to music systems, people nearby can become part of an ad hoc networked musical performance. With networked multi-user systems, mobility allows musical engagement beyond eye-to-eye contact. It also asks the NIME musician to consider social aspects in everyday public space, an environment not primarily dedicated to music use and where people might already be involved in a number of adjacent and simultaneous activities.

2.3 Another Dimension to Creative Engagement with Consumer Products

Many mobile music projects draw on earlier popular electronic music movements such as remix- and DJ-culture, file-sharing or playlists. They extend creative ways of engaging with portable consumer audio technology by weaving them into ever-changing geographical and social contexts. One example is tunA (Bassoli et al. 2004), where people in close proximity can share music by listening to each other’s mp3 playlists, getting a taste of people’s musical preferences across various social situations. There is a broad range of possibilities in terms of making music with mobile consumer devices, from ringtones, to mobile soundscape recording or sound art. Widespread platforms such as mobile phones are used as musical instruments and interfaces, encouraging the public to explore new ways of looking at their personal mobile devices. Projects working with such communication technologies invite musicians and lay people alike to participate in performances, group improvisation, sound art or remixing, for example collaborating with strangers in same physical space (e.g. on the bus), or jamming with remote friends while strolling around town.

3 International Workshops

At its early stage, mobile music was rapidly gaining popularity and relevance but lacked a clear sense community and an explicit demarcation as a field. As for any emerging field, it was therefore important to establish a community of people who could share experiences and communicate ideas. A good way to achieve such a goal is focused workshops. For instance, the NIME conference series grew out of a workshop at CHI 2001 (Poupyrev et al. 2001b). In 2004, we started a series of international workshops on the subject in order to establish and develop the field of mobile music. The workshops have gathered a mix of researchers, designers, musicians, new media artists, and representatives of the industry. They have raised awareness about existing projects as well as helped actors of the field with backgrounds in multiple disciplines to identify common goals and issues, share resources, and introduce one another to relevant technologies, methods and concepts. The purposes and formats of the workshops have varied as the community evolves but activities in common include presentations of projects, in-depth discussions, brainstorming sessions and hands-on activities.

The first International Workshop on Mobile Music Technology was organised at the Viktoria Institute in Göteborg, Sweden, in June 2004. The purpose was to gather a number of researchers with a shared interest in mobile music, and to attract additional people who might be interested in making the community grow. This workshop focused on presenting existing projects and defining the field. It had 15 external participants, plus organisers and student volunteers.

The second workshop was organised in May 2005 at NIME 2005 in Vancouver, Canada. This time, the community was better defined, and the workshop time was shared between presentations of new projects, in depth-discussions and hands-on brainstorming activities. It attracted 18 external participants.

The third edition of the workshop was a two-day event that took place in March 2006 at the University of Sussex, Brighton, UK. It gathered nearly 30 participants and focused on the locative media aspect of mobile music, with presentations by invited speakers, feedback sessions about work-in-progress projects, and hands-on activities with the latest mobile music technology.

3.1 Projects

The workshops feature state-of-the-art mobile music projects, in the form of presentations by guest speakers and peer-reviewed papers, posters and demonstration sessions (see Fig. 1a), as well as feedback sessions for works-in-progress. These projects are at the centre of the field’s development and demonstrate its diversity and potential. Many use generic mobile platforms such as mobile phones or handheld computers; others use hacked or custom-made technology to better respond to specific needs and requirements. All have in common taking advantage of the mobile nature of mobile technologies and situations as an intrinsic part of their work. Projects can be grouped along the following emerging themes.

Fig. 1

Mobile music projects: a Sonic City and b Malleable Mobile Music

Several projects explore collaborative music making with mobile technology. Malleable Mobile Music (Tanaka 2004) (see Fig. 1b) is a location-based and peer-to-peer networked remixing system. TGarden (Ryan and Salter 2003) is an interactive environment for theatrical music making using wearables. Sequencer404 (Jimison and Thatcher 2006) allows multi-user control of a musical sequencer through telephony and Voice over Internet Protocol (VoIP). In CELLPHONIA (Bull et al. 2006), people engage in a location-based mobile phone karaoke opera. The collaborative public art performance China Gates (Clay and Majoe 2006) synchronises a set of tuned gongs with GPS as participants follow different routes. Mobile phones are used for interacting with a sound installation in Intelligent Streets (Lörstad et al. 2004). Finally, IMPROVe (Widerberg and Hasan 2006) is a architecture for collaborative improvisation with sounds recorded with mobile devices.

Some of the projects in the genre of mobile music making enable individual users to manipulate sounds and create music by interacting with environmental factors: the physical urban environment with Sonic City (Gaye et al. 2003) (see Fig. 1a), and ambient lighting conditions in Solarcoustics (Barnard 2005). A mobile user-interface platform for such interactions in a personal area network (PAN) was also demonstrated (Yamauchi and Iwatake 2005).

Another theme is mobile music listening and sharing. Some projects address the sharing of playlists and music across peer-to-peer networks, enabling users to listen to their neighbours’ music either synchronously (SoundPryer (Östergren and Juhlin 2004) and tunA (Bassoli et al. 2004)) or asynchronously (Push!Music (Jacobsson et al. 2005)). Other projects transform music albums into narratives spread across geographical space (Location33 (Carter and Liu 2005)), or enable the cultivation of public “sound gardens” located in Wi-Fi connections, as an overlay on physical space (Tactical Sound Garden [TSG] (Shepard 2006)).

A third area is dedicated to HCI and mobile music. It includes SonicPulse (Anttila 2006)—a project providing an acoustic way of passively monitoring or actively exploring a shared music space, Music Mood Wheel (Andric and Xech 2006)—an auditive interface for navigating music spaces on the move, and Minimal Attention Navigation via Adapted Music (Hunt et al. 2006)—a musical navigation system for pedestrians.

Meanwhile, some workshop participants have taken a more sociological or media-studies approach, looking at the relation between music taste, use and identity (D’Arcangelo 2005), soundscapes and people’s everyday experience of place (Phillips 2006), mobile phones and its use in sound-art (Behrendt 2004), and mobility, sound and urban culture (Bull 2001). These contributions have brought insightful humanistic perspectives for the development of the field of mobile music, grounding it on social realities, aesthetics and already emerging practices.

3.2 Group Activities

The workshops included group activities with structured brainstorming sessions in the first two workshops, as well as feedback sessions and hands-on experience of mobile music technology in the third one. These were combined with in-depth discussions on various topics relevant to mobile music and on current issues, opportunities and challenges in the field—e.g. the relationship between mobile devices, space and the body in movement, or how to approach context-aware platforms developed in the field of ubiquitous computing with a NIME perspective.

3.2.1 Brainstorming Sessions in the 1st Workshop

An important function of the first workshop was to map out the field and define future directions. We organised a series of structured brainstorming activities that ran over two days. Participants were divided into three groups, each dedicated to one of the following topics: mobile music creation; mobile music sharing; business models and the future of the mobile music industry. In addition to this, we had pre-defined a number of themes to investigate: infrastructure and distribution; genre and formats; social implications; ownership; business models; creativity; interaction and expression; mobility; users and uses.

Each group would choose four themes from the list, and discussed them from the perspective of their overall topic. For instance, the group on Business models discussed the theme Genre and format, raising issues such as length of compositions, use of meta-tags or potential revenue from different kinds of formats. After the first day of brainstorming, results were presented to the other groups. The second day was dedicated to defining design dimensions for mobile music applications based on day one’s sessions (for example solo vs. collective, foreground vs. background), and to mapping the emerging design space to existing or future projects. The sessions raised a number of issues, including “in-between” states that are neither mobile nor stationary, how musical taste is used to establish personal identity, to the meaning of ownership and where added value could be elicited.

3.2.2 Bodystorming Session in 2nd Workshop

In the 2nd workshop, hands-on activities were kept to one afternoon. They focused on bodystorming of mobile music applications and scenarios. Bodystorming is a method where participants act out a particular scenario of use, taking the roles of e.g. users or artefacts and focusing on the interaction between them (Buchenau and Fulton Suri 2000). With this method, participants explored various mobile music themes, developed simple application ideas, and physically enacted scenarios of use in order to get an embodied understanding of design challenges and opportunities specific to mobile music.

Fig. 2

Mobile music workshops: a Project demonstration and b scenario body-storming

Participants first combined randomly chosen instances of the following categories: situations (e.g. driving a car while it snows); users (e.g. school kids); technological infrastructures (e.g. Wi-Fi, GPS); types of music uses (create, share, organise...). Combinations were assigned to each group and developed into 3 application or scenario ideas per group during short brainstorming sessions. Each group decided on one idea and further developed it through bodystorming. Scenarios were then acted out to the rest of the workshop to stimulate discussion. An example of scenario was a bicycle-taxi working as a peer-to-peer server and broadcasting its clients’ music on loudspeakers in Kingston, Jamaica (see Fig. 2b). This scenario generated discussions on mobile ways of sharing and outputting music in public space, and of their social adequacy.

3.2.3 Feedback Sessions and Hands-On Activities in the 3rd Workshop

On the first day of the third workshop, selected work-in-progress projects grouped in parallel sessions received expert feedback during critical and supporting discussions. Through this participants identified crucial issues and presented their findings to the other groups. The second day was hands-on and gave participants access to technologies for mobile music that they might otherwise not have been exposed to. Participants were given tutorials on sensors for mobile music, and on miniMIXA, a mobile music software mixer and mini recording studio for hand-held devices. They were also introduced to socialight, an audio space annotation platform for sharing location-based media. As a follow-up, participants sketched out possible applications combining such technologies.

3.3 Dissemination of Results

The output from the workshops has been presented in contexts outside of NIME and of the workshop itself. Two of the co-authors moderated a panel discussion at the ACM SIGGRAPH 2005 conference on the subject of Ubiquitous Music (Holmquist and Tanaka 2005) where the majority of the panellists selected by the conference had previously participated in the workshops. As the largest international conference on digital media and emerging technologies, the SIGGRAPH panel underscored the pertinence of mobility and musical interaction to a wider field. Authors have also given lectures about mobile music in art and design schools. Currently, the results from the first workshop are being edited into a book that will be a key reference emphasising the creative potential of mobile music technologies.

4 Conclusions and Future Work

We have presented the field of mobile music, its current state-of-the-art, as well as a workshop series with a decisive influence on its development. During the workshops, a multitude of emerging key topics concerning the socio-cultural, artistic, technological and economical aspects of mobile music have been identified. The overall experience of these events has been very positive. Out of the participants has crystallised a core group, which is very active in the field. The community continues to grow, with new people being attracted to each workshop and the number of relevant projects increasing consistently. The future of mobile music is now being shaped by a collective community effort and promises interesting future developments. In order to further extend and consolidate the mobile music community and support these developments, we will continue to organise new workshops and will soon publish a website as a resource for mobile music projects and related publications.

Appendices

Author Commentary: Mobile Music Technology: From Innovation to Ubiquitous Use

Frauke Behrendt

A time before smartphones becomes more difficult to imagine by the day, a time before digital, networked, sensor-studded personal mobile devices became ubiquitous. In such a time, more specifically in 2004, drawing on music’s rich history of mobility and responding to emerging developments and innovations in mobile technology, I was part of an interdisciplinary group of researchers and designers that came together to experiment with and analyse mobility and music in the context of increasingly ubiquitous networked devices, and we thus contributed to establishing the field of mobile music.

We organised five ‘International Mobile Music (Technology) Workshops’ between 2004 and 2008, in Gothenburg, Vancouver, Brighton, Amsterdam and Vienna. These are mentioned in the paper and documented in more detail in the book accompanying the final event (Kirisits et al. 2008). By 2008, mobile music had become mainstream and an integral part of several fields of research and practice, including NIME and app design culture.

Our 2006 paper on the emerging community around mobile music technology has been used and developed in a range of research areas, by researchers and designers from around the world, both within and beyond the NIME community, as becomes evident from reviewing the papers and patents citing the paper. The main contribution of the paper has been for those designing mobile music products or services, such as mobile phone apps, software or hardware, for example (Wang 2009). Almost equal interest has come from those designing or evaluating interactive and/or collaborative performances with mobile phones for performers and for audience participation (e.g. mobile phone orchestras or social music making platforms). Overviews, classifications and taxonomies of the field of mobile music from various perspectives have also drawn on the work, for example by considering the social, cultural and historic dimensions of mobile music (Gopinath and Stanyek 2014). The field of sonic interaction design (Rocchesso et al. 2008), the field of locative music and sound (e.g. GPS sound walks), the educational use of mobile music (e.g. mobile phone music learning for children), sound studies (e.g. ubiquitous listening) and media studies (e.g. global mobile media), are other areas where the paper has made a contribution.

My own research contributions on mobile music technology include developing a taxonomy of mobile music with four categories: musical instruments, sonified mobility, sound platforms and placed sound. These categories were explored in more detail through a number of detailed analysis of specific artworks and apps, drawing on empirical material gathered through interviews, observations, ethnographies and case studies. This research material was analysed in light of theories and concepts from media studies, mobility studies, NIME and sonic interaction design (Behrendt 2015). More recently, I have drawn on the field of mobile music as research partner on the NetPark project¹ that turned a public park into an ongoing and growing collection of mobile and locative artworks, many of which focus on sound and music. This presents a platform for ongoing research on both the design process and the audience perspective/user experience of the NetPark and the works hosted and curated in it. There is also a close relation between this most recent engagement with mobile music technology and my other research around mobile media, smart cities, the Internet of Things and sustainable mobility, in that all my work considers mobility and musical/sonic perspective, in part inspired by this NIME paper. Over time, my engagement with the mobile music technology community has shifted from a more technical perspective and an active engagement in the NIME community towards a more theoretical and empirical analysis of the social and cultural aspects of mobile music in the field of media studies. In the years since our early community and the series of workshops on mobile music technology, mobile music has become so ubiquitous that the topic is now well-established in a range of research and practice communities.

Expert Commentary: Mobile Music Making Paradigm: Towards a New Culture of Use

Koray Tahiroğlu

Although the primary focus in the first era of mobile music research was on the ways in which mobile devices raised unique opportunities in locative media, it is clear from a historical perspective that the actual goal was to establish mobile music making as a research field and to create scientific and artistic legitimacy around it. This was achieved by bringing together the NIME community’s early adopters to explore mobile technologies as new platforms for music making. The idea of organising workshops was successful in getting the attention of the musicians, designers, researchers and industry people who shared interests common with the fundamental NIME approach to music, interaction and technology. These workshops, where the projects, ideas and concepts of practitioners were introduced and shared, were the first steps taken towards establishing a platform for mobile music. It is important to remember that early workshops were organised before the first generation of smartphones. Regardless of the mobile technology specifications, the creative and interactive focus differentiated mobile music from existing forms of practices and presented possibilities that were clearly distinct from traditional interactive music systems.

The early workshops presented ideas for some aspects of future developments in the field, such as possible social-music experiences and interaction models that question the roles of artists and listeners in the creative process. However, the future of music practise using mobile platforms was explored only with general statements and its relation to the relative economic, cultural and mediated paradigms was barely considered. Many of the projects presented for new ways of creating music focused significantly on the design constraints of working with state-of-art technology. The music industry was undergoing a transitional period at the time, as it can be argued it still is, so the workshops could have explored the technological context of mobile music making from a wider perspective; consciously evaluating the economic, cultural and social factors in the way musicians have always had to. Perhaps proposing tentative hypotheses on the evolution of mobile music for a period of time when mobile devices have developed beyond portable-playback devices to smart systems, could have offered more insight into future directions.

Nevertheless, after the first era of mobile music research, the collective community continued in its efforts to reflect on and share the developments in the field of mobile music. For instance, the Designing Musical Interactions for Mobile Systems workshop was organised in order to discuss the specific interaction design challenges for deploying engaging and creative musical activities on mobile devices in the smartphone era (Tahiroğlu et al. 2012). Simultaneously, the explosion of commercial music apps has directed the industry and the research to the widespread potential of music on mobile devices. During the workshop these different categories of use were discussed in detail with a set of interface design models for music instruments, controllers, portable studios, game and ambient interactions, social/network components for creating rich musical interactions that push the capabilities of present day mobile phone technologies (Tahiroğlu et al. 2012).

Mobile music making holds a special place in social-interactive aspects of research in NIME community (Bryan-Kinns and Healey 2004; Yang and Essl 2015). Mobile phone technology supports mobile and casual music-playing, facilitating interactive performances (Wang 2009). Current smartphones are powerful, network connected and equipped with Audio I/O, touchscreens, cameras and other embedded sensor input mechanisms. The increasing processor power of mobile devices makes real time signal processing and sound synthesis possible, enabling advanced music composition and performance tasks to be carried out on a mobile device. Most importantly, mobile devices advance opportunities for interaction in a collaborative context and have created a culture in music practices that was unlikely foreseen. In order to envision new strategies for mobile music making that could allow mobile technology to expand the meaning of “mobile,” it is worth considering the ways the community has defined work methods, practices and criteria for musical expressivity (Tanaka et al. 2012).

It is important to be aware of the unrealised potential of mobile music making. More design work needs to be done in order to explore the full potential of mobile technologies, the different ways in which user interfaces can be manipulated and the gestural capabilities of the devices. Furthermore, it is critical to consider the paradigms of computer music, “real world” instruments and the listening experience within an appropriately broad view of musical interactions. This can be achieved by giving equal weight to the performer, audience, technologies and cultural forces when making them mobile.