1 Introduction

A growing corpus of studies investigates how stimulations of the sense of touch may be used to enhance experiences related to music, e.g., digital musical instrument playing [31], music listening [3, 24, 36], movie experience [33], and accessibility for impaired listeners [40]. Along the same lines, a broad range of audio-tactile systems for musical purposes has been proposed in the literature. Some of these systems are embedded in seats used during music listening (e.g., in chairs [4, 20, 36, 40]), others in wearables (e.g., suits [19], gloves [33]). Typically, such systems have an external computational unit (e.g., a laptop) and are not equipped with wireless connectivity. Today’s technological progresses provide designers of such haptic interfaces with the possibility of creating embedded systems at affordable costs that can wirelessly communicate with external devices both locally and with the Internet.

There has been increasing interest in haptic wearable devices targeting music performers [6, 23, 26, 38], which has led to the proposal of their categorization into a class of devices termed as musical haptic wearables for performers (MHWPs) [53]. These wearable devices can include haptic stimulation, gesture tracking, and wireless connectivity features. They were conceived to enhance communication between performers as well as between performers and audience members by leveraging the sense of touch. Such devices may be particularly useful in musical contexts where a visual or auditory communication is difficult or not feasible, but a tactile one can be delivered. An example of application is to improve accessibility to music making for the visually impaired. Another example is to support communication between performers and the sound engineer in scarce light conditions and/or long distance.

The MHWP concept was validated in various studies [6, 23, 26, 38, 53]. In the study reported in [53], the authors developed three chest-, foot-, and arm-worn MHWPs respectively for co-performer, performer-conductor, and performer-sound engineer interactions, which were assessed with twenty-five participants. Results showed that very high accuracies could be obtained for musical actions expected after instructions wirelessly communicated via tactile signals. Such results provide evidence that MHWPs can be an effective medium of communication in the context of electronic music performances. Along the same lines, a MHWP was proposed in [25] to assist a classical music conductor with a tactile representation of metronome clicks. Results suggested that such system may be used in lieu of auditory click tracks typically used by conductors for accurate tempo following (e.g., in film music).

The works reviewed above fall within the remits of the emerging field of musical haptics [42], which investigates the application of haptics research to the musical domain. To date, little research in musical haptics has been conducted to investigate the use of wireless haptic wearables to augment the audience’s experience at live music performances. By “augmentation” we imply here to go beyond the ordinary, for instance to provide access to new information or greater degrees of freedom. The audience’s experience during a performance may be augmented by harnessing additional sensory content or increasing agency in the reception process.

In this paper, we propose a vision for a class of wearable devices, the musical haptic wearables for audiences (MHWAs). MHWAs are characterized by embedded intelligence, wireless connectivity to local and remote networks, a system to deliver haptic stimuli, and tracking of gestures and physiological parameters. They are specifically devised for musical applications, with a primary use in the context of live music performance. The embedded intelligence enables the communication with other external devices, processes input data, and generates music-related haptic stimuli. The MHWAs vision described in this paper is based on the recent works reported in [54] and [56]. This article groups together these works and extends them with new content on design, implications, and limitations.

Both MHWPs and MHWAs are instances of a wider class of “Musical Things” within the “Internet of Musical Things (IoMusT)” paradigm [55]. The IoMusT is an emerging research field that relates to the network of objects and interfaces dedicated to the production, interaction, and reception of musical content. Musical Things embed electronics, sensors, data forwarding and processing software, and network connectivity enabling the collection and exchange of data for musical purpose.

The remainder of this article is organized as follows. Section 2 surveys works and technologies related to the envisioned MHWAs. Section 3 presents our design approach for MHWAs. Sections 4 and 5 describe two experiments aiming at preliminary validating our MHWAs concept. Section 6 provides a general discussion and a set of design considerations that arose from the results of the two experiments. Finally, section 7 provides concluding remarks and future perspectives.

2 Related work

This section introduces the research areas connected to the proposed MHWAs and describes their domain of application.

2.1 Augmented live performances

Several studies have investigated how to augment the audience experience in the context of live performing arts, such as theater [48], dance [18, 48], and music [22]. We present here a non-exhaustive list of criteria to classify previous approaches for augmented performances according to human-computer interaction and signal processing techniques: (multi)modality, using e.g., audition [41, 48], or vision (e.g., LED PAR lights [15], score visualisations [59], virtual objects [32, 48]); media (e.g., augmented reality [32, 48], screen-based interfaces [18, 59]); user interfaces (e.g., smartphones [13, 18, 32, 59], tangible user interfaces [47]); participation (e.g., audience creative agency can be increased with technologically-mediated participatory systems [59]); performer’s gesture augmentation (e.g., using inertial sensors or cameras [1, 18, 41]); techniques such as music information retrieval (e.g., real-time audio analysis [1]); or sound synthesis (e.g., [41, 48]).

2.2 Haptic devices to augment musical listening

Targeting seated listening situations, some authors have explored ways to embed actuators into chairs to provide vibrotactile stimuli in response to sounds. Merchel and Altinsoy developed a seat capable of delivering vertical whole-body vibrations and investigated the influence of such stimuli on the perceived quality of audio reproduction of DVD [36]. Their results showed that participants’ perceived quality of concert DVD reproduction could be improved using such type of vibrations. Nanayakkara et al. developed the Haptic Chair to enhance the listening experience for hearing-impaired people [40]. This device consists of a chair equipped with integrated speakers, which recreate audio-related vibrations mechanically. Results of a perceptual study showed that the system was effective in enhancing the experience of music for deaf participants. Along the same lines, Karam et al. developed Emoti-Chair, a chair-based sensory substitution system devised to improve music accessibility for deaf or hard-of-hearing people, which brings a high-resolution audio-tactile version of music to the body [27].

Armitage and Ng [2, 4] developed a vibrotactile interface consisting of a bi-dimensional array of sixteen actuators to be installed at the back of chairs. This device targeted audiences of live music performances, and was used for compositional practices. A similar work is that of Hayes who developed a chaise longue with six embedded vibration motors and two tactile transducers [20]. Such a device was used to explore the relationship between sound and physical sensation in compositions created for this purpose.

Seatings are not always provided at concerts and in many genres (e.g., pop, rock, electronic) the audience etiquette implies that concert-goers stand up and do not necessarily stay at fixed locations during the concert. For this type of settings, haptic wearables may be more suitable than seat-based solutions. Gunther and O’Modhrain designed a suit enhanced with twelve high-frequency and one low-frequency actuators capable of providing musically structured spatio-temporal patterns of vibration on the body surface [19]. Mazzoni and Bryan-Kinns developed Mood Glove, a prototype system consisting of a glove enhanced with actuators, conceived to amplify the mood of music in film via vibrotactile stimulation. The results of perceptual studies performed to validate this system suggested that vibrotactile stimuli have the potential to enhance emotional responses of audiences during a cinematic audiovisual experience.

2.3 Participatory live music performance

One of the motivations behind participatory art is to bridge the gap between audiences and artists by blurring the roles of creators and receivers [57]. Recent studies in interactive digital arts have proposed new technical and aesthetic principles enabling the creative participation of audiences in the production and reception of art works based on digital information and communication technology [22]. Wu et al. provide a review of such approaches in the case of participatory live music performance [59]. The proposed MHWAs provide potential to transform the communication flow and channels between performers and audiences at live performances. They could hence be used as a means to reach new forms of audience creative participation in the reception and/or production of artistic content.

2.4 Tactile music composition

Several authors have proposed to incorporate the haptic channel into composition practice. Following this trend, composers augment the listening experience of their music through tactile stimuli. The description of works and concerts conceived and held for “skin and ears” is reported in various studies (e.g., [3, 19, 20]). In their seminal work [19], Gunther and O’Modhrain coined the term “tactile composition” paving the way for the definition of a compositional language for the sense of touch. Various composers, installation artists, and researchers have dealt with the challenge of using the body surface as a compositional parameter by leveraging haptic devices for musical listening as those mentioned above.

3 Design approach for MHWAs

We defined the following design requirements:

DR1::

Easy to use wearable

DR2::

Unobtrusive and comfortable

DR3::

Able to enable tactile music composition

DR4::

Able to enable creative audience-performer interactions

DR5::

Able to track audience’s response during performances

We distinguish MHWAs that only act as receiver of control data and MHWAs that possess both receiving and data forwarding capabilities with sensors gauging audience members’ response and/or actions. We leave the sensing and data forwarding functionality as optional to allow for flexibility in the design of MHWAs.

3.1 MHWA architecture

Our proposed architecture for MHWAs is shown in Fig. 1. The material layer supports the MHW technology at the physical level. The design of the material layer requires to explore various materials (e.g., fabric, plastics, textiles), structures, as well as manufacturing technologies and methods. More specifically, the involved materials need to be chosen to optimize the efficient transmission of the haptic stimuli. Interaction with the skin can be direct or indirect e.g., through clothing.

Fig. 1
figure 1

Schematic view of the architecture and data communication channels of a musical haptic wearable for audiences (MHWA)

Full size image

The second key component is the hardware technology that needs to be embedded in the material layer. This component involves the design and integration of various technologies, namely a computational unit for data processing and synthesis of haptic signals; a wireless module for receiving and (optionally) transmitting data; if the used actuator is controlled by analog signals, a digital-to-analog converter (DAC) of the digital haptic signal (multichannel, if in presence of an independent control of multiple actuators), which may further need to be amplified by a dedicated amplifier; a haptic delivery system (including e.g., actuators for vibrotactile stimuli, heating/cooling devices for thermal stimuli, mechanical systems to deliver pressure, frictions or texture information to the skin); optionally, sensors capable to measure user-centric data such as that correlated to user motions (simple and/or complex), physiological responses (e.g., heart rate, electrodermal activity), physical environment characteristics (e.g., temperature); optionally, the H/W part of a user interface (e.g., for custom settings of the delivered haptic stimuli through tangible elements such as buttons), which could alternatively be leveraged by an external device (e.g., smartphone); a power supply system.

The third component of a MHWA consists of the software system, which must account for several tasks including real-time analysis and processing of sensor data (e.g., feature extraction), of incoming data from the wireless module, and processing of user interface data; the application of mapping strategies between values of the sensors or custom settings and parameters related to the haptic stimuli; the real-time synthesis and delivery of haptic stimuli; the delivery via a haptic player of the haptic signals received from the wireless module; the synchronization of the produced haptic stimuli with the sounds resulting from the musical performance; optionally, the real-time delivery of sensor data to other connected devices.

As shown in Fig. 2, in the proposed architecture design, the mapping between music-related information (characterizing gestures and/or resulting sounds) generated by performers and parameters of the haptic stimuli received by the audience is not performed by the MHWA, but by a smart device on the performer side (e.g., a smart musical instrument as defined in [52]), which forwards the control/signal data to the MHWA. This architecture design choice is motivated by the fact that control data for the haptic synthesizer or haptic signals for the haptic player are lighter in size than audio data and therefore more suitable for broadcast. Compared with haptic data and haptic signals, audio signals have a much higher sample rate (e.g., 2 KHz for haptics vs. 44.1 KHz for audio), bit depth (e.g., 8-bit for haptics vs. 16-bit for audio), packets stream rate, and packets size, which are variables that might also affect transmission latency and jitter. Additionally, this allows one to perform on the transmitting device the mappings (between audio and/or sensors data, of one or many performers, and the haptic content) only once for all the connected MHWAs. As the smart device performing the computational mapping process (which first requires the extraction of features from the musical and/or sensor content) does not have the constraint to be a wearable, a more powerful computing unit can be chosen than those typically embedded in a wearable.

Fig. 2
figure 2

Audio-haptic interactions between performers and audience members mediated by MusTPs and MHWAs

Full size image

3.2 Performer-audience interaction

The proposed MHWAs aim to enable novel types of interactions between performers and audience members. Figure 2 illustrates a schematic view of four types of audio-haptic interactions mediated by MHWAs and Musical Things for the performer (MusTPs): (a) one performer-to-one audience member; (b) one performer-to-many audience members; (c) many performers-to-one audience member; (d) many performers-to-many audience members.

Figure 2 (a) illustrates the audio and haptic signal communication chain mediated by a MusTP (performer) and MHWA (audience). The MusTP can be a smart musical instrument [52], or a conventional instrument coupled with a smart device such as a smartphone. In the latter case, the produced musical content not only reaches an audience member through sounds but is also sent to a smart device. The smart device extracts audio features from the musical content (or gathers control data for its generation, such as MIDI messages) and maps them to a set of parameters defining a haptic stimulus. This whole process can be fully accomplished by a smart musical instrument. Figures 2 (b), (c), and (d) use simplified representations of the content shown in greater detail in Fig. 2 (a). Haptified data are wirelessly sent to the connected MHWAs as a (multichannel) haptic signal for the haptic player or as control messages for the embedded haptic synthesizer. Audience members receiving this signal on their MHWAs can customize or alter the production of the haptic stimuli respectively via settings from the user interface and/or the data collected by the sensors. Reciprocally, sensor data from MHWAs can be streamed to the MusTP (equipped with a system for their display) for collaborative music interactions. Figure 2 (e) details the case where multiple performers are connected to a single MusTP that transmits the haptic signal(s)/haptic control data and receives sensors data from multiple audience members.

The next two sections describe two kinds of MHWAs and two concert-experiments aiming at a preliminary validation of our MHWAs concept. Both experiments focused on unidirectional tactile communication from performers towards the audience.

4 Experiment 1: haptification of performer’s control gestures

The first experiment aimed at investigating how the use of MHWAs influences the audience’s understanding of the instrumental control gestures of electronic music performers as well as the sense of connection between the audience and performers.

With conventional acoustic instruments, performer’s instrumental control gestures are tightly bound to the generated sound following physical principles with which audiences have a long cultural association. In contrast, digital musical instruments (DMIs) [39] pose particular challenges for the audience’s perception of performers’ musical actions. Indeed, with DMIs, many of the cues helping the audience to understand the gesture-sound relationships are lost due to the miniaturization of the control interfaces (e.g., a knob) and potentially complex mappings [14, 46].

We hypothesized that providing vibro-tactile stimuli related to performers’ control of the DMIs would help the audience to better understand musical expression and deepen the sense of presence of the performers.

4.1 Apparatus

Ten MHWA prototypes were created to provide tactile stimuli on both arms. Their hardware components (see Fig. 3) consisted of a small fanny pack; two elastic armbands; the Bela board for low-latency audio processing [35]; a Wi-Fi USB dongle compatible with the IEEE 802.11ac standard exploiting the 5-GHz band; four vibration motors (i.e., PWM-controlled eccentric rotating masses), two for each armband (these particular motors were chosen for their capability of providing a wide range of dynamics given a maximum vibration amplitude of 7g, and quick rise and decay time, 28 ms and 49 ms, respectively); a lightweight power supply. At software level, data processing and synthesis of the tactile stimuli were accomplished using Pure Data applications leveraging the Pulse Width Modulation technique. The same applications implemented data reception and forwarding through OSC messages over UDP. The motors embedded in each armband were connected to the Bela computing board using wires which were strapped to the participants’ clothes using small clips (see Fig. 4). The performers used two laptops and four MIDI controllers.

Fig. 3
figure 3

Prototype of musical haptic wearable for the audience (MHWA) used in the study

Full size image
Fig. 4
figure 4

Performers and audience during the concert-experiment

Full size image

All MHWAs and the laptops were connected using a router. The average latency and jitter of the local network (one way, not roundtrip) were 1.7 ms and 0.66 ms, respectively. Clock synchronization of the MHWAs and laptops over the wireless local network was achieved using the Ableton Link protocol. Each laptop ran four applications for live electronic music, which were developed using the Ableton Live digital audio workstation. These were composed by the performers who used different MIDI interfaces to control them (two drum pads and two keyboards). Each laptop also ran a Pure Data patch that mapped the MIDI messages controlling the Ableton Live applications into OSC messages wirelessly transmitted to all MHWAs. Some of the MIDI controller knobs, which generated MIDI control change messages, were mapped to the beat message of the Link protocol, to continuously control the tempo of the performed piece.

During the experiment, the performers played on a small stage with the audience standing in front of them (see Fig. 4). As we wanted the audience members to be able to relate to the control gestures from both performers through haptic feedback, the MHWAs were set up so that each armband corresponded to one performer. Table 1 describes the mappings utilized to associate the MIDI and Link messages to the synthesized tactile stimuli triggered by the MHWAs. The tactile stimuli were designed based on “tactile music composition” procedures [19] and by consulting the two performers to find a relevant mapping with their control gestures. A pilot test was conducted with two participants (who did not take part in the subsequent concert experiment) to test the validity of the tactile stimuli. The haptification was based on the following MIDI messages: MIDI Note on messages triggered when keyboard keys and drum pads were pressed, MIDI Control change messages occurring once knobs were turned, and MIDI pitch bend messages produced when the pitch bend wheels of the keyboards were used. MIDI program change messages were not included as they did not directly trigger or modify a sound (hence their effect could be confusing). After-touch actions on pads and keys (and therefore the associated after-touch MIDI messages) were not used by performers. During tempo changes, a beat change message was produced through a designated knob on the keyboards to synchronize the tactile pulse on the performed beat. Since the perception of the tactile pulse is affected by the rise time of the motor and the skin sensitivity, the vibration was triggered 60 ms before the beat occurred (this anticipation was empirically tuned during a pilot study with 4 participants). When a new action was performed on the controller before the end of the tactile stimulus associated to a previous action, the current haptic stimulus was interrupted so that the most recent action could be haptified.

Table 1 Mapping between messages and tactile stimuli
Full size table

4.2 Participants and procedure

Prior to the concert, the two performers were invited to prepare four pieces together, two with a fast tempo (130 BPM) and an exciting character, and two with a slow tempo (80 BPM) and a relaxing character.

The audience included 20 participants (8 females, 12 males, aged between 20 and 52, mean age = 32, SD = 7.5) which were divided into two groups of 10 members each. In each piece, one group used the MHWAs while the other did not (control group). Participants were instructed that they would experience the performance using wearables producing haptic stimulations but they were not told how they functioned in relation to the performers. To assess whether the system was intuitive and self-explanatory, no familiarization stage was conducted. Each group experienced the fast and slow pieces with and without the MHWAs; the order of the sessions was as follows: session 1—fast 1 (MHWA: group 1); session 2—slow 1 (MHWA: group 2); session 3—slow 2 (MHWA: group 1); session 4— fast 2 (MHWA: group 2).

Each session lasted 10 min. This design enabled to investigate the effects of the tempo and haptic wearable factors on the experience of the participants. After each music piece, the participants were invited to complete a questionnaire using computers in a lecture room located next to the performance venue. The questionnaire was identical for both the MHWA and control groups and was composed of the following questions to be evaluated on 7-point Likert scale: Arousal: “Please rate how calm or exciting you perceived the music to be.” [1=very calm, 7=very exciting]; Valence: “Please rate how negative or positive you perceived the music to be.” [1=very negative, 7=very positive]; Engagement: “Please rate your engagement level during the performance.” [1=not engaged at all, 7=very engaged]; Enjoyment: “I liked the performance.” [1=strongly disagree, 7=strongly agree]; Clarity: “The actions of the performers were clear to me.” [1=strongly disagree, 7=strongly agree]; Understanding: “It was easy to understand the musical expression of the performers.” [1=strongly disagree, 7=strongly agree].

After the performances, participants had to complete a post-questionnaire comprising two parts. The first part consisted of Likert items selected and adapted from the mutual engagement questionnaire described in [8]. The second part consisted in reflective feedback using the Likert items listed in Table 3 and others about the vibratory sensations.

4.3 Results

Table 2 shows the number of occurrences of the different MIDI/Link messages involved in each session, which are also the numbers of haptic stimuli following the mappings reported in Table 1. All participants received the same stimuli and no packet loss occurred in the wireless transmission (as verified on the analysis of log files).

Table 2 Number of occurrences of each MIDI/Link message for each session
Full size table

Figure 5 shows the results of the questionnaires provided at the end of each session for the MHWA and control groups. The participants’ answers to Likert items were not normally distributed and therefore were subjected to the Mann-Whitney-Wilcoxon non-parametric test to assess the effect of the Wearable between-subject factor. The analysis showed that in session 2 (slow 1) the perceived clarity of the performers’ actions and the understanding of the musical expression of the performers were significantly higher for the group wearing the MHWAs compared with the group not wearing them (respectively U = 95.5, p < 0.001 and U = 75.5, p < 0.05). All other comparisons were non significant.

Fig. 5
figure 5

Results of the questionnaire provided at the end of each session. Legend: ***p < 0.001, *p < 0.05

Full size image

Regarding the first part of the post-performance questionnaire on mutual engagement, 14 out of 20 participants deemed that the best performances were produced when using the MHWAs, 5 without using them, and 1 did not express a preference; 11 participants reported that they felt more satisfied with the performances when wearing the MHWAs, 5 without wearing them, and 4 did not express a preference; 11 participants reported that they enjoyed themselves the most with the MHWAs, 6 without and 3 did not express a preference; 13 participants reported that they felt most involved with the performers when wearing the MHWAs, 3 without wearing them, and 4 did not express a preference. These results show that the majority of participants preferred the performances attended when using the MHWAs.

Table 3 reports the results of the reflective questionnaire. With the MHWAs, audience members felt slightly more connected to the performers (M = 4.8, SD = 0.38) and more engaged with the music (M = 4.55, SD = 0.46). Participants tended to enjoy the vibrotactile feedback (M = 4.4, SD= 0.43) which was not found to be irritating (M = 2.95, SD= 0.35). They also expressed to be satisfied wearing armbands (M= 4.85, SD= 0.31). The additional questions related to the experience about the vibrotactile feedback showed that 15 participants deemed the strength of the vibrations appropriate, 4 too soft and 1 too strong; 7 participants reported that the vibrations occurred an appropriate number of times, 7 too rarely and 6 too frequently; 12 participants reported that each armband produced vibrations related to the actions of a single performer, 4 that each armband produced vibrations related to the actions of both performers, 4 did not had an opinion on this matter. Most of the participants understood that each armband related to a specific performer, and that their vibrations related to the actions of the performers. On average, the vibrations were appreciated by participants and were found appropriate. However, participants had different preferences for the frequency of the vibrations. This indicates that it could be favorable to let users personalize the vibration frequency in the MHWA interface.

Table 3 Questions and results (mean ±standard error) of the post-session questionnaire (evaluated on 7-point Likert items: 1 - strongly disagree, 2 - disagree, 3 - slightly disagree, 4 - neutral, 5 - slightly agree, 6 - agree, 7 - strongly agree)
Full size table

In the open comments, 9 participants commented that the vibrations should have been related to the music rather than to the performers’ actions (e.g., “There was not enough of a link between the music and the vibrations, they just related to the performers’ actions.”). In particular, 3 participants suggested to synchronize the vibrations to the beat or rhythmic patterns (e.g., “I think that vibrations would work best if more synced to the beat and the rhythm.”). Six participants also reported to be enthusiastic about the MHWAs.

4.4 Discussion

Results showed that the MHWAs and produced vibrotactile feedback did not significantly affect the emotional response, level of engagement and enjoyment of the audience. However, in one out of four sessions, session 2 (slow 1), the use of MHWAs significantly increased the clarity of the performers’ actions and the understanding of their musical expression.

The positive feedback expressed about the MHWAs after the sessions (Table 3) contrasts with the lack of significant effects in-between sessions. This may be related to the rather small number of participants in each group (N= 10) for the in-between Mann-Whitney-Wilcoxon tests which affects their power. Another aspect which may influence the results is that in the post-session questionnaire, participants’ answers are made considering both the slow and fast pieces, providing a more general assessment than for the in-between session questionnaires, which are made only for specific pieces.

As shown in Table 3, although participants tended to be able to relate the wearable vibrations to the actions of the performers (M = 4.3, SD= 0.39), using them did not enhance the experience of the music in a clear way (M = 4.15, SD= 0.41). Since computer music gestures do not convey sensations of effort in listeners [46], the effects of their haptic mapping is inherently problematic to evaluate. This may be due to a lack of cause-and-effect between the haptic and audio domains hampering the ability of listeners to connect the haptification of the gesture with the audible result. Indeed, with DMIs, the effects of control gestures can be highly non-linear and not necessarily synchronous to the sound production. Even if the mapping from gesture to haptic can be understood, failing to interpret the mapping from sound to haptic may limit or jeopardize the benefit of MHWAs. Further research is needed to design musical haptic stimuli driven by control gestures, ensuring that meaningful interpretations can be made both for the gesture-to-haptic and sound-to-haptic domains. This is supported by the analyses of the open comments highlighting the desire by some participants to experience haptic stimuli related to the produced music (e.g., its rhythm) rather than to the performers’ actions.

5 Experiment 2: smart mandolin and musical haptic gilet

Differently from the first experiment, which involved DMIs, the second experiment aimed at assessing the role of vibrotactile stimuli in affecting the perception of live music generated by a smartification of an acoustic instrument [51]. For this purpose, we designed a study where a smart mandolin performer played live for an audience wearing a gilet-based musical haptic wearable, which provided vibro-tactile sensations in response to the music. The vibrotactile stimuli were devised by a professional composer (the first author), according to tactile composition techniques [19]. The specific research questions we investigate are as follows: (i) to what extent do audience members appreciate live music with vibrations?; (ii) is there a consensus by the audience about the way in which the vibrations influence the live music experience?

5.1 Apparatus

The apparatus consisted of a smart mandolin, two haptic gilets, two laptops, and a wireless router.

Smart mandolin.:

The smart mandolin [50] (see Fig. 6) comprised a conventional acoustic mandolin enhanced with different types of sensors, a high-quality contact microphone, a loudspeaker, wireless connectivity to both local networks and the Internet, embedded battery, and the Bela low-latency audio processing board. The audio engine was coded in the Pure Data real-time audio processing environment and comprised a variety of ad hoc sound effects modulating the instrument’s string sounds, a library of sound samples to be triggered, as well as mapping strategies to control the sound production from the data gathered from the sensors as well as from the real-time extraction of features from the audio signal captured by the microphone.

For the experiment, the smart mandolin was configured with seven sensors: five pressure sensors, one ribbon sensor, and one distance sensor. The ribbon sensor was attached, thanks to its adhesive film, on top of the strip pressure sensor in order to create a device capable of providing simultaneous information about finger position and pressure. Such sensors were mapped to parameters of audio effects and sound samples triggers as described in Table 2. In addition, we extracted the note onset from the audio signal captured by the microphone, by leveraging the Pure Data object fiddle\(\sim \) [44].

Wireless data reception and forwarding were achieved leveraging the Wi-Fi protocol and the Open Sound Control (OSC) messages over the User Datagram Protocol.

Haptic gilets.:

The haptic gilets [58] are musical haptic wearables that distribute thirty ERM vibration motors over the wearer’s torso. Twelve motors are placed on the front of the torso and eighteen on the back. A schematic representation of the haptic gilet motors placement is illustrated in Fig. 7. Five driver boards are distributed on the garment which respond to OSC messages and generate PWM signals for six motors each. The driver boards connect to the Wi-Fi network using ESP8622 microcontrollers, specifically the ESP-12S modules. The PWM signals from the ESP-12S module are conditioned using the LM1930MC bidirectional motor driver integrated circuits. The involved motors (VPM2 from Solarbotics) were characterized by a maximum vibration amplitude of 1 G, and a rise and decay time of respectively 15 ms and 400 ms [16]. Power supply was accomplished by five 3.7 V lipo batteries, one for each board.

Laptops.:

A laptop controlled the gilet. A Max/MSP application was created, which received the OSC messages from the smart mandolin and mapped them into patterns of activations of the motors. The mappings are described in Table 2. A second laptop served the purpose of recording the OSC messages transmitted by the smart mandolin.

Router.:

The smart mandolin, the musical haptic gilets, and the laptops were connected to a local wireless network created by the router TP-Link TL-WR902AC, which was configured to support the IEEE 802.11.n Wi-Fi protocol over the 2.4 GHz bandwidth. The overall average latency between the smart mandolin and the musical haptic gilets was measured as 75 ms.

Fig. 6
figure 6

The smart mandolin with the indication of the sensors utilized during the experiments

Full size image
Fig. 7
figure 7

A diagram of the haptic gilet, in its front-left, back and front-right sides, and with the numbering of the 30 motors

Full size image

5.2 Stimuli

Two conditions were tested in the experiment: audio and audio-haptic. During the audio-haptic condition participants experienced the music with concurrent haptic stimuli displayed by the gilet. They consisted of patterns of activations of the vibrotactile motors that were inspired by the types of sounds that the smart mandolin could produce according to its configuration. They were devised with the goal of enriching the music experience. Specifically, the activation of the haptic patterns was associated to (i) performer-sensor interactions, and (ii) each note played when no sensor was concurrently active. Table 4 illustrates how each sensor and the extracted audio feature (i.e., the note onset) were mapped to both the electronically generated sounds and the haptic stimuli.

Table 4 Mappings between the OSC messages related to the smart mandolin sensors and extracted audio feature, the associated electronically-generated sounds, and the tactile stimuli delivered by the haptic gilets (for motors numbering see Fig. 7)
Full size table

5.3 Procedure

Twelve subjects (4 females, 8 males), aged between 21 and 43 (mean = 31.5, standard deviation = 6.52), took part in the experiment. All participants reported normal hearing. On average, they took 45 min to complete the experiment. The experiment was carried out in a quiet room, which provided ecologically valid conditions of a live concert in a small concert venue (see Fig. 8).

Fig. 8
figure 8

A picture taken during one of the experimental sessions showing, from the back to the front of the picture, the smart mandolin performer, the two participants wearing the haptic gilets, and the experimenter monitoring the data collection

Full size image

The experiment comprised six experimental sessions. Each session consisted of three trials in which the player played the smart mandolin for an audience of two participants. Each trial consisted of an extemporaneous improvisation on a theme. The involved themes were “O sole mio” by composer Di Capua, a Swedish folk song “Eklunda Polska #3,” and a theme composed for this work. The order of the themes was randomized across participants. The performer aimed to make the trials as similar as possible across participants (i.e., by using similar elements and adopting a similar playing style). Each trial lasted 6 min, during which the conditions with and without haptic feedback were automatically alternated every one minute by an application running on the first laptop. Therefore, in each trial, participants underwent for 3 min both the conditions with and without haptic stimuli. The experimenter indicated to the performer when to start and stop. The order of alternation was randomized across trials. Therefore, the performer did not know when the audience would have received the haptic stimuli so his performance could not be affected by this information.

Participants were asked to wear the haptic gilet described in Section 5.1 and to sit on two chairs at 1.5-m distance from the performer. They were told that during each trial the gilets might have provided some vibrations. They were not provided with any information concerning the purpose of the experiment and did not undergo any phase of familiarization with the technology. Participants were asked to respond to between-sessions questionnaires and a post-experimental questionnaire, as detailed in Sections 5.3.1 and 5.3.2.

5.3.1 Between-sessions questionnaire

At the end of each of the three trials participants were asked to evaluate on a visual analog scale (VAS) the following questions: Irritating. I found the vibrations irritating while listening to the music; Enjoyed. I enjoyed the music with the vibrations; Distracting. I found the vibrations distracting from the music; Engage. I found the vibrations helped me engage with the music; Vib-music. I understood a correspondence between the vibrations and the music; Vib-actions. I understood a correspondence between the vibrations and the performer’s actions; Enhanced. The vibrations enhanced my experience of the music.

5.3.2 Post-experiment questionnaire

At the end of the experiment, participants were asked to evaluate on a visual analog scale (VAS) the following questions: Preferred. “I preferred the performance with the vibrations compared to without”; Satisfied. “I was satisfied with wearing the gilet during the performance”; Helped. “The vibrations helped me to better understand the music”; Enjoyed. “I enjoyed myself the most when when I experienced the vibrations”; Engaged. “I felt more engaged with the music when I experienced the vibrations”; Connected. “I felt more connected to the performer when I experienced the vibrations”; Enriched. The vibrations enriched my experience to listening to the music; Arousal. Please rate how calm or exciting you perceived the music to be with the vibrations; Valence. Please rate how negative or positive you perceived the music to be with the vibrations.

In addition, we asked participants to answer to the following three questions: How would you describe the experience with the vibrations compared to without?; Did you prefer the experience more with or without the vibrations? Why?; What would you change about the vibrations or the vest to improve the experience, if anything? Finally, participants were given the possibility to leave an open comment.

5.4 Results

5.4.1 Results of the between-sessions evaluations

Fig. 9 (left) shows the results for the evaluations of all participants in terms of mean and standard error. Note that these aggregated scores hide the presence of different subgroups within the participants. An in-depth analysis at the subject level revealed that there were two groups, those more positive towards the vibrations (7 subjects), and those more negative towards them (5 subjects). In the reminder of the paper we refer to those groups as “positive group” and “negative group.” The mean and standard error of the evaluations of the two groups is shown in Fig. 9 (right). An analysis conducted using the Mann-Whitney-Wilcoxon test showed that the positive group evaluated the level of irritation caused by the vibrations as significantly lower compared to the negative group (U = 297, p < 0.001); the evaluations of the level of enjoyment caused by the vibrations was significantly higher for the positive group compared to the negative group (U = 61, p < 0.01); along the same lines, the positive groups rated the enhancement of the music experience caused by the vibrations as significantly higher than that reported by the negative group (U = 80.5, p < 0.05).

Fig. 9
figure 9

Mean and standard error of the between-session evaluations for all subjects (left) and for the two identified groups (right). Legend: *p < 0.05, **p < 0.01, ***p < 0.001

Full size image

Furthermore, to assess the effect of stimuli repetitions across time, we checked for differences between the trials considering all subjects. The mean and standard error of the participants’ evaluations after each trial are illustrated in Fig. 10. A statistical analysis conducted between the ratings of the first and last trials for each questionnaire item, using the Mann-Whitney-Wilcoxon test, showed that the level of engagement of participants induced by the vibrations was significantly higher for the last trial compared to the first (U = 31.5, p < 0.05). All other comparisons were not significant. A tendency towards significance was found for the level of distraction caused by the vibrations (U = 31.5, p = 0.08), which was higher for the first trial compared with the last.

Fig. 10
figure 10

Mean and standard error for the evaluations after each trial, for all subjects. Legend: *p < 0.05

Full size image

5.4.2 Results of the post-experiment evaluations

Results for the post-experimental questionnaire for all subjects are illustrated in Fig. 11 (left). Again the mean from all participants blurs the evidence: an in-depth analysis at subject level revealed that the same subjects identified as belonging to the positive and negative groups in the between-sessions evaluations could be also grouped for the post-experimental evaluations. Results for the evaluations of the two groups of subjects are shown in Fig. 11 (right). Using the Mann-Whitney-Wilcoxon test, significantly greater evaluations of the positive group compared with the negative one were found for preference for, enjoyment of, and engagement with the music with the vibrations (respectively U = 0.5, p < 0.01; U = 6, p < 0.001; and U = 4, p < 0.05). Moreover, the positive group rated that the vibrations enriched the music experience with significantly greater evaluations compared with the negative group (U = 0.5, p < 0.01). Furthermore, the positive group rated the valence as significantly higher than the negative one (U = 3, p < 0.05).

Fig. 11
figure 11

Mean and standard error of the post-experiment evaluations for all subjects (left) and for the two identified groups (right). Legend: *p < 0.05, **p < 0.01

Full size image

5.4.3 Thematic analysis

We analyzed participants’ answers to the open-ended questions using an inductive thematic analysis [7]. The analysis was conducted by generating codes, which were further organized into themes that reflected patterns, as described below.

Attention

According to two participants the vibrations enhanced the attention to the music as they stimulated them to search for the relationship between what was heard with what was felt (e.g., “The vibrations made me pay attention to the music to try to match it with the vibrations” or “Vibrations added a new level and I found myself searching for relationships between music and vibrations”). Conversely, two participants reported that in some cases the vibrations distracted them from the music. This happened for instance in presence of uncomfortable sensations caused by the vibrations in certain locations (e.g., “It was too distracting from the music when vibrating below the stomach as it was an unpleasant sensation”) or when a connection between the music and the vibrations was searched and not found (e.g., “I could not make a connection between vibrations and music, so it was mentally distracting”).

Music-vibration connection

Six participants reported that the vibrations did not fully correspond to the music heard. While in some cases they were able to find a clear connection, in other cases they did not perceive coherence between what they were hearing and what they were feeling. In general, this was reported to have affected negatively the audio-haptic experience (e.g., “I preferred the experience without vibrations because I didn’t see much correspondence between music and vibrations” or “I preferred when the vibrations corresponded strongly with an effect or to the strumming. If there wasn’t a clear connection between music and vibration it was distracting”). Two of these participants commented to have enjoyed the experience the most when a correspondence was clear to them (e.g., “I didn’t understand the correlation between the music and the vibrations for most part, but when I did perceive them to go well together I enjoyed it”). All of them suggested to design the vibrations in such a way to have a more intuitive audio-haptic correspondence (e.g., “The relation between the sound and haptic feedback needs to be more understandable”, “I’d make the music correspond to the vibrations more precisely” or “Ideally, the vibrations should be modulated by the intensity of the music, which I felt was not the case always”). One participant commented to have perceived a latency between the music and the vibrations (e.g., “Sometimes it felt like if the haptic feedback was not at the same time as the music, like if there was a delay”).

Adaptation time

Two of the participants commented that the first times they experienced the vibrations were not pleasant and a period of adaptation was necessary to them to get used to the vibrations and enjoy the experience (e.g., “The stimuli were at first distracting but then I slowly got used to them. I mostly enjoyed the experience of the vibrations towards the end” or “At first I preferred the musical experience without the vibrations but then I liked it because of the challenge to make a connection with the sound”).

Arousal

Two participants reported that the vibrations induced them to feel more excited (e.g., “With the vibrations the experience is more exciting”). One of these participants also reported that vibrations were enhancing the arousal of the more exciting musical parts (“When there was a very exciting passage the vibrations enhanced the music, I even felt the need to dance”). Conversely, one participant reported that vibrations induced a state of relaxation, especially when the music had a calm mood (“The vibration helped me to relax while the music was more calm”).

Richer experience

Five participants reported to have enjoyed the experience of the vibrations as they led to a novel, interesting, or richer experience (e.g., “I liked a lot the experience with the jacket and the vibrations”, “The experience of the music with the vibrations is more engaging. It creates a sense of being more involved”, or “I prefer the experience with the vibrations. My experience was more intimate, as if someone was interacting with me”).

Unpleasantness

Three participants deemed parts of the haptic experience unpleasant. This was due to the fact that in some cases the vibrations were perceived as uncomfortable since they stimulated parts of the body where participants were more sensitive. In particular, two of those participants suggested to not use the vibrations in the region of the stomach (e.g., “Avoid the vibrations on the whole part of the abdomen, they are sometimes painful if you are a woman” or “Sometimes the feeling is uncomfortable. Don’t provide vibrations in the region below the stomach”).

5.5 Discussion

The results of both the between-session and post-experiment evaluations consistently revealed the presence of two groups within participants, which could be categorized on the basis of their positive or negative appreciation of the provided vibrations. The thematic analysis carried out on the open-ended questions revealed various causes that led the participants of the negative group to generally prefer the experience of the music without the vibrations, as well as some of the participants of the positive group to rate with not very high values the evaluation scales assessing the vibrations appreciation. Some of those participants addressed the lack of appreciation of the haptic experience to the sensation of unpleasantness in some parts of the body where they were particularly sensitive, while others to the lack of full comprehension of the relation between the music and the vibrations (revealing to have appreciated mostly the audio-haptic experience when such a relation could be found). These aspects were reported to have distracted participants from the music.

On the other hand, seven participants deemed that the vibrations actually enriched the live music experience (their average evaluation on such a scale was 7.2 out of 10) and reported to have enjoyed the experience of the music in presence of the vibrations. The causes that led to such enjoyment were different as participants addressed the enhancement of the music experience to the capability of the vibrations to induce either a state of excitement or of relaxation. Similar considerations on the affective interactions between the two compositional media were provided by the participants of the Gunther and O’Modhrain’s study [19]. The increased excitement experienced by some of the participants in presence of the vibrations parallels the findings of Mazzoni and Brian-Kinns on the influence of vibrations on arousal in mood music of movies [33]. Interestingly, some participants reported that the provided vibration patterns spurred them to a higher level of attention to the music in order to find the relationship between what was listened and what was felt.

However, even for the positive group, the vibrations were not effective in drastically enhancing the music listening experience of participants. This result is only in part in line with the findings available in the literature of musical haptics involving recorded music. The studies reported by Merchel and Altinsoy [37] and those by McDowell and Furlong [34] showed that vibrations were generally effective in improving the listeners’ music experience. Nevertheless, those studies involved recorded music, which is devoid of the vibrations that can be naturally perceived by the body during a live music concert. The vibrations provided in those experiments aimed to recreate the haptic sensations that could be experienced during a live music setting, which showed to have a positive effect in the perceived quality of the music heard.

On the other hand, those studies used wearables very different than the one used here. Such devices were based on vibration speakers and the haptic stimuli were tightly synchronized with the music participants were listening to. In our experiment, participants experienced vibrations that superimposed onto the ones already perceived by the body through the live performance setting. Because of a high latency in the wireless transmission and generation of the vibrations (75 ms), the haptic stimuli could not be delivered in perfectly synchronous way with the musical stimuli. The timings in the patterns of vibration were designed to be temporally coherent with the music played. For instance, the delay time of the delay effects was coherent with the temporal distance between the vibrations (e.g., the half, the same, or the double of the delay time, see Table 4). However, due to the latency, those patterns of vibration did not happen at the very same time of the sound repetitions of the delay feedback. It is plausible that such delay between the heard music and the experienced sensations had an effect on participants’ perceptions. The comments of one of the participants about the perceived latency supports this hypothesis.

Notably, some participants reported the need of some time to get used to the vibrations and, as a consequence of this, enjoy the experience. The results of the analysis of the participants’ evaluations after each trial support these comments. From Fig. 10, a general trend emerges where the evaluations about the irritation and distraction caused by the vibrations decreased from the first to the last trial, while the evaluations about the level of the engagement induced by the vibrations significantly increased from the first to the last trial. It is also worth noticing that the results of the questions on engagement and enhancement/enrichment of the post-experimental questionnaire were on average slightly greater than the corresponding ones of the between-session questionnaire (see Fig. 9 and Fig. 11). These results are in agreement with the findings of Gunther and O’Modhrain [19], who used a wearable device with spatially distributed vibration speakers (although involving pre-recorded music delivered via headphones). Some participants of their study commented that at first it was difficult for them to make sense of the perceived vibrations, but that their ability to understand and appreciate the played tactile compositions improved with the time.

The study reported here mostly focused on artistic and expressive applications of haptics technology, which is in line with the endeavors of composers adopting tactile composition techniques to augment the audience’s music experience (see e.g., [3, 19, 20]). The vibrations were designed according to the composer’s aesthetic choices, which were however aiming to create at haptic level a coherent representation of the played music. Notably, aesthetics is a topic that has been relatively overlooked in haptic design research [5, 10, 11, 20, 43] and has recently been encouraged by haptic designers [21]. This study also aimed to contribute towards a discussion on aesthetics in musical haptic practice.

6 General discussion

The results of the two experiments reported in Sections 4 and 5 provide a preliminary assessment of our MHWAs concept. The first experiment targeted a scenario of electronic music performance and the investigation on the role of the sense of touch in helping the audience to understand the causality between performers’ actions on DMIs and the generated sounds. The second experiment investigated the role of vibrotactile stimuli in affecting the listeners’ experience of a solo smart mandolin performance. Nevertheless, it is worth noticing that our studies present limitations due to the small sample size. More experiments involving a larger number of participants are needed to validate more extensively the proposed MHWA vision.

MHWAs may find application in several other scenarios. We believe that haptic stimulations appropriately designed to create meaningful perceptual effects in relation to concurrent musical content have the potential to enrich audience’s experience and engagement in live music performance. Examples of use cases include the enhancement of rhythm perception at tactile level using stimuli synchronized with the music beat of the musical content; the rendering at tactile level of the vibrato of a violin, which could allow the audience to “feel” the player’s musical expression and and perhaps feel more connected to him/her or the music; thanks to the settings interface an audience member could tune the MHWA to a particular instrumental section of a symphonic orchestra, e.g., to feel more the brass than the strings, and could switch between the sections or to the whole orchestra to have a partial or global haptic representation of the produced musical content.

Differently from most of previous haptic systems conceived for musical purposes (e.g., [4, 19, 20, 33, 36, 40]), MHWAs have the capability of wirelessly exchanging information with external devices as well as being synchronized with them, which opens for the possibility of creating a new set of applications. For instance, MHWAs can be envisioned as interfaces for novel audience creative participation exploiting web and sensing technologies. Screen-based interactions for participatory live music performances (e.g., using smartphones [32, 59]) may distract from the performance itself while MHWAs could be used seamlessly as wearables: for instance by exploiting free use of body movements for conscious interaction (e.g., triggering of a notes pattern associated to a simple gesture), or less consciously in response to monitored physiological data. Moreover, MHWAs can be used concurrently with other technologies designed to augment the performance at visual level (e.g., apps on mobile phones [32] or stage screens [9]) as they rely on a different sensory channel.

MHWAs have also important implications for both visually and hearing-impaired persons as they may provide a better appreciation of music during live performance. Indeed, (non-wearable) haptic systems have been proved successful in enhancing the musical experience of deaf persons for recorded music [40]. In addition, despite the fact that MHWAs are devised for the live music context, they could also be used in other performing arts (e.g., theatre, dance, opera) and non-live contexts (e.g., music listening). Notably, MHWAs represent an area of research that should be addressed by interdisciplinary collaborations between artists, scientists and engineers. Related disciplines includes performing arts, music composition, Internet of Things, perception, computer science, sensor, and haptic technologies. Besides this specific challenge, we have identified three categories of aspects that challenge our MHWA concept: technological, perceptual, and artistic.

The main technological challenge to our MHWA concept lies in the tight synchronization between the musical content generated by performers and the haptic stimuli received by audience members. Such a challenge encompasses various aspects, which parallel the issues of technologies for networked music performance [45], in particular over wireless local area networks [17]. The most relevant is the latency in the overall chain from the generation of the musical instrument sonic output to the stimulation of the skin (see Figs. 1 and 2). The sounds produced by performers must be first digitized via a soundcard in case the instrument is not digital or smart [52]. Subsequently, the acquired audio content has to be mapped to the desired haptic representation (including the use of audio feature extraction techniques that might be required), and for this purpose time-efficient methods are needed. Then haptic data have to be transmitted over the wireless network. To this end, novel methods for ultra-low latency transmission are needed [55] along with novel dedicated protocols and standards conceived and developed to account for the properties of the haptic content. Finally, the haptic stimuli have to be efficiently processed inside the MHWA and then delivered in the shortest amount of time. Therefore, it is paramount to develop efficient methods capable of minimizing delays occurring in each of these steps. Synchronization mechanisms (e.g., sharing of a clock) are also relevant to this end [45].

Another technological and perceptual challenge consists of creating devices that must be as silent as possible in order to not affect negatively the auditory appreciation of live music performances. Typically, devices involving actuators for vibrotaction generate noises or low-quality sounds that are unwanted considering the goal of enriching the experience of the concurrent musical content. The design of a MHWA and its haptic stimuli should be informed by haptic perception and psychophysics research [29]. This includes a deep knowledge of the gamut of haptic parameters and their ranges of variation [19], tactile illusions [30], audio-tactile interactions [28], affective haptics [12], and crossmodal correspondences [49].

MHWAs provide composers with the possibility to use the sense of touch as a compositional parameter. Composers interested in haptics as a creative medium should take into account haptic perceptual effects, especially in relation to auditory perception and under the light of emotional communication [12]. However, to date, tactile composition is an area that is largely unexplored. Still a few composers have composed for the sense of touch with systems conceived specifically for this purpose. Only little research has been conducted on the definition of a haptics language for compositional practice [19] and on related aesthetic considerations [21]. Such a compositional practice would deserve more investigation, particularly in light of recent advances in haptic perception research and of the possibility of creating more easily and at an affordable cost haptic devices for musical applications such as MHWAs.

6.1 Design considerations

Based on the results of both experiments, we delineate the following design considerations that may benefit designers of musical haptic wearables focusing on enriching the musical experience of audiences of live music.

Co-design

The issues of lack of comprehension of the connection between the music and the vibrations call for a better design of the haptic stimuli in relation to the music. One possible strategy to cope with this issue is that of involving audience members into the design process.

Personalization

To avoid unpleasant sensations that some people may experience in certain parts of the body, it is important to empower the audience members with the possibility of personalize their musical haptic wearable. Such personalizations may account for the selection of which parts of the body one wants to experience the vibrations on (this might imply the deactivation of certain motors impacting certain regions of the body), the regulation of the maximum amplitude of the vibrations, or the choice of specific vibrotactile patterns among a set.

Latency reduction

The synchronization between the music delivered by a musical instrument and the related vibration delivered by a musical haptic wearable seems to play a relevant role in the audio-haptic experience. Therefore, it is crucial to minimize the latency between the two media. This may be achieved by leveraging wireless communication protocols faster than the one used in the present experiment. Latency may also be reduced by involving vibration speakers, which have a minimal rise time (differently from PWM controllable motors such as eccentric rotating masses).

Familiarization phase

When designing a live music performance, it is important to reserve some time before its beginning to make the audience members experience the vibrations. Especially for some participants, a certain time for adapting to the sensations caused by the vibrations is needed in order to understand and appreciate the played tactile compositions.

7 Conclusions

In this paper, we proposed the musical haptic wearables for audiences, defined as untethered body worn devices with embedded intelligence, wireless connectivity, sensing, and haptic delivery features. MHWAs are conceived to enrich the audience’s experience of live music performances by providing musically related haptic stimuli, as well as open possibilities for active participation in the music creation process. The novelty of the paper is represented by three main contributions: the presentation of two particular implemented designs for MHWAs, the presentation of two systematics evaluation with audiences, and design considerations drawn from both such evaluations.

The paper presented two studies focusing on two musical genres (electronic music and electro-acoustic music) which involved two kinds of MHWAs prototypes (respectively based on an armband and a gilet). These studies were conducted in the form of a concert-experiment with live music, and provided a validation of the feasibility and effectivity of our vision. The paper also provided a discussion of several technical, perceptual, and artistic challenges that remain to be addressed, and proposed a set of design considerations that emerged from the results of both studies.

In future work, we plan to develop other devices that implement our vision for MHWAs, apply them in live music performance contexts (both leveraging local and remote networks, as well as considering audience’s participation to the creative process), and assess their functional and artistic validity. Finally, it is hoped that the content of this paper could stimulate discussions within the musical haptics community, both at technical and artistic level.