Learning Tonal Harmony Through Augmented Reality: Bridging the Gap Between Music Embodiment and Digital Experiences

Abstract

This paper aims to highlight how a technologically augmented approach can help developing tonal harmony awareness in young learners. The proposal is rooted in previous experiences conducted by the same research group and dealing, on one side, with the embodiment of music concepts, and, on the other, with a reenactment of the same activities in the digital domain. To the latter goal, a publicly-available web platform containing three types of exercises has been released. In order to bridge the gap between the physical experiences and their digital counterparts, we present a methodological approach based on the adoption of augmented-reality technologies. Such a vision has driven the design and implementation of an app for mobile devices that incorporates image recognition algorithms.

1 Introduction

Making young students acquire musical competences is not a trivial task. It involves motivation, engagement, experience sharing and the acquisition of motor and cognitive skills.

During the first part of the 20^th Century, a vast movement of innovative theories and methods has developed in the field of music education. Among these, it is important to mention Carl Orff’s Schulwerk (1950–1954), and particularly its American edition [17], where multiple movement games and bodily activities are proposed with the aim of acquiring motor skills and providing a better understanding of the musical form. A similar approach was followed by the Swiss musician Émile Jaques-Dalcroze, who developed a method for training musical thought and expressiveness through coordinated movements and dance [34]. All these fundamental contributions have deeply influenced the way many instructors teach music, making games – and particularly movement-based ones – a well established practice. With these premises, the advent of personal digital devices and video games did not find music teachers completely unprepared. Nevertheless, much can be done to further encourage the use of digital games in order to increase the interest and the involvement of children in everyday classroom activities.

Many video games, and many music-oriented educational applications using game elements as well, are nowadays freely available on the web. Let us mention, e.g., the Chrome Music Lab suite,¹ which offers meaningful musical experiences using eye-catching graphics and intuitive responsiveness to user’s interaction.

If, on the one side, the use of digital games can greatly enrich music teaching, on the other side a fundamental element is lacking: movement. Actually, there are systems and applications that are able to couple playful elements and movement. The WizeFloor system provides a rich choice of movement games through the use of motion tracking technologies and interactive floor projections.² The SMALLab system, a room-sized interactive environment where students are up out of their seats, moving as they learn, employs similar technologies, but integrates the learning platform with a web tool for the individual prosecution of learning activities.³

The mentioned solutions require to set up specific environments and present a non-trivial cost for their implementation in a formal learning space, such as a school. Despite the potential efficacy, their diffusion and usability is greatly biased by these limitations. Thus, every effort to combine the expressiveness of gestural or full-body interaction with the richness and versatility of the digital domain must be regarded with great interest. This is exactly the aim of AREmbody, a mobile application which extends the functionalities of Harmonic Touch, a web platform for the knowledge and practice of tonal harmony.

The present paper is the extension of a previous work focusing on the use of augmented reality on mobile devices to develop harmony awareness. That contribution was presented at the 4$^{th}$ International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020) [3]. In this article, we are going to generalize such an approach to different harmony-related experiences.

The rest of the paper is organized as follows: Sect. 2 explains the basic musical theory behind our proposal, Sect. 3 recalls the milestones in our background activities, Sect. 4 focuses on open research questions, Sect. 5 describes a technological platform to answer these questions, Sect. 6 presents a mobile-phone app that implements the technological solution, Sect. 7 discusses it, and, finally, Sect. 8 draws the conclusions.

2 A Spatial Approach to Develop Tonal Harmony Awareness

The basic idea we propose to foster tonal harmony awareness is the use of spatial representations for music relationships. According to the spatial schema introduced by the Swiss mathematician Leonard Euler in 1739 [16], chords can be seen as areas disposed on a surface crossed by three music intervals axes (fifths, major and minor thirds). This representation, called the Tonnetz, has been later revisited by Hugo Riemann, a German music theorist. In 1896, Riemann published his ideas about harmonic functions, subdividing the tonal space in primary chords and parallel chords [30]. The former category includes the tonic (T), subdominant (S), and dominant (D), built on the first, fourth, and fifth grade of the major scale respectively; the latter category embraces the parallel tonic ($T{_p}$), parallel subdominant ($S{_p}$), and parallel dominant ($D{_p}$), built on the sixth, second, and third grade of the major scale respectively. For the sake of clarity, in the key of C major, the pitches are C for T, F for S, G for D, A for $T{_p}$, D for $S{_p}$, and E for $D{_p}$. As a noticeable property, in a major tonality, all the primary chords are major triads and the parallel chords are minor triads.⁴

Fig. 1.

Chord representation on a circle according to Riemann’s theory.

Following Riemann’s theories, a graphical representation of the harmonic space can be mapped onto a circle (see Fig. 1), where:

• Primary chords are in the lower part, with T in the middle, S on the left, and D on the right;

• Parallel chords are above the corresponding primary chords in the upper part, with $T{_p}$ in the middle, $S{_p}$ on the left, and $D{_p}$ on the right.

This layout greatly simplifies the possibilities offered by tonal harmony. For example, only triads are represented (thus excluding seventh chords or more complex note sets), chords are always in root position (i.e., no inversion is supported), the VII degree of the scale (i.e. the leading tone) is not present, there is no possibility to change from one tonality to another (modulation), and so on. Nevertheless, such a simplified layout turns to be useful to clarify basic tonal functions to a non-expert learner, since tonal centers and their harmonic function are easy to be identified. Please note that all locations can be reached without crossing other tonal areas.

Recalling the spatial approach, harmonic cadences⁵ can be represented by paths connecting the nodes. Figure 2 shows a plagal cadence ($S \rightarrow T$), an authentic cadence ($D \rightarrow T$), and an extension of the authentic cadence including also the fourth degree ($S \rightarrow D \rightarrow T$). More complex patterns can be performed as well, covering the chord sequence of an unexpectedly wide range of popular tunes.

Fig. 2.

Some standard harmonic cadences represented on the chord circle: in green a $S \rightarrow T$ cadence (top left), in pink a $D \rightarrow T$ cadence (top right), in orange a $S \rightarrow D \rightarrow T$ cadence (bottom). (Color figure online)

Fig. 3.

A more complex representation of a major tonality harmonic space including also the secondary dominant $D_s$.

Anyway, this simplified schema can also be seen as the starting point towards more advanced representations of the harmonic space that may significantly improve its functionality. For example, the inclusion of secondary dominants could be evaluated by duplicating the sub-dominant parallel chord, as denoted by $D_s$ in Fig. 3. The higher complexity introduced in the schema is counterbalanced by a lower degree of usability, or by higher requirements in terms of harmony-related competence from the user.

Fig. 4.

The chords of the harmonic space in the A minor key.

The representation of a minor tonality harmonic space could be experimented as well. As shown in Fig. 4, the subdivision between primary and parallel chords is less evident than in the major mode. Primary chords are not all minor triads, since D must always be a major chord. The case of parallel chords is even more complex, as only $T_p$ is a major chord; $S_p$ is a diminished chord, while the $D_p$ is an augmented one. All these aspects of complexity, which clearly affect the usability of the chords circle, could be addressed in a more flexible way by employing different chord layouts to accommodate the music material.

3 Background Work

The space-based proposal to gain tonal harmony awareness has been developed by our research group along two directions, so far: i) harmony embodiment, namely the use of body and physical interactions to enact music-related concepts, and ii) computer-based activities, i.e. ad-hoc exercises to be performed on a personal computer or other digital device. In the former case, chords are represented in a physical space that the learner can bodily explore; in the latter case, the harmonic space is reconstructed through suitable graphical interfaces and user’s movements can be mimicked through a pointing device (e.g., a mouse or the cursor of a touch screen).

Both the approaches have been explored in background research activities, resulting in two solutions already tested in educational environments: Harmonic Walk and Harmonic Touch. These applications will be shortly described in the next sections.

3.1 Harmonic Walk

A first attempt to apply embodiment to music harmony awareness led to the design of a system called Harmonic Walk. The goal of the initiative is to associate chord sequences to the movements of a user in a large-scale bi-dimensional environment. A free exploration of space should let the player acquire awareness of music chords and learn the paths referable to the most common harmonic progressions [26].

From a technical point of view, Harmonic Walk is made of a suitable combination of hardware and software components. The user’s interface consists of a rectangular carpet positioned on the floor that virtually contains the chord circle. The visual reference for chord locations is provided via specific cue points, e.g., small crosses drawn on the carpet with duct tape (see Fig. 5).

Fig. 5.

Image extracted from YouTube, https://youtu.be/iQlYP5dztDY.

A young learner using Harmonic Walk.

As it regards interaction, when the user steps on an interactive landmark, the corresponding chord sound is triggered. This activity could be performed also manually. For example, in a low-tech environment, a music teacher could evaluate the position of the learner on the floor by sight, and play a traditional instrument, like a piano or a guitar, accordingly. Harmonic Walk, conversely, implements an automatic evaluation of user’s position over the sensitive area. To this end, the architecture include a software module aimed at video analysis. First, a video camera mounted on the ceiling and oriented perpendicular to the floor captures user’s movements. Then, the video module analyzes the images provided by the camera in order to detect the user’s position. Coordinates are sent via OSC⁶ to the sound-synthesis module. Such a component, implemented in MAX,⁷ aims to play the chords triggered by the user’s movements detected on the floor.

The dimensions of the tracked area mainly depend on the camera-to-floor distance and the lens’ field of view. As the system is designed for classroom activities, monitoring usually involves a limited area, about 3 m $\times$ 4 m. Bad lightning conditions and occlusions by other users are potential issues to consider during the activity.

Apart from these technicalities, the most relevant problem experienced with Harmonic Walk is the portability of the experience, that requires the availability of a suitable space, a non-trivial hardware setup, and fine-tuning operations.

3.2 Harmonic Touch

Harmonic Touch is a Web platform for the exploration and practice of tonal harmony. It was conceived as a step-by-step path that leads users towards the development of tonal harmony awareness by leveraging on chord perception, gestural interaction and gamification techniques. The platform is publicly and freely available at http://harmonictouch.lim.di.unimi.it/.

Fig. 6.

The interface of Harmonic Touch, Experience $\sharp$1.

Fig. 7.

The interface of Harmonic Touch, Experience $\sharp$2.

As explained in detail in [4], three groups of experiences are proposed to the user:

• 1. Experience $\sharp$1 – “Recognition of the implicit harmony”. The user is asked to match a short music tune with the single chord that best fits the whole melody [9]. The chord is picked from the set of primary and parallel chords introduced in Sect. 2. Chords are represented on a circle, randomly rotated and without any cue about scale degrees or tonal functions (see Fig. 6). The user can freely explore the chord layout during a pre-exercise training phase, so as to mentally build a map of the spatial relationship among chords. This activity can be seen as a digitally revised implementation of Harmonic Walk;

• 2. Experience $\sharp$2 – “Timed recognition of harmonic changes”. After listening to a complete piece (leading tune and chords together), the user is asked to reconstruct it. Each tonal area is triggered by moving one step ahead over a map at the exact time when a new chord is expected. If the click does not occur at the right timing, music stops; if it is performed in advance, a part of the tune is skipped. This exercise focuses on the harmonic rhythm only, with no need to recognize tonal functions. An example of interface is shown in Fig. 7;

• 3. Experience $\sharp$3 – “Melody harmonization”. The final step invites the user to select the right chord at the right timing in order to accompany a known music tune. This exercise has been conceived as the natural evolution of the previous ones, being based on both the recognition of the best-fitting chord (Experience $\sharp$1) and the occurrence of harmonic changes in time (Experience $\sharp$2). The graphical interface, shown in Fig. 8, recalls the one of Experience $\sharp$1: it adopts the same spatial relationship among chords, but tonal functions are made explicit by roman numerals.

Harmonic Touch extends the concept of tonal-harmony space exploration introduced by Harmonic Walk, bringing it into a Web environment. Physical movements in a bi-dimensional space are now translated onto a PC screen, without the hardware and software requirements and the space-related constraints described in Sect. 3.1. The Web framework embeds both the on-screen movement tracker and the sound synthesis module. Another advantage with respect to Harmonic Walk is the possibility to automatically assess and anonymously save user performances into a database [5].

Fig. 8.

The interface of Harmonic Touch, Experience $\sharp$3.

An early experimentation was conducted in two Italian primary schools in February 2021, in the midst of the resurgence of the COVID-19 pandemic. Educational activities involved 46 children (29 females), aged 8 to 11, from 4^th and 5^th-grade classes. For further details, please refer to [6]. Other schools are currently using Harmonic Touch, but the experimentation is still running and results have not been analyzed yet.

4 Open Issues and Research Questions

Harmonic Walk and Harmonic Touch represented successful initiatives, tested in educational environments and documented in the scientific literature. Nevertheless, a critical analysis of both solutions makes some open issues emerge.

Harmonic Walk proved to be problematic to export to other contexts. For example, the required hardware and software equipment typically is not available at home, thus confining the educational activity to school time.

Concerning Harmonic Touch, out-of-school availability is guarantee by a suitable computer-based interface. Nevertheless, two key issues hamper its extensive adoption. The first problem is that exercise settings are fixed: chord layout and positions cannot be altered (e.g., being simplified for cognitively-impaired learners or changed in dimension, shape, and color for visually-impaired users); available chords are those listed in Riemann’s theory (e.g., root notes are predefined, non-triadic chords are not supported, no modulation can be performed, etc.); and so on. Even if the platform lets authenticated users add new songs, thus supporting a certain degree of customization, exercise models are well defined and unchangeable.

The other issue of Harmonic Touch concerns the metaphor itself of moving in space, that is one of the pillars of our proposal. In this platform, the idea of embodiment is translated into the movement of a cursor in a screen through a pointing device. Such a solution presents a number of advantages, including the removal of physical-space constraints and the active participation by physically impaired users, but its didactic efficacy should be further investigated.

In conclusion, the activities taking place in the physical domain (Harmonic Walk) and in the digital one (Harmonic Touch) both offer pros and cons. What we are searching for is a technological solution capable of bridging the gap. The main research questions are:

• RQ1. Is it possible to couple the efficacy of embodiment, intended as movements in a physical space, with the potential of a computer-based interface in order to foster tonal harmony awareness?

• RQ2. What it the most suitable technology, also considering practical issues such as portability, affordability, accessibility?

• RQ3. To what extent is the solution extensible, so as to respond to custom educational goals and accommodate user-tailored experiences?

5 The Technological Solution

In order to bridge the gap between the physical and the digital domain and positively answer RQ1, we designed and implemented an app for mobile devices that embeds real-time image tracking in a video stream and integrates augmented reality.

The first design choice regarded the expected user interaction. In order to selects a tonal area or trigger a new chord, we conceived a mechanism based on markers, namely physical objects carrying graphical information and univocally assigned to chords. In our opinion, the most intuitive way to interact with markers is to temporarily cover them in the frame captured by the device camera. Different strategies can be employed, depending on the characteristics of markers and the device position: a part of the body (e.g., a feet, a hand, etc.) or the whole body, a covering object (e.g., a paper sheet, a book, a paddle, etc.), a direct interaction with markers (such as flipping the sheet), and so on. The system is designed to promptly react to these events; moreover, when needed, the app can also record the timing of user actions. In detail, Experience $\sharp$1 presents a fixed number of markers and simply fosters chord exploration, with no need to track timed events; Experience $\sharp$2 supports a variable number of markers, but activation time is fundamental to assess user performances; finally, Experience $\sharp$3 is based on a predefined set of markers and detects timed events.

Concerning the technological platform addressed by RQ2, the idea of releasing an app started from the analysis of some limitations in previous research. First, educational activities should be easily experienced out of the school. To this goal, as the reference technological platform, we identified an “all-in-one” portable device, equipped with a camera and able to reproduce sounds, with sufficient computational power to track images in a video stream and to synthesize sound. Mobile devices such as smart phones and tablets could do the trick.

As a side effect, this solution potentially facilitates also classroom activities, thanks to a BYOD⁸ approach. Even if more common in higher education [1, 14, 33] and sometimes criticized for its potential risks [2, 10, 32], BYOD has been experimented also in primary school [19, 27, 35] and appreciated for a number of aspects, including: educational continuity across school and home contexts, thus blurring boundaries between formal and informal learning spaces [22]; high levels of student engagement through interactive assignments [18], and parental engagement in their children’s education [25]; personalized and self-regulated learning [29]; reduction of the cost pressure for one-to-one technology provision in schools [12]; relief for technology support [28].

As mentioned before, the app was designed to track markers representing tonal functions. Images are acquired from the device’s camera and processed by ad-hoc recognition algorithms. The use of printable and movable graphical markers instead of a prepared physical space or a predefined computer interface presents many advantages. In this way, it is possible to customize, e.g., the number, size, position, graphical aspect, and intrinsic meaning of markers. An example of markers for Experience $\sharp$3 is provided in Fig. 9.

Fig. 9.

A possible graphic design of the markers for Experience $\sharp$3.

A first noticeable effect consists in overcoming the limits of a 4 m $\times$ 4 m rectangular area (Experience $\sharp$2) and of a circular layout (Experience $\sharp$3). In these scenarios, markers can be placed in space according to any schema, in any desired number, and even moved during the educational activity to accomodate users with special needs. If space constraints are hindering the experience, markers can be printed in smaller size and disposed, e.g., on a table instead of the floor.

Thanks to the use of mobile devices and printed markers, the educational experience can occur in any place, also at home or outdoor. For example, Experiences $\sharp$1 and $\sharp$3 can be performed in the school gym or in the auditorium; and Experience $\sharp$2, that adopts the metaphor of the treasure map, can be proposed in the open air, e.g., in the schoolyard or in a garden.

For Experiences $\sharp$1 and $\sharp$3, the layout made of 6 tonal areas could be initially simplified to meet the needs of beginners (e.g., including only T, S, and D) and incrementally extended, even beyond the diagram reported in Fig. 1.

The graphical representation of chords can be customized, too. In Experience $\sharp$3, standard chord symbols can be replaced by pictograms, sheets with diversified colors and shapes, etc. to improve or reinforce the recognition by young learners. Linear barcodes, QR codes, and fiducials can be used as well.

Concerning accessibility, such a solution can help overcome visual, physical, and cognitive impairments. For example, it is possible to emphasize graphical differences between chord signs by stressing color contrast, increasing markers’ sizes, using bigger or more readable fonts, etc. Moreover, visual markers can be associated with tangible cues for BVI⁹ people. Concerning physical impairments, users are no more required to walk on a carpet or firmly hold a mouse, but they can, say, occlude markers on a table by hand or move the camera away; and these gestures still preserve the relationship between space and harmony. Finally, addressing cognitive impairments, the exercises can be suitably simplified, for example by reducing the number of choices or adopting other spatial layouts (e.g., linearizing and bringing together chord positions for Experiences $\sharp$2 and $\sharp$3).

Please note that all the mentioned aspects go in the direction of flexibility and customization of the user experience, as requested by RQ3.

The last piece of the puzzle consists in the adoption of an augmented reality (AR) approach. The capability of creating responsive environments where information may be superimposed to real objects, typical of AR, already proved to be extremely appealing in fields such as engineering analysis [24], maintenance [8], medical diagnosis [15] and many others.

The alignment of instructional design, system affordances, and didactic goals is the best way to fully exploit the learning potentials of AR environments. The fields of music perception and notation skills may benefit of AR applications. In [23], a melody composed of painted notes is filled by the children on paper and then recognized in real time by a mobile’s camera; once the image is acquired and processed, the notes may be played and checked by the system. A similar application, called Augmented Songbook, is based on pre-printed music sheets [31]. These approaches are very engaging for children and, consequently, they represent effective tools for music education. The simplicity and ubiquity of mobile devices make them suitable for use in classroom and outdoor activities.

Fig. 10.

Image taken from [3].

Recognition of markers and superimposition of chord symbols in AR.

In this proposal, AR is employed to put virtual chord symbols over the corresponding markers in the app interface, as shown in Fig. 10. From a technical point of view, the use of AR mainly responds to three goals: i) making markers easily retrievable during the work session, ii) confirming that the tracking system is correctly managing their recognition, and iii) providing a graphical feedback when symbols are physically covered by the user, thus triggering a music event. In addition, the integration of AR can foster engagement [11] and even improve the app usability, when this issue is properly tackled [13, 20, 21].

6 AREmbody: An AR-Based App for Tonal Harmony

AREmbody is a freely-available iOS app for iPhone and iPad that currently implements only the most advanced activity, namely Experience $\sharp$3. Born as a proof of concept in order to test the feasibility and the educational efficacy of the approach described in Sect. 5, it can be easily extended to the previous two steps.

At the moment of writing, the app is still at a prototype stage, and, for this reason, it is not available in Apple App Store, but it can be downloaded from http://harmonictouch.lim.di.unimi.it/app_download_ios.php. Due to the limitations imposed by Apple on the use of AR, AREmbody can be installed on models from iPhone 6s on and running iOS 13 or superior.

AREmbody was implemented in Swift, a programming language for macOS, iOS, watchOS, tvOS and other Apple platforms¹⁰. The user interface was realized in SwiftUI¹¹.

The application adopts AR to let the user interact with the scene. To this end, two frameworks by Apple, ARKit and SceneKit, have been used. ARKit¹² aims to create a correspondence between virtual and real spaces, thanks to the technique of visual-inertial odometry. This process combines the information coming from the sensors of the device (e.g., the accelerometer, the gyroscope, etc.) with the analysis of the scene framed by the camera. This framework is able to recognize the most important characteristics of the scene, track the differences in the positions of cue points across frames, and compare this information with that coming from motion sensors. In this way, the device can accurately model its orientation and position in space. As a result, ARKit connects the real world with the virtual space prepared by the developer to contain AR objects.

SceneKit¹³ is a 3D graphics framework that lets the programmer create 3D scenes within an application. It combines a high-performance rendering engine with descriptive APIs that allow the ingestion, manipulation and rendering of 3D assets. In order to structure the content of the scene, SceneKit implements a so-called scene graph, consisting of:

• the root node of the graph, which defines the coordinate space for the whole scene;

• other nodes populating the scene and carrying visible content, such as 3D assets.

The spatial and logical structure of a SceneKit scene is determined by the hierarchy of the nodes it contains.

AREmbody integrates ARKit and SceneKit in order to analyze the scene, recognize the presence of markers, and suitably add the corresponding AR objects. In order to improve their recognition by ARKit, markers should be as different as possible. In this sense, the customization of graphical representations of chords not only represents an opportunity to exploit as a learning reinforcement techniques, but also responds to a precise technical goal.

One of the noticeable aspects of AREmbody is the possibility to manage on a single device a number of different users, also organized by school and class. In this way, a game session addressing a group of participants (e.g., schoolmates in a classroom) reduces the dead time for setting up the app and switch players. Moreover, a single app instance can trace, analyze and compare: i) the results obtained by different users in a single game session, ii) educational activities spanning across different sessions, and iii) an experimentation conducted in different classes and schools. In the lower part of Fig. 10, it is possible to notice the name of the current player (in this case, “Mauro”) and the controls to switch to other players.

The app can run on a tripod-held and a hand-held device as well, as shown in Fig. 11.

Fig. 11.

Images taken from [3].

Different ways to hold the device while the app is running.

At the end of each session, results are stored in the device and can be exported in JSON¹⁴ format, so as to feed a database or an external analysis module. The non-trivial problem of assessing these raw data has been investigated in [7].

7 Discussion

AREmbody is a portable solution that, in our intention, should bridge the gap between Harmonic Walk and Harmonic Touch. It shares with the other solutions the aim to make young learners acquire awareness in the field of tonal harmony, by adopting embodiment and a gamification approach. Anyway, the design goal was to simplify the setup requirements of Harmonic Walk, but preserve a kind of embodiment in a physical space that is lacking in Harmonic Touch.

AREmbody supports a personal user experience: educational activities can occur not only during school time in the classroom, but also out of the school. Many game aspects previously constrained by the setup can now be customized, ranging from the appearance of markers (graphical content, size, shape, etc.) to their layout in the physical space (number of markers, relative position, distance, etc.) and the kind of interaction. Any learner equipped with a mobile device can play whenever and wherever, playing alone or with peers.

AREmbody has a great potential, but it needs to be further developed along different directions. First, it currently supports only the last type of experience, which is enough to appreciate its applicability to the full set of activities; nevertheless, a complete educational path based on AREmbody requires the implementation of all experiences in a unique framework.

Moreover, the AR component of the game play is still underused. In the current version of ARembody, the main purpose of AR is to enrich the user’s experience, basically to highlight the recognition of markers, to reinforce their position, and to make the game play more appealing. In a future version, AR could support other customizable features. For example, colors, font types, text dimensions, etc. could fall under the user’s control, thus improving the user’s experience and helping impaired players. Besides, the app could adapt to specific environmental scenarios, such as poor lighting conditions or crowded scenes. Finally, the AR content itself could become more informative, providing alternative text or guidance in form of animations. For example, a visual hint could indicate the proximity of a chord change and suggest the right choice.

Finally, the app should be extensively tested in an educational scenario, and the results should be compared with both traditional classroom activities and those obtained with Harmonic Touch and Harmonic Walk. Unfortunately, two aspects have been affecting the experimentation so far: i) the COVID-19 pandemic, that has been hampering in-class activities throughout the last school year, and ii) the high requirements in terms of user device, a limitation that hopefully will be overcome by natural technological advances and the release of an Android version of ARembody.

8 Conclusions

In this paper we have discussed the use of embodiment to gain tonal harmony awareness, in accordance with theoretical approaches from the past and contemporary gamification strategies.

In order to demonstrate the feasibility and the educational valence of this approach, we have designed three kind of space-based musical activities, already implemented and tested in a web environment. Nevertheless, a critical analysis of our previous work has underlined the opportunity of bridging the gap between embodiment in the real world and the experiences proposed in the digital domain. To this end, we have designed and released AREmbody, an iOS app aiming to foster tonal harmony awareness in young students through embodiment. Its capabilities to analyze the real environment captured by the camera and add AR objects to the scene provide the user with a gamified approach to interact with the physical world. The app supports a number of customizable features that extend the original fixed layout of exercises as implemented in Harmonic Walk and Harmonic Touch. The app also improve performance assessment, ranging from the support to articulated game-play sessions (multi-player, multi-class, etc.) to the possibility to automatically collect, export, and analyze results.

Concerning future work, we hope that the school year 2021/22 will bring students back to school, giving them the possibility to test the prototype together and share their experiences. It will be an occasion to evaluate the efficacy of the proposed methodology with respect to traditional music learning and the fully digital alternative represented by Harmonic Touch. In the meanwhile, technological advancements in the field of portable devices and the implementation of the Android version are expected to foster also a personal and out-of-school use of AREmbody.