Keywords

1 Introduction

Autism Spectrum Disorders (ASD) are developmental disabilities of a pervasive nature, encountered in a number of sub-conditions and variations [1, 2] and affecting diverse aspects of cognitive, social and emotional integrity [3]. Notable symptoms include, but are not limited to, communication skill deficiency [3], irregular behaviors or interaction patterns [4] and adherence to stereotypical activities [3]. The origin of ASD incidence in humans cannot be precisely determined, and may be attributed to hereditary factors coexisting with environmental ones [5].

Over the last few years, the familiarization of individuals with ASD with music education and performance practices, has acquired particular interest. Music has been identified as a helpful medium in stimulating emotions and nurturing imaginative behaviors for people with ASD [6]. The active engagement of ASD learners with music may assist in highlighting their personality qualities and confronting unfavorable patterns induced by the disability [7]. In addition, music therapy, as a strategy for ASD treatment, possesses the potential to enhance awareness of musical concepts and regulate everyday behaviors [8], foster interpersonal communication [9] and stimulate eagerness to perform activities, especially in the cases of impromptu music performance and composition [10]. Moreover, the increasing dependence of music therapy interventions on Information and Communications Technology (ICT) [11, 12] constitutes an important impetus for debate and practical experimentation regarding the use of computer software to instruct creative concepts of music. Specifically, the hypothesis behind imagining an ideally practical approach to computer-based musical activities derives from the inherent appreciation of music as a factor positively affecting the learner’s capacity to engage in creative music composition, for purposes of wellbeing improvement and rehabilitation [13], rather than resort to the reproduction of existing musical content.

Considering the above, the current paper is devoted to how the Terpsichore software music interface, named after the Muse for dance and musical theater in Ancient Greek mythology, attempts to serve the above objectives, while operating as a tool potentially helping learners compose original tonal music and soundscapes. This procedure ideally aims to nurture the development of various non-musical skills associated with behavior, communication, concentration and emotional state, while preserving an educational character and satisfying a reciprocal relationship between music education and therapy.

2 Related Work

In the last two decades, a number of research projects have attempted to incorporate aspects of music education in digital technology, to provide comprehensive assistance to individuals with ASD. To begin with, the SoundBeam [14] simulates the execution of orchestral instruments by utilizing the auditory effect caused by body gestures on an electrically conductive surface. It provides a series of tools enabling users to engage in entry-level composition tasks [15], and was most recently employed in a ‘holistic’ educational practice for preschool children with special needs [16]. Benemin [17] and Octonic [18] were designed to offer advanced reproduction and composition capabilities based on interpretative music phraseology, in the attempt to stimulate emotional consciousness and promote carefree music education. In addition, Google’s Chrome Music Lab online platform [19] concentrates on the instruction of fundamental music notions including rhythm, basic composition and spectral analysis of instruments, while it encourages users to combine and manipulate different types of audio signals.

Research endeavors in the 2010s attempted to employ forward-thinking technological means related to interactive music performance and composition. One of these is the Skoog [20], a cubic tactile and pliable interface that allows the composition of phrases in the form of free improvisation, exploiting the translation of hand-based interactions into MIDI messages as means of sound emission from the computer. This interface has provided structural inspiration for the VESBALL [21], which emphasizes on simplicity in operation for the average instructor and learner and expressivity through piezoelectric sensor interaction, although its didactic features are not extensive. The Music Spectrum software [22] allows users to interact with various musical instruments on a fundamental level, and is particularly tailored to individuals with ASD with a view of alleviating their disability-specific symptoms and fostering their social inclusion. PLAIME [12] is designed to acquaint disabled learners with music theory principles, while placing increased emphasis on action monitoring and feedback. The SoundScape project [23] concentrates on the familiarization of adolescents and adults with various concepts of music and composition in a collaborative environment, and utilizes the iPad as means for mediation amongst users with ASD to enhance social integration through music. Meanwhile, the requirement of inserting explicitly didactic elements in the music cognition and composition processes has led to such endeavors as ImmertableApp [24] and Suoniamo [25]. In the former, participants familiarize themselves with sound synthesis in a simplified but immersive manner, whereas the latter exploits the power of online interaction to facilitate music note recognition and execution processes.

3 Rationale

The projects presented above exemplify the persistent evolution of research in the popularization of creative music practices or associated theoretical elements for mentally disabled people. However, they do not address the simulation of a detailed music curriculum, designed to implement a stepwise and targeted approach towards the composition of original tonal melodies and environmental soundscapes. The proposed curriculum-based approach should ideally encourage learners with ASD to occupy themselves regularly with music in their quotidian life, with an eventual purpose of externalizing their emotions, regulating their behaviors, and improving their quality of life. This statement is justified by the role of music education in empowering the learner’s motivation to perform diverse activities through ‘sensory stimulation’ [26].

The rationale behind the construction of Terpsichore is based on two arguments. Firstly, an ASD diagnosis should by no means inhibit the acquaintance with knowledge fields and techniques that a musically literate neurotypical person normally takes for granted. To render musical notions understandable for learners with ASD, emphasis should be placed on images, contrasting colors and symbols associated with such concepts as notation. This proposal is intended to mitigate possible emotional and cognitive challenges that impact learning potential and cause unnecessary distractions or confusion. Secondly, it is necessary to provide participants with an environment characterized by versatility in activities offered, while still possessing a coherent structure with a focus on user-friendliness and creativity. This includes the interpretation of music as either a succession of notes and consonances, adhering to Western or other tonal conventions, or as any form of sound combination, irrespective of whether its pitch content is clearly defined, similar to the principle of ‘organized sound’ [27].

In general terms, the following three factors were taken into consideration so that Terpsichore possesses an acceptable format worthy of preliminary use and testing:

  • Inclusion of two distinct working modes, labeled as Tonal or Definite Pitch, and Soundscape and Indefinite Pitch (SIP), in order to adapt learning methods as desired, without exclusively resorting to pitch-defined auditory sequences.

  • Ability to automatically provide on-screen feedback to learners, on various actions performed within the interface.

  • Emphasis on practical areas related to the learner’s mental condition and sensitivity, such as frequency filtering and time limitation in interface use, to prevent computer addiction and other adverse consequences in wellbeing.

4 The Terpsichore Software

The version of Terpsichore employed for testing in actual participants with ASD was constructed in 2018, using SuperCollider (version 3.9.3), a choice justified by the ability of the language to convert appropriate code to sound clips, sequences, visual structures and button arrangements, all designed to perform various actions [28]. The interface was transformed, using Platypus [29], into a standalone application for computers running on Mac OS X Snow Leopard (10.6) and above. Figure 1 displays the main menu of the software, from which four different environments can be accessed, starting from the top right and moving clockwise: Tonal Mode, SIP Mode, Software Termination and Options. It is worth mentioning that the Options window allows practitioners to modify time- and frequency-dependent parameters in the software levels prior to making them available to learners, as also shown in Fig. 1.

Fig. 1.
figure 1

Left: Main menu of Terpsichore. Right: ‘Options’ window

Full size image

The objective of the Tonal Mode is to progressively instruct compositional tasks compliant with Western music conventions, such as representation of notes as symbols and letters from A to G. Included activities allow creative freedom to a significant extent, due to some learners with ASD possessing an unconventional perception of musical structure and tonal relationships [30]. However, the potential of tonal music to enhance analytical skills, concentration and overall brain functionality [31] was taken into close account while designing levels. Participants are familiarized with the necessary procedures that allow them to create their own music, and exploit this activity for purposes that extend beyond strict compositional frameworks, such as the understanding of various social situations and prompts given by peers to encourage activity completion [32].

The structure of the Tonal mode follows a distinct direction from fundamental music knowledge towards the modification of existing patterns and the composition of original melodies without a reference song being required. Specifically, Levels 1 through 5 refer to the instruction of notes and durations in music, Levels 6 to 10 concentrate on the modification of existing melodies, and Levels 11 to 14 prepare the ground for unconstrained music composition, which is the clear objective of Levels 15 to 18. Figure 2 presents screenshots of four levels, one for each goal mentioned above. It should be noted that instructors may activate, in most levels, an embedded User Manual, which provides them with on-screen guidelines on how to operate each level; these can be disabled to prevent unnecessary distractions on the learner’s part. The same applies to the Reward Mode, which assigns tasks that learners need to complete successfully to prevent the software from ceasing operation altogether.Footnote 1

Fig. 2.
figure 2

Screenshots of four representative levels in the Tonal Mode. In the Level 8 screenshots, observe that the User Manual (person with book) and Reward Mode (cup) have been activated.

Full size image

The functionality of the SIP mode is principally centered around audio stimuli that comply with soundscapes lacking a definitive temporal and frequency content, including sea waves, moving tree leaves and urban environment sounds. Its objective is to shift the concentration of learners to the overall sound rather than its tonal constituents, especially when lexical cue overload or the process of learning notation induces cognitive challenges. In the mode’s fifteen levels, the areas predominantly addressed are the combination of environmental stimuli encountered in everyday life, the arrangement of sounds on a timeline, the real-time composition of audio through granular synthesis, and the manipulation of externally recorded sounds. This mode adopts previous therapeutic practices used to complement the familiarization of learners with sound. One such example is Level 3, whose reliance on the auditory representation of various emotions assists learners in identifying them through the facial expressions and actions of peers [33]. Likewise, Level 9 pertains to the ability to change the position of sounds in the stereo field, in an effort to assist learners in approximating the origin of sounds and mitigating auditory localization issues [34]. Screenshots of the above levels, alongside Level 7 (percussive phrase composition) and 11 (granular synthesis) are shown in Fig. 3.

Fig. 3.
figure 3

Screenshots of four representative levels in the SIP Mode. Exactly as in the Tonal Mode, all text can be disabled to accommodate the needs of learners with ASD.

Full size image

5 Methodology

During the Terpsichore research, key emphasis was given to the assessment of software efficiency in the creativity development and mental health treatment domains. To achieve this, a four-month pilot study was conducted, which involved a sample of N = 28 adolescent and adult participants (16 males and 12 females) from four different institutions in the Attica region of Greece, aged between 12 and 29. The severity of their ASD condition was specified, by each of the three tutors, 2 as ‘mild’ for 13 learners, ‘moderate’ for 12 and ‘severe’ for the remaining three. Moreover, the vast majority of HP have not been professionally occupied with music, although none has reported aversion to or irritation by music. Prior to the study’s onset, a number of sessions were dedicated to the familiarization of tutors with the software, who were assisted by a detailed walkthrough video.

Throughout the study, instructors were mainly responsible for incorporating the software in conventional music and general education curricula, and identifying the responsiveness of learners to various aspects of the Tonal and SIP Modes, their ability to compose music and soundscapes, and the evolution of their behavior, communication and emotional state. Tutors 1 and 3 devoted eighteen sessions of roughly 45 min each to the software, while Tutor 2 employed Terpsichore in six hourly sessions administered for her four assigned participants. Assessment was completed thanks to a comprehensive semi-structured questionnaire, which includes multiple-choice questions in numerical (0 to 10) and Likert (‘fully negative’ to ‘fully positive’) scales, and optional open-ended questions where tutors provided extensive input and explanation to standardized responses. Sections in the questionnaire involve pre-Terpsichore evaluations of behavior, mental health attributes and relationship with music, performance and operation comfort in the Tonal and SIP Modes, and post-Terpsichore assessment of areas previously addressed.

6 Results

The majority of participants were impressed by the graphical layout of Terpsichore, including colors and shapes, and seemed enthusiastic about engaging in a novel creative activity, after tutors informed them of their prospective occupation with the software. The lexical content was generally more comprehensible for participants with milder ASD symptoms; for this reason, tutors occasionally disabled the on-screen User Manual and Reward Modes to prevent potential distractions. Moreover, through the respective open-ended responses recorded by tutors, it was determined that they intended to employ Terpsichore mainly in the treatment of emotional state, task concentration and music improvisation or composition.

The responsiveness of participants to the Tonal Mode’s demands may be evaluated as generally favorable (M = 6.96, SD = 1.74), while ratings of 7 or above in this area account for three-quarters of the sample. According to questionnaire scores, provided by tutors and based on the quality of produced melodies, participants tended to display more efficient performances in levels where the modification of default phrases is possible (M = 7.68, SD = 2.04) than in levels associated with unconstrained composition from point zero (M = 7.00, SD = 2.05). In other words, learners performed better whenever required objectives were paired with compositional guidelines, rather than requiring full independence in formulating musical phrases. Half of the study’s participants managed to complete all Tonal Mode tasks without particular difficulties or distractions. Main issues reported with regards to the remaining learners include complexity of certain levels towards the end of the mode. More precise estimates for learner performance in the Tonal Mode may be provided through the magnitude of Average Performance per Human Participant (APHP) across all levels, indicating overall good ratings (M = 7.43, SD = 2.02). Participants recorded values as high as 9.32 out of 10, while an APHP of 7 and 8 is respectively consistent with the sample’s 21st and 45th percentile, demonstrating that the resourceful operation of the Tonal Mode is a manageable task for learners. Furthermore, in groups of levels and tasks associated with a specific knowledge area, performances generally increased when activities involved less initiative-taking and greater feedback or guidance from the software environment.

As far as the SIP Mode is concerned, responsiveness of learners to demands was rated, on average, slightly higher than in the Tonal Mode (M = 7.57, SD = 2.08), while an 82% of participants scored 7 or above across the board. The most preferred levels pertained to the organization of environmental stimuli into a moving timeline, and the alteration of amplitudes and panning positions of audio sources rotated around a virtual sound map. The only areas in which participants were particularly susceptible, were levels associated with granular synthesis as a means of constructing an electroacoustic soundscape. The assessment of SIP Mode performance via APHP scores yielded almost identical results to the Tonal Mode (M = 7.45, SD = 1.78). This denotes that the existence of symbols and letters to represent notes, does not negatively affect understanding and performance. In the SIP Mode, average ratings per participant are as high as 9.31 out of 10, while overall scores of 7 and 8 refer to the sample’s 20th and 52nd percentile respectively. The above arguments designate that, although Tonal Mode levels tended to exhibit higher complexity than the ones of the SIP Mode, efficiency was almost equal amongst participants. In fact, no significant differences in learner performance were detected for detected for the two constituent modes of Terpsichore, as the associated non-parametric Friedman test yields \( \chi^{2} \left( {28} \right) = .143,p = .705 \).

Responses of tutors to questions pertinent to post-Terpsichore evaluation, indicate that emotional state, behavior towards peers, elementary communication and concentration on basic everyday tasks, have improved in twenty-four participants focused on the software. No noticeable positive changes have occurred for the four learners supervised by Tutor 2, mainly due to the serious nature of their ASD condition and the reduced time devoted to Terpsichore. Table 1 presents the overall influence of software-based sessions on key aspects related to the development of their mental state, while Table 2 refers to changes in musical literacy and compositional potential as a result of this four-month occupation.

Table 1. Influence of Terpsichore sessions on notable aspects of mental condition.
Full size table
Table 2. Influence of Terpsichore sessions on music cognition and composition skills.
Full size table

Changes in emotional state, behavior, communication and concentration are all strongly correlated with APHP in both working modes, as linear correlation analysis yields \( \left[ {r\left( {28} \right) \ge .824, p < .001} \right] \) for the Tonal Mode and \( \left[ {r\left( {28} \right) \ge .85, p < .001} \right] \) for the SIP Mode. This denotes that participants tended to translate more efficient performances into a more comprehensive short-term satisfaction of diverse non-musical goals, suggesting a discrete relationship between music education and therapy, even though larger sample sizes are needed to further verify this statement. Tutor reports on the performance of learners establish that the software contributed to the acquisition of entry-level music theory knowledge, while it considerably sparked the participants’ eagerness to engage in the composition of tonal music and soundscapes more regularly.

Responses of learners to Terpsichore, as displayed in Tables 1 and 2 above, were regarded as highly promising, while instructors approve of the software’s potential to nurture creativity and support mental state treatment in a constructive manner. Indicatively, Tutors 1 and 3 reported, following the study, that her learners were eager and impatient to use the software in class, thanks to the multiple interesting activities included, while Tutor 2 highlighted the breadth of capabilities offered in diverse disciplines of music, despite the limited time she had to instruct the interface. Overall, tutors strongly recommend Terpsichore to practitioners and music therapists, and concur that further development may lead to the software possessing a pronounced social impact.

7 Discussion

In this paper, we examined the principal characteristics of the Terpsichore software and the effect it had on various adolescents and adults with varying intensities of ASD and relevant symptoms. The results from the study demonstrate, in principle, that the interface is capable of fulfilling the desired education and therapy objectives, while maintaining a manageable level of complexity for the majority of participants. Terpsichore exhibits a compact nature, determined by its operation directly through a personal computer, without the requirement of adding and calibrating external objects. Thanks to the use of different colors and clearly defined learning components, learners are motivated to perform various compositional tasks without pre-existent knowledge of music being necessary, and are empowered to achieve pleasantly sounding musical results.

Following the instruction procedure on learners, the emotional state of most participants has improved noticeably, as has their behavior, interpreted as reduction of tantrums and impulsive actions, motivation to compose music, and excitement to use the software again. A prominent factor for these outcomes may be the classification of diverse knowledge fields into levels, in similarity to sequences of music therapy ‘events’ [35] designed to address principal areas of wellbeing. In the Terpsichore case, the gradual familiarization with core composition-oriented disciplines ensured that learners were continuously encouraged to control the musical content they produced, instead of perceiving music therapy as a commonplace extracurricular process. Moreover, occupation with the software increased the comfort of learners in communicating, at least to a fundamental extent, with individuals they encounter in everyday surroundings. The Tonal Mode was also deemed convenient in the instruction of music theory, as most participants exhibited reasonably good performances in this field, translating into an eagerness to compose melodies either independently or with occasional support from the peer tutor. In addition, the absence of a definite musical language framework in the SIP Mode, certainly affected the slightly increased ability of learners, on average, to independently operate environments associated with soundscapes and environmental audio, without actively seeking guidance from their instructors.

Although the occupation of most learners with Terpsichore can be positively evaluated, the four participants under the supervision of Tutor 2 exhibited less promising performances throughout both modes, while three of these discontinued use with the software at approximately the 70% mark, principally due to tiredness. This finding is responsible for statistical significances in various ratings, for the three distinct conditions of mild, moderate and severe ASD, even if the sample size is relatively small. Performances across both modes were significantly higher when the software was instructed for at least fourteen hours in total, in contrast to the six hours dedicated by Tutor 2 (one-way ANOVA yielded \( p < .001 \) in both cases). Obviously, less serious manifestations of ASD are congruent with an increased ability in meeting the software’s demands. However, it is interesting to investigate, in future endeavors, whether general difficulties in maintaining attention can be alleviated through extensive training or are consistent with aversion to the software altogether.

8 Contributions to the Field

Terpsichore attempts to further the development of sound and music composition interventions in such a manner that didactic and therapeutic elements coexist, so that the above occupation is ideally complemented with the achievement of alternative goals depending on the learner’s personality. The software intends to foster the establishment of an education-therapy loop, where development of compositional principles and techniques elicits appreciation and enthusiasm, inciting the user to employ Terpsichore on a more sustained basis. This can be confirmed by the reactions of most participants, whose positive attitude was coupled with remarks of impatience expressed towards tutors, on when the software would be employed in prospective classes. Most importantly, software sessions and associated performance ratings attest that a standard computer-based structure is sufficient and practical for the purposes served, especially considering that modes and levels are arranged according to discrete knowledge fields and entail task requirements capable of being completed without external connectivity options. Given that the software’s operational principles are not constrained to music education reinforcement, but are also consistent with the alleviation of disadvantageous conditions caused by ASD, Terpsichore opens up new possibilities for caregivers and practitioners who do not have music as their primary area of specialization. In short, the interface advances such special education environments as [19,20,21] by increasing functional convenience for both users with ASD and supervising individuals with varying musical competence levels, while simultaneously placing multiple forms of regulated and freeform composition at the forefront of mental health treatment endeavors.

9 Conclusion and Future Development

Terpsichore constitutes a sustained effort in constructing a software interface that would not exclusively serve as a supplementary classroom or extracurricular aid for ASD special education routines, but would rather favor the composition of original musical phrases and soundscapes, with an eventual objective of rendering the music instruction process engaging and beneficial for the learner’s quality of life. A subsequent step towards software optimization is the extension of research to broader and more sizeable target groups of participants, whose disability is not necessarily congruent with ASD. Moreover, calibration of remote interaction with MIDI, TouchOSC and touchless gestural mechanisms, is another initiative intended to increase the software’s versatility, as is the translation of Terpsichore’s Graphical User Interface (GUI) to various spoken languages and such coding systems as Braille for learners with impaired sighting. Finally, it is important to establish prospective research on prolonged occupation with the software, in order to assess sustainability in music education development and mental health treatment, something that would represent a crucial stepping stone en route to distributing Terpsichore for commercial use.