Keywords

1 Background

In the era of rapid development of the information society, artificial intelligence technology has been popularized in all aspects of our lives. Studies have shown that highly-immersive learning experience, which can make children more focused, engaged and creative during the learning process, provides better learning outcomes than lowly-immersive experience [2]. AR, namely Augmented Reality technology, can integrate the environment of the real world with the virtual world, thus bringing the reality into the virtual world, and integrating the virtual world into real conditions [1]. The project focuses on children’s immersive smart music education, and the research is completed in cooperation with the smart education team of an enterprise. Through the analysis of preschool children’s cognitive characteristics and needs for music education, in combination with the research of AR technology in children products, this project extracts the design points based on preschool children’s needs for music learning, develops the prototype product in the process of iterative experiment and design, and further optimizes the design through usability testing.

2 Current Situation of Currently Available Children’s Educational Products and Trends

The authors conducted a lot of research on smart products for children’s education at home and abroad, and found that most of the currently available products focus on developing children’s ability in specific subjects such as mathematics, Chinese and English. For example, in foreign market, there are such products as “AR Math Game”, which cultivates children’s math ability [1]; and “SpeechBlocks”, which helps children with English learning [11]; in Chinese market, there are “Read Boy”, a brand of early education tablets and “Marvellous Circuit” [4], a brand of circuit education aid used in the teaching of physics, etc. The authors have found that there are few educational products based on music learning, especially on the learning of music theory. At present, there are mainly “Drums”, which helps practice rhythm and reaction speed, and “Children’s Music Cognition”, which helps children play a few simple nursery rhymes. Both of them have a common problem: they are limited to music playing, but do not cover the most fundamental knowledge of music in their functions. By means of literature review, the authors conducted research and analysis on the content of music theory courses and children’s music courses, and found that in children’s music learning courses, obscure and complex knowledge of music theory (chords, sums, etc.) is more difficult to teach, and introductory courses usually teach relatively simple things like notes and beats. This study will further conduct the innovative design of the immersive introduction learning of music theory for preschool children.

3 User Study

The target population of this study is children aged between 4–7 years, who have short attention span, poor ability of autonomous learning. They are at the initial stage of concrete thinking ability and have no patience to accept complex and boring content. Through the analysis of the training target in the Guide to the Learning and Development of Children, we have learned that the abilities that preschool children need to develop include artistic perception (musical sense, aesthetic sense), understanding of appearance description (color, size, shape), basic mathematical ability, basic expression and social contact, and perception of basic physical attributes (quality, volume) [5]. Through a close observation of children in this age group for a period of time, the authors have found that preschool children are more sensitive to musical sense -- they can swing to rhythms and can feel different emotions transmitted in music (Fig. 1).

Fig. 1.
figure 1

Cultivation targets of children aged 3–6 years (from the authors)

Full size image

Through an interview survey of 30 families with children, the authors studied the learning status of preschool children, and analyzed the common problems of preschool children in learning: a. Too many tutoring classes have deprived the happiness of learning and made children tired of learning before they go to school; b. Preschool education is turned into “primary school education” too early; c. Direct instillation of knowledge is not conducive to children’s all-round development; d. In traditional classrooms, children do not have enough concentration; e. Parents attach great importance to education, but they do not have enough time and energy to teach their children, and their professional knowledge is not complete enough to enable them to do so.

4 Game Design

This study designs and develops an AR recognition-based game to help preschool children aged between 4–7 years to learn music theory, and designs the supporting hardware and software. In the game, when children take out an object and put it in the recognition area, the system will recognize it and then gives feedback (music feedback and feedback of whether the operation is right or wrong). The learning of music theory mentioned in this paper is discussed with the knowledge of chord as an example. Children can first learn music theory step by step, and then play the music piece and the chord. The music theory game involves various types of interaction and feedback mechanisms.

4.1 Tangible Interaction

In the interaction of game levels such as level 1, children need to take out the blocks according to the voice instructions of the game, assemble them through self deliberation according to the prompts and then put the them in the recognition area, where the system will then recognize the blocks. After correct recognition, the children put aside the blocks that have been assembled, and then carry out the subsequent chord assembling. The assembling of different blocks represent different chord tones in the game. After all the chords in a music piece (the number of chords selected is 3–5) are assembled, the entire accompaniment of the music will be carried out. The theme of the music does not change and is continuously played. The sound of chords will be inserted in the music only when the correct chords are taken out in time. When a chord is missed or the wrong blocks are picked during the period, the chord sound will not be played in the music.

4.2 Touchscreen Interaction

The game interface is designed for touchscreen interaction based on mobile terminals. The steps designed in the game are: “starting the game - playing the background story - selecting levels - playing the chord course - tangible assembling -AR recognition”. To enter the game, click the “Play” button; When playing the background story, click anywhere on the screen to skip the video, and click anywhere on the screen at the end to switch to the level selection page; Click the level menu to enter the game; After entering the level, the game plays a video explaining the concept of a chord, and then two buttons will appear on the screen: “Try again” or “Continue the game”, and again, click anywhere on the screen to continue at the end of each level.

4.3 Feedback Mechanism

When the building block corresponding to the first note is successfully recognized, the mobile terminal will play the sound of that note, thus enhancing children’s sense of learning experience and sense of achievement, and stimulating their learning interest and curiosity. When a whole set of blocks are assembled, the sound of corresponding chord will be played after successful recognition. Finally, the sound of completion rewards will be played when the game is successfully completed, which will visually enrich the game and produce some changes. When the whole piece is accompanied in a big level, the keynote is played constantly. In terms of chord accompaniment, the sound of chord will be played when the chord corresponding to the paragraph of a single unit is recognized. Otherwise, the sound of chord will not be played. Successfully played chords will get points, while wrong or vacant chords won’t get any point. At the end of the level, there will be statistics of score and ranking, thus stimulating children’s initiative.

5 The Key to Design Implementation

5.1 Principle Experiment of the Recognition Technology

The immersive music education product based on AR recognition relies on the camera recognition of mobile terminals. In order to verify the feasibility of the recognition technology, two recognition schemes were prepared: ① The rear camera of iPhone11 can recognize static objects placed on the desktop in a space. ② Other front or rear cameras recognize plane graphs (if building blocks need to be recognized, stickers must be designed to assist the recognition). Based on the two recognition experiments, the authors have conducted two related experiments of the cameras, turning once and twice respectively, as shown below:

Experiment 1.

Rear camera of iPhone 11 with two turnings. The iPhone11 was chosen as the game device, and the lens steering device was designed using the principle of light reflection on plane mirrors. For ease of use, the desktop area in front of the phone need to be reflected into the lens. Therefore, two mirrors are needed to realize the transverse change of the optical path. If the vertical angle remains unchanged, the path of the reflected light will be wider when the included angle of the mirrors is obtuse. The mirror facing the camera needs to have an angle greater than or equal to 45° with the cross section formed on the back of the phone (as shown in Fig. 2). Under the condition that the angle between two mirrors remains fixed, the authors changed the angle of the whole device in the vertical direction, and carried out the simulation diagram drawing of several reflection structures. The proposal shown in the figure below is the optimal one (as shown in Fig. 3).

Fig. 2.
figure 2

Comparison of the light path width (from the authors)

Full size image
Fig. 3.
figure 3

The optimal proposal (from the authors)

Full size image

Experiment Conclusion.

The authors modeled the draft model from the Angle shown in Fig. 3. As shown in Fig. 4, the two surfaces in the upper right corner we pasted with glass for verification, and it was found that due to the auto focus function of iPhone, the problem of focus instability occurred when facing the two mirrors. The focusing states of the picture changed too frequently, making it impossible to recognize the picture successfully. So the second recognition experiment was designed.

Fig. 4.
figure 4

Modeling of the draft model (from the authors)

Full size image

Experiment 2.

Recognition of plane graphs with the cameras of other devices. The authors conducted the design for the second recognition method. The second method uses the front-facing camera for recognition. Considering the ease of operation for children, a recognition area was set on the product. The front camera can recognize blocks that children place in the area. A mirror was installed to reflect the image in the gray recognition area of the desktop to the front-facing camera.

Experiment Conclusion.

In the experiment, it’s found that the image formation in single reflection is stable and the focus is not changed randomly. There is no problem in the recognizing effect of the stickers.

5.2 Design of the Supporting Product

After the test of the recognition technology, the authors began to design the supporting product for the mobile terminal. In the front of the supporting product, an AR recognition area and a block assembling panel were designed to facilitate various ways of game playing. The built-in drawers on both sides of the product can store teaching aids. The detail of the drawer buckles makes it not easy to be detached. At the same time, division of the recognition area solves the problem of children’s poor spatial response and reduces their cognition burden when using the device. The way of mirror reflection makes the recognition more cost-effective. The supporting product can also be used as an iPad or phone stand, thus realizing higher rate of repeating utilization. Figure 5 shows the processing effect of the physical model.

Fig. 5.
figure 5

Renders and physical models of the final proposal (from the authors)

Full size image

The authors also designed the APP interface and recognition stickers, from the story background, detailed content of knowledge for explanation, specific ways of gameplay, and changes of difficulties (as shown in Fig. 6). ① Explaining the story background: this can attract children’s attention and interest to play, the content of the story also has a certain degree of motivational effect. ② Inviting the user to participate in the chord part of the game: it shows that the chord is not the theme of the song, but bears the role of icing on the cake instead. The authors chose the knowledge of chord, which has not appeared in children’s music education, and then simplified the obscure knowledge. In order to do this, the authors have done a lot of learning, understanding and research, summed up the connotation of chord, and simplified it. ③ In level 1, single chords are taught, which then leads to the entire chord. The design of the stickers has a corresponding relationship with the notes, and there are certain rules, explaining and guiding the children to do things step by step. The blank space in the design encourages children to think by themselves. It allows children to constantly deliberate while playing the game, thus developing their ability of logical thinking. ④ Level 2 and Level 3 follow the same way as Level 1. ⑤ In the big level, children play along with the music, and the system simulates the real effect of playing. ⑥ Considering that the piano can help beginners consolidate their basic skills and make it easier to understand [9], the authors referred to the seven keys of the piano when designing game stickers, which correspond to CDEFGAB respectively. There is a gray scale below, with falling notes on the left and rising notes on the right, thus reaching the range of 49 keys of the piano. This leaves room for the design of more difficult games later. ⑦ The game cultivates such skills as autonomous thinking, memorizing, comprehension, strategy and musical sense.

Fig. 6.
figure 6

The design of the APP interface and the stickers (from the authors)

Full size image

5.3 Operation Process of the Game

The specific operation process of the game is as shown in Fig. 7. When children see the game instructions on the iPad or mobile interface, they will put corresponding blocks or the combinations of blocks in the recognition area, so as to have learning interaction and pass levels according to the feedback shown on the interface. In addition, spare blocks that have been assembled can be arranged on the gray assembling panel for easy access.

Fig. 7.
figure 7

Demonstration of game operation (from the authors)

Full size image

5.4 Usability Testing

In the stage of lab usability test, the authors verified whether the design can help children better understand and learn the knowledge of music theory by observing and recording the behavior performance of a 5-year-old boy with no knowledge of music and a 7-year-old girl with some foundation of music theory and playing skills during the music game of the product (as shown in Fig. 8) and analyzing the problems during the use. The task flow in the test is: first, we told the story behind the music theory game to stimulate their interests in the game; Then we played the video of chord story for input learning, so as to help them understand the basic concept of chords; Finally, we guided the children to play the primary level 1 and the advanced level 4, observed their behavior during the game and the problems they had, and assisted them with explanation and provided guidance when necessary. In the test, both children were interested in the assembling of Lego bricks, and both of them liked the music game. But the test also revealed some problems of the design: ① Children often get Lego blocks in order to build the shapes they want, and will sometimes easily ignore the rules of the game; ② The AR recognition area and the feedback mechanism need to be improved, children are impatient when playing interactive games. ③ Children without knowledge of music are more likely to understand the concept of chords in the game, while children with a foundation of music theory may find it more difficult to understand (due to the influence of the instrument knowledge they have learned) ④ The rhythm of the advanced level is fast, and children cannot quickly understand the appearance of each chord; ⑤ The basic points of knowledge in the game content are not easy to understand, such as the concept of “stave”, etc., requiring further optimization of the content.

Fig. 8.
figure 8

Usability testing of the prototype product (from the authors)

Full size image

6 Conclusion

To sum up, the tangible game and music feedback based on immersive interaction brought more excitation and immersive experience to beginner-level children. As an “Internet +” solution of traditional tangible music instruments, this design helps children understand what these tangible blocks represent and guides them to play music with the assembled blocks, thus enabling them to participate more in the interaction of music theory and promoting their physical coordination, spatial ability, etc. in a new way of learning.

The following aspects need to be considered more in our work of optimization and iteration in the future. First, AR games are a novelty for children, but when the feeling of novelty disappears, they will lose their interest in the games [6], we need to further study how to design games to not only increase children’s learning experience, but also improve the difficulty of the games and enrich the interactive content, so as to keep children fresh and motivated to learn more; second, in addition to the most basic knowledge of music theory, such as chords, we also need to focus on the games of other knowledge points (the playing method of higher-order games with a wider range), the design of dynamic effect and voice interaction, and the collection of big data for children’s individual learning. In future research, we will focus on these fields and design gameplay, interaction and feedback mechanisms in detail so as to effectively provide children with immersive and stimulating learning experience.