Keywords

1 Introduction

Influences and mutual interactions between mathematics and music are connected by the abstract power of category theory [24]. A category is constituted of objects (points) and transformations between them (arrows), that verify associativity and invertibility properties. Transformations between categories (functors) map objects into objects, and arrows into arrows. Transformations between transformationsFootnote 1 (natural transformations) are also defined. The formalism of categories is particularly suitable to the description of nested structures, and to the translation of structures from one context to another. Categories have been applied to music, especially in music theory [13, 20, 31, 34], and in the study of musical gestures [3, 4, 20, 32]. Recent developments [25, 26] apply basic concepts (e.g., 2-categories, see [24]) to simple music structures,Footnote 2 as well as to more complex phenomena such as gestures of the gestures (nested and composed gestures; for ‘gesture of gestures’ see [32]) found in both conductors and performers in the orchestra, and ‘gestural’ communication between the conductor, the composer (who ‘thinks of musical gestures’), and the listener, in terms of colimit (the conductor) and limit (the listener) [25]. This leads to the definition of gestural similarity within music to investigate analogies between the gestures of different performers, and the perception of these gestures musically. For example, two movements with increased energy that both produce the same transformations on their respective sound and sound spectra, such as an increase in loudness, are similar. Also, a simple musical sequence and a simple drawing with lines and points are ‘gesturally similar’ if they appear as being produced by the same gestural generator: e.g., a ‘staccato’ movement can generate either a sequence of staccato notes on the piano, or a sequence of points on a piece of paper [25]. This topic is connected with synesthesia, crossmodal correspondences, and the definition of audio-visual objects [23, 37]. A detailed analysis of the implications of these studies in the psychology of perception, and of music perception [36], is outside of the scope of this paper. Nevertheless, problems such as the connection between the different art forms, or the extraction of ‘essential information’ from a visual or a musical object inclusive of the comparisons between them, are strongly related to psychology. In particular, a psychological experiment was recently run to assess the degree of similarity between a set of short musical sequences and visual shapes [8], validating gestural similarity [25]. Other studies have been focused on finding analogies between musical sequences and curves drawn by listeners [21] in the light of psychology. Music analysis can benefit from the mathematical theory of categories. For example, an analysis of a piece of Western music, and the listening to its performance, usually lead to different results, with a non-negligible difference between constructs and saliencies [30] that can be categorically described through non-commutative diagramsFootnote 3 [27]. The union of this information provides complete musical information, while their intersection gives essential information about the considered piece of music. Other applications include strategies to navigate complex piano scores, which may be modeled via categories [1, 2].

In this paper, we present a new application of categories. In Sect. 2, we discuss the definition itself of a musical instrument from a categorical point of view. Then, we analyze the interaction between performers and non-classical instruments, such as augmented musical instruments, which include purely electronic, as well as acoustic instruments which have been augmented by electronics.Footnote 4 In this framework, there are also new instruments that involve muscular tension of the performer. Muscular tension, in this case, is not used to put into vibration a classical instrument, but acts as a direct source of information, opportunely detected, and mapped, into sound [11]. Moreover, there are some instruments that use embedded artificial intelligence, allowing transmission of information between them, and between audience and performersFootnote 5 [38]. This opens new paradigms in human-machine interactions, and it makes the diagram of Fig. 1 commutative. Of course, we can define new functors and natural transformations to connect all these areas (music theory, lower and higher structures, performance). Finally, in Sect. 3, we apply a metaphorical use of the concept of classifying topos, recently proposed as a bridge between different areas of mathematics [7], to investigate connections between music and the visual arts.

(1)

2 Definitions and Commutative Diagrams

A classical definition of a musical instrument is: ‘a tool to make sounds/musical notes’. Thus, a musical instrument is characterized not only by its physical body but also by the performance techniques. This reminds us of the mathematical definition of a group: a set with operations defined on its elements, with the identity, the inverse element, and associativity. We may wonder if musical instruments might be seen as a group, with the physical body of an instrument as a set, and playing techniques as group operations. However, as we will see, there are some issues.

If the performer does not make any movement, he or she is making an ‘identity’ gesture; if the performer is playing and then stopping the sound, he or she is making an ‘inverse’ gesture: the violinist can detach the bow from the strings to interrupt the sound, and the flutist can interrupt the airstream. However, such an ‘inverse gesture’ is not unique: even if we consider only one instrument, gestures to stop the sound can differ in speed, time, and so on. Thus, the group definition does not apply. Another issue is about composition of playing techniques. It is in general not possible to compose gestures of different musical instruments: hitting a violin with a hummer could only destroy the violin.Footnote 6 Composition of gestures on a musical instrument should only involve gestures that naturally ‘live’ in the space of gestures for that instrument.Footnote 7

(2)

We may try to define musical instruments as a category, defined by objects (instruments) and morphisms (playing techniques). This is not a formal categorical definition, but a qualitative one. The same issue as before about techniques arises also here. Instead of considering generic ‘playing techniques,’ we should focus on musical gestures on each musical instrument: a violin is defined by the ‘body’ of the instrument and by the gestures (specific for the violin) the performer has to do in order to make the violin sound. We can still refer to playing techniques, but with this restriction: ‘playing techniques for this or that specific instrument.’ Composition of morphisms would involve specific gestures for the violin, for example a crescendo gesture (increased bow pressure/acceleration) followed by a staccato phrase (sudden interruption of the contact between bow and strings), or a staccato phrase played with a crescendo dynamics. Now, let us consider the whole orchestra. We can extend the categorical diagrams that include gestures of orchestral performers and conducting gestures presented in [25], by considering new instruments. Diagram 1 is made commutative through new techniques of electronically augmented instruments allowing for instrument-instrument communication. Additionally, we may hypothesize the existence of a virtual ‘conductor’ (not a robotic one that conducts human performers),Footnote 8 e.g., a control device that automatically, and autonomously, coordinates electronic instruments. If this is put in communication with a human conductor on the top of the diagram, the two vertices in Diagram 1 are connected, and the diagram assumes a ring shape. The reality of the musician-musician, and instrument-instrument communication, is more complex, involving feedback and thus two-sided arrows.

Communication between smart instruments may be made similar, or completely different, from communication between human performers. In general, it is not just a “copy” of human communication. Also, smart instruments may be programmed to be more autonomous, to allow a higher degree of control by human performers. Diagram 2 includes extended techniques,Footnote 9 that can be freely thought of as Kan extensions [24] of traditional techniques. The idea is then summarized in Diagram 3.

The diagram of Fig. 1 represents the relationships between the musical score, the performance, the body of the instrument, the playing techniques, the produced sound. The musical score and the physical body are ‘static’ elements, while both performance and playing techniques are ‘dynamic’ elements. The definition of a musical instrument not only as a physical body, but in terms of playing techniques, may be compared with the duality of music itself, such that music consists of scores (in the cases where a score is present or available) and performances. In fact, as pointed out by Gérard Genette, music can be seen as a “2-state immanency” that are, respectively, formed by the score and the performance [15]. Transitions from scores to performances have been mathematically investigated [33]. Both the physical body of an instrument and its playing techniques admit extensions, which may be investigated categorically. Also, we can wonder provocatively if a musical instrument can be considered as such only when played. Conversely, a generic object could be considered as a musical instrument if it is used as such. Duchamp’s provocations about the concept of “art object,” concrete music, and, earlier, futuristic ‘intonarumori’ [35] move in this direction.

(3)
Fig. 1.
figure 1

Our investigation of musical instruments and their playing techniques can be compared with the dualism between musical scores and performances. Scores and physical bodies of the instruments belong to a static dimension, while performances and playing techniques belong to a dynamic dimension.

Full size image

3 Classifying Topoi, Bridge Objects, and Music/visual Arts

We may generalize our analysis, including areas previously investigated, such as the relationship between music and visuals through categories [25]. A unifying concept is a ‘bridge object’ inspired by classifying topoi, used to unify geometrical theoriesFootnote 10 [5, 7]. Topoi have already been proposed in music [31]; however, our approach is meant to be more intuitive and interdisciplinary. We are introducing here topoi and not simply (small) categories to be able, at this level of generality, to borrow a connecting formalism between disciplines via a metaphorical use of topoi as bridges [5]. We may connect different objects, related the one to the other, via a third one, which may be built independently, or out of each. Also, we may solve problems in a field by using methods from another. We may associate a topos to any mathematical theory, the ‘classifying topos of the theory.’ According to Olivia Caramello, it “represents the framework in which the theory should be investigated, both in itself and in relationship to other theories.” Classifying topoi may be metaphorically used as bridges to transfer information from one theory to another, and thus, topoi can also be considered as tools for dynamic unification (translations of objects, isomorphisms, structures) of mathematical theories. The ‘transfer’ operation is usually done in category theory via functors; here, we use the concept of bridge for transfers to contextualize the idea of ‘general generator’ of both sounds and images, as discussed later. Two theories having the same classifying topoi (or toposes) are called Morita-equivalent [7]. Two Morita-equivalent topoi have some common ‘semantic content.’ Such a semantic core would be constituted of “different manifestations of a unique property.” The idea of topos as a ‘translational tool’ is based on the existence of multiple perspectives, and multiple representations for that topos, allowing for its use as a bridge and as a connecting object; we may then intuitively think of “different instances of a unique pattern” [6]. We also need to define a ‘semantic core’ when we wish to investigate analogies between different works of art, or analogies between techniques and strategies in different forms of art. This would constitute an extension of the concept of gestural similarity: not only the search for a common gestural generator — what Caramello would call a static unification, with a general item generating two other items through descending arrows — but a dynamic unification, through a path from item 1 (that is music in our analysis) to item 2 (that is visual arts in our analysis), see Fig. 2. Here, we use the terms ‘bridge object’ instead of ‘classifying topos’ to remark that we are not using this concept in a strict, technical way; instead, we use it as a metaphor [6] to extend the idea within an interdisciplinary framework. Also, as bridge objects are more general than topoi, we may define invariants — and in our artistic framework, we are strongly interested by invariants. We may imagine, for example, that Romantic music and Romantic paintings are metaphorically ‘Morita-equivalent’. We may also wonder if new perspectives on artificial intelligence in music technology and visual images may also be, in some way, equivalent.Footnote 11 The use of a dynamic unification can be useful not only for the analysis of existing works, but also for the making of new works of art.

Fig. 2.
figure 2

Static unification (top) and dynamic unification (bottom) in Caramello’s theory, with references to music and images. The ‘essential idea’ characterizes the common gestural generator that produces a sound or an image that are similar between them. Some detail of music and visual arts’ worlds are given in Fig. 3; a possible relationship between essential ideas and bridge objects (inspired by classifying topoi) in this framework is described in Fig. 4. The unifying object, according to Caramello’s theory, is the bridge object itself. Upper and lower diagrams can be connected, creating a commutative diagram, via an arrow joining the bridge object with the essential idea. We might dare to identify the bridge with the essential idea (and thus, that arrow would represent an identity), that could either be ‘specified’ into a specific artistic world (diagram above) or first obtained via an abstraction from a specific artistic world and then specified into another one (diagram below). With a unifying object such as a colimit the upper diagram implies the lower one; with a topos, also the reverse is true.

Full size image
Fig. 3.
figure 3

An attempt to join musician’s playing, traditional and smart musical instruments, visual artworks and visual artists’ activity exemplified in drawing activity, and gestural similarity comparisons through the unifying power of bridge objects. The diagram here contains the ‘essential idea;’ how this is connected with bridge objects, is shown in Fig. 4. Smart instruments that enrich potentialities of traditional instruments may inspire smart tablet to enhance visual creation. As a ‘categorical fractal’ (a potentially nested structure) we can draw similar diagrams within each of the objects: for example, music/sounds and drawing/image can be analyzed in detail, comparing musical structures with visual shapes, analyzing their similarity, and the inner movements that ideally depict each melodic line or harmonic progression, obtaining a nested structure. We could also define essential ideas for smaller parts of the diagram with connections and transformations between them.

Full size image
Fig. 4.
figure 4

The essential idea at the basis of musical or visual creation, that can be connected (see caption of Fig. 2) to gestural similarity studies, helps to build an intuitive concept of a bridge object — inspired by classifying topoi — that is meant to be a sort of ‘bridge’ to compare two different worlds, translating properties from one to another.

Full size image

Here, we use a unifying concept, the ‘bridge object,’ inspired by the classifying topos, to connect music with visual arts. Actually, both music and visual arts have theories and the body of knowledge defined by techniques, styles, materials, contents, and ideas. Topoi-inspired bridges may be general, and abstract tools to catch at once the variety within a given art form and transfer/translate it to another field, see Fig. 4. The elements of music and visual arts are described in Fig. 3. In Fig. 3, the same ‘basic idea,’ contained within the bridge/topos, can be represented and developed in the form of different artworks — we can think of artistic currents, where similar ideas may be present within visual arts, music, and also, literature. The essential idea can be caught by artists, and translated into specific gestures on canvas/paper (with brush or pencil) or on musical instruments.Footnote 12 Also, bridge objects ‘indirectly’ inform specific gestures the artist must make to produce the art object. Thus, we may compare the gestures between them, and we may also compare music/visual art which results from these gestures. In this sense, while investigating gestural similarity, we analyze arrows (gestures) and points (final results). At first, we can represent mental constructions with points, arrows, and commutative diagrams for the simple, ‘practical’ applications. Going higher with the abstraction, topoi can constitute a source of inspiration for the metaphor. However, underlying all these metaphors there are universal constructions. One of them is explored in [25]: within the orchestral setup, the role of conductor can be seen as a colimit, and the role of the listener/audience as a limit. In this paper, we extended this approach to include not only gestures of musicians but also the instruments they use and the sounds they produce. Also, the ‘gestural generator’ cited in [25] for music and images is here extended to not only the gestures of visual artists but also to their tools and the drawings they produce. In the specific case of smart musical instruments, we have extended sounds, which enrich sounds obtained with traditional instruments. On the left side of the diagram of Fig. 3, there are commutative squares with instruments, smart instruments, extended sounds and traditional sounds. A similar structure applies to visual media in the right side of Fig. 3. If we include robotic performers, we might analyze some robots playing traditional instruments, humans playing augmented instruments, and vice versa, with all their various and interconnected communication strategies. The same applies for a human which conducts robotic performers and a robot that conducts human performers. Playing smart instruments may require additional, specific gestures, which may be compared with (traditional) gestures through natural transformations (not shown in the diagram here), that can also be compared with extended techniques. The diagram in Fig. 3 can be made tridimensional, with the inclusion of external references, such as beautiful geometric and/or natural shapes that can inspire both visual images and the production of music [28]. The same geometric form can inspire visual studies as well as the design of new instruments. This is the case of the Telemetron [14], designed with the shape of a dodecahedron and able to sound in the absence of gravity. This is also the case of the CubeHarmonic [29], which embodies concepts from music theory such as the Tonnetz, as well as the permutation of Rubik’s cube — a tonnetz with permutations. Also, the structure of musical instruments itself influences the necessary gestures, and the peculiarity of each instrument affects the expressive potential of music performed on it. Thus, some inverse arrows could be defined and investigated. The strength of this representation is the gradual connection between the ‘matter’ of music, the musical instruments, and its high-level content; that is, between instruments’ physical bodies and playing techniques, the musical pieces, and their performers, and possible external music references within the other arts (or nature) as a source of inspiration, see Fig. 5. Actually, categories already inspired connections between objects and processes in nature and abstract thinking [12]. All this would allow a more unitary vision of different things. Mathematics can intervene as a source of inspiration or analysis, as well as provide a kind of connecting language between these elements. In Fig. 4, the ‘essential idea’Footnote 13 at the basis of Fig. 3 can be used to build a topos-like bridge for a generic ‘artistic expression.’ We may investigate whether any structure would imply gestural similarity, making the diagram commutative, and expecting to find restrictive conditions — on the set underlying the category — for commutativity. In this way, we do not limit our investigation to the research of the ‘general generator’ of both sounds and visuals, as done in gestural similarity studies [25], but we may suggest how to translate constructions and knowledge from one area into another area.Footnote 14 As a practical example, any technology that augments the sound potential of musical instruments may be used to develop smart drawing devices to create complex visual structures from simple pencil movements. Also, teaching strategies that have been successfully exploited in the framework of instrumental performance can be applied to other areas. In Fig. 4, arrows from the musical world to the visual-arts world may, of course, be inverted, obtaining another perspective on the ‘topos.’ The use of bridge objects could initiate a fertile debate about the problem of ‘invariants’Footnote 15 and the concepts of the ‘universal’ in the arts, and it is no coincidence that mathematics often investigate invariant properties, and universal constructions play a decisive role in category theory.

Fig. 5.
figure 5

Drawing by M. Mannone.

Categorical formalism of functors can be applied to nature’s forms and growth processes of different species [28]. A suitable mapping, as the action of a sonification functor S, can transform these forms, their comparisons, and their transformational processes into music, music variations, and developments [26].

Full size image

4 Conclusion

In this paper, we extended current research between category theory and music to the definition of (traditional or augmented) musical instruments, and (traditional or extended) playing techniques, including the mutual interaction between performers and instruments, allowed by embedded artificial intelligence. The dialogue between instruments and musical performance practice/composing is two-fold: musical thinking influences the development of new instruments, but also the presence of new instruments stimulates musical thinking.Footnote 16 External influences to music do not involve composition/musical structures only, but also the instruments themselves. We also cited the concept of classifying topoi and their extension to ‘bridge objects’ to generalize current categorical studies on the relationship between music and visuals.

Future research could include both theoretical and computational developments of the proposed ideas. Some future, concrete applications may include musical analysis and improved pedagogical strategies to approach new instruments, extended techniques, and strengthen their connection with traditional instruments and techniques. Also, future research should be devoted to the connection of isolated applications of categories in the areas of music theory, music performance, score analysis, composition, comparisons with some extra-musical material, and musical instrument analysis. Thus, perhaps through the definition of suitable natural transformations, categories can be used not only to analyze topics from music but also to connect these topics from a unitarian perspective. Finally, taken as a whole, this may help us understand the richness of music and, in general, of the arts, and the potentialities of contemporary mathematics to describe in concrete terms the variety of human thinking. These ideas help overcome stereotypes of separation between disciplines, in the framework of research, and, in particular, of STEAM (Science, Technology, Engineering, the Arts and Mathematics) pedagogy.

The authors are grateful to Olivia Caramello for conversations and reading suggestions about classifying topoi.