Hermeneutic Implications of Cultural Encoding: A Reflection on Audio Recordings and Interactive Installation Art

Abstract

This article proposes a reflection on what the emerging discipline of digital philology means to the encoding, preservation and access of multimedia cultural material. In particular, it focuses on audio recordings and interactive multimedia installations. It describes a general setup required for the re-mediation of audio recordings that supports a philologically informed methodology for long-term preservation, and it points out some of the major challenges posed by interactive art in long-term preservation. This article extends the discussion started at the IRCDL in 2013, in the light of the authors’ activity during this time.

1 Introduction

Digital libraries are defined by dedicated and often sophisticated storage, retrieval and management systems. But most importantly they provide their users with trustworthy content. Trustworthiness applies both to the accuracy of the data and to the reliability of the content, i.e. to its (cultural) meaning within a specific context or community. The services offered by digital libraries are only effective if the content they manage meets the requirements of integrity, consistency and reliability (a term related to the much more complex concept of authenticity). This article proposes a reflection on what the emerging discipline of digital philology means in the preparation of cultural digital material aimed at digital libraries. In particular, this article evolves the discussion on digital philology started at IRCDL in 2013, in the light of the authors’ activity in the field of the digital humanities during this time.

1.1 Cultural Encoding

The way in which inscriptions are photographed and text corpora are transcribed and encoded, as well as the way in which a sound recording is re-mediated¹, “is crucial for the way in which these research objects will be studied in the future” [1, p. 11]. In other words, the digital representation of data (the bit stream) and the organization and presentation of the data (the cultural interfaces [2]) are not neutral with respect to the final perception that users have of the ‘real’ or ‘original’ object, which may often not be available for comparison. In addition, the relationship between the digital objects and the real objects they allegedly represent “is not only very complex, it also develops over time” [1, p. 10]. In order to ensure that the electronic sources, that scholars and the general public are getting more and more accustomed to refer to for their research and personal interest, meet the requirements of authoritativeness, accuracy and reliability (see Sect. 2), it is necessary to define what makes a digital document authoritative, accurate and reliable. This definition implies the understanding of what the document means in its cultural context and it is therefore not limited to technical questions that only computer scientists and engineers should be in charge of. “Digital philology” is the multidisciplinary research field that addresses this problems. While philology (“without adjectives” [3, p. 65]) has a long tradition, the reflections about digital philology are very recent, and so is the academic production about it. There is much work to be done in order to provide new methodological and operative tools to scholars whose aim is to author electronic editions based on digital sources. The shift that the electronic medium introduces in how texts are coded and accessed goes beyond their presentation, affecting the perception of the content.

The existing scientific literature about digital philology is mainly limited to written texts²: making a parallel in the audio domain, any recording of the same event can be considered as a variant copy of the “original” event, thus being the equivalent of a manuscript for textual criticism. It remains to be defined what the text can be for a multimedia interactive artwork where the aesthetic experience (the “effect”) is only achieved in a dynamic process and with the combination of physical and virtual elements that may change their micro- and macro-relations over time.

Unlike the field of text encoding, analysis and philology, where a longer tradition provides ascertained references to scholars and researchers, the field of audio documents long-term preservation is relatively new and it lacks a similar background of knowledge and experience. Audio documents have gained the status of documentary sources only recently, and it is not uncommon to find that texts and icons are still considered “first class” sources while sound recordings are considered “second class” sources for scholarly studies. Unless certain criteria to ensure that digital audio (and other multimedia) can be considered authoritative, accurate and reliable sources [4], this trend may be hard to get rid of in the future.

Besides the on-going multiplication of digitized documents, the number of born digital documents is also increasing, making the definition of procedures for storage and cataloguing even more urgent because a “physical original” is never available for comparison. The hybrid nature of multimedia installations imposes the consideration of a grey zone between analogue, digitized and born digital documents, i.e. cultural material where meaning is found in the integration of physical and virtual elements.

This article is organized as follows: Sect. 2 explores the concept of digital philology, with specific connections with traditional philology and textual criticism. Section 3 discusses some concepts involved in the preparation of the cultural digital materials. Section 3.2 instantiates these concepts in the preservation of audio documents and introduces some challenges posed by interactive installation art.

2 Digital Philology

“Where lies the truth – or where lie the truths – about philology?” [5, p. 3]

The philological activity of text comparison goes back to Antiquity: one of the means to compare texts was to present them in parallel columns. For example, Origen’s third-century Hexapla presented six versions of the Old Testament (including the Hebrew text, a transliteration of the Hebrew in Greek characters, and four Greek translations) in parallel alignment [6]. The comparison, a core activity in the philological analysis, is implicitly based on a notion of “text”. According to philology, any text is a variant copy of an archetypal text, which the philologist is trying to reconstruct. Before even mentioning sound and music, let us consider how the adjective “digital” is associated to the term “philology”. First of all, the simple fact that a text is represented in electronic format and examined on a computer screen rather than a paper book does not justify the expression “digital philology”. Just as an electronic reproduction of Leonardo’s Mona Lisa is not “digital art”. The medium is certainly not neutral, and we are addressing this issue in a few lines, but the adjective “digital” next to philology needs to have a greater impact on the concepts and the methods of philology. It is, in fact, a new research field, stemming out of the combination of two disciplines: philology and computer science. Computer science brings to the table its approach, some of its concepts and methodologies – not only technology. From its part, philology is changed in its own concepts and methods thanks to the encounter with computer science. “Going beyond, discovering the new frontiers of philology, does not depend on the computer. It depends on us, the scholars” – said Marcos Marìn in 2001, when apparently “many people [were] still looking at computers as enemies of the intellectual activity that has characterized the approach to texts from the beginning. We are not convincing them by saying that we get more accurate accounts of words, or thousands of examples. What is different is not the quantity, it is the new insights, the new questions that we can ask.” [7]

The advantages that computer science offers “traditional” philology (or probably we could start saying non-digital philology) are twofold:

• 1. technological tools (for quantitative analyses and for sharing corpora through a network);

• 2. methods and concepts that stem from a theoretical reflection.

From the point of view of how deeply computer science modifies philology by contributing to it, two distinct steps can also be distinguished:

• 1. preparation of the working materials;

• 2. support to textual criticism (assisted or [semi]automated analyses).

Often the expression “digital philology” refers to the “effort of reproducing texts on paper in electronic format as faithfully as possible” (point 1), but the output of this operation is in fact “the starting point of the philologist’s work” [3, p. 68]. Although it is desirable that the second step is also accomplished (point 2), this first and preliminary step is crucial because the trans-coding of the text may lose some information and/or introduce errors and artefacts. While the digitization of a text can happen in multiple ways (direct acquisition by means of scanner device, or text encoding by means of formal languages, often based on tags such as XML), audio is generally digitized by extracting the signal from the source medium and by memorizing it onto another medium (usually a redundant array of independent disks). Innovative methods to store audio recordings are based on optical systems such as PoG (Photos of GHOSTS: Photos of Grooves and HOles, Supporting Tracks Separation) [8]. These systems may not only bring about new approaches in the workflow of preservation, but could (and should) trigger new reflections on the meaning of re-mediation itself. Regardless of the source medium and of the digitization technique, it should be clear that re-mediation and [cultural] trans-coding are never neutral operations.

3 Preparation of the Working Materials

“Even an image capture and editing, which may at first sight be a rather straightforward and ‘objective’ procedure [...] require intellectual, critical choices, interpretation, and manipulation.” [9]

In order to plan and to perform the trans-coding, it is necessary to have a model of the object/document. The definition of the model is where computer science and philology may have communication problems, because “humanities generally show terminological ambiguity due to the heterogeneous and elusive object of study”, while computer science deals by definition with the “processing of data (encoded information) expressed in non-ambiguous languages” [10, p. 48]. Creating a model of the object to be digitized means to analyze it and to select what relevant features will represent it. And as a reminder of the multidisciplinary approach that should characterize any work in this field, it is important to keep in mind that the creation of the model is not required exclusively for the sake of the computer: it should rather be seen as an “important space for analysis and for the formalization of the knowledge about the subject of the study” [10, p. 53].

The definition of the model is a “crucial” part of the preliminary activities (see [1, p. 11] quoted in Sect. 1): any further analysis will manipulate the electronic representations of the original physical objects. Gigliozzi [10] suggests that a model already underlies any written text (a code for graphic symbols, syntax, narrativity, language, ...), and that it can be useful to reflect on what features make it effectively represent the message (information) intended by the author. Starting the transition to the audio domain, the notion of “text” is among the first to be defined. Is any recording of an acoustic event or electronic composition a witness of the archetypical sound? Is there anything such as an archetypical sound at all? In the authors’ methodology (see Sect. 3.2, and [11, 12] for more exhaustive explanations) every audio document is considered a “master” recording in the sense that it bears witness of the acoustic event in a unique way, due to the manipulation and conditions that the document has been subject to through the years, including the mere ageing of the carrier. Without attempting to say that an archetypical sound exists, the authors see each recording as the equivalent of the written “text” according to philology.

Considering that the equivalent of textual criticism is performed by musicologists, linguists and experts other than computer scientists, the author’s work finishes when the equivalent of the diplomatic editions (defined in the next lines) are ready. The process of preservation goes from the diagnosis of the physical document to the preparation of the digitized document for access (resources and interfaces). The documents for access may have been re-organized during the cataloguing, in function of the contents: the type of object that is produced directly from the digitization is called “preservation copy” and is by definition the equivalent of the diplomatic edition (see [13]). According to the definition of diplomatic edition, the eventual mistakes or imperfections of the physical document are maintained in the digital copy, because they provide information about the author’s creative process (e.g., erased words), about the history of transmission of the document, and about the ageing of the document (from dog-eared pages to patches of mould). The implication is that no restoration is allowed in the audio of the preservation copy (such as noise removal or speech enhancement).

The details of the model always have to be documented and publicly accessible. Anyone who accesses the digital resources has the right to access this information. Resources that “do not declare their objective and their limits, nor the procedures employed, nor the quality of the data to which the procedures have been applied, are low-profile products.” [10, p. 120] According to the same source, a low-profile product is one that “uses the potential of computer technology without reflection” [10, p. 121].

The expected output does not only consist of the digital set of preservation copies, but also of the audio resources, analyzed and catalogued by an experts team, ready for access from the final users. The entire process represents the preliminary step to the real philologist’s work, as mentioned a few paragraphs earlier. Preservation (which also includes the concept of restitution, because data that is stored not to be used again in the future is a waste of time and resources) consist in the preparation of the working materials (the preservation copies). A valid critical edition must base on diplomatic editions [10, p. 122] that meet the requirements of authoritativeness, accuracy and reliability. These three need to be “a primary concern in long-term preservation [...With physical documents,] trustworthiness was all wrapped up in the concept of authenticity so that an authentic document was also reliable and accurate. This is no longer true.” [4] Authenticity needs to be re-defined for electronic documents, because they can not “be preserved as [...] unchanged resources: we have only the ability to reproduce them” [14] and (un)intentional modifications introduced at some point of the files manipulation may be very difficult to backtrack. “Authenticity cannot be recognized as given, once and for ever, within a digital environment”, and it can only be “approached asymptotically” [15].

According to InterPARES 3³, authenticity refers to “the trustworthiness of a record that is what it purports to be, untampered with and uncorrupted: it must be based on its identity and integrity, and on the reliability of the records system in which it resides. Reliability is the trustworthiness of a record as a statement of fact: it must be based on the competence of its author, its completeness, and the controls on its creation. Accuracy is the correctness and precision of a record’s content: it must be based on the above, and on the controls on content recording and transmission” [4].

3.1 Support to Textual Criticism

“Virtually all parameters in the process [...] require intellectual, critical choices, interpretation, and manipulation.” [9]

What could be the digital equivalent of text comparison in the audio field? Traditionally, musicologists are trained in the study of musical scores. Only a few who specialized in the Twentieth century repertoire consider audio recordings a relevant documentary source for their studies – although the score (when existing) has always a powerful gravitational attraction. How can recordings on different tapes be effectively compared? Is it easier to do so with digital files? What are the sound parameters that are relevant and meaningful to a musicologist? Are there any? The list may go on, and it is clear that open issues in this field are definitely not only of technological nature. Answering these questions is the only way to enable the development of truly innovative methods and tools to assist or automate the musicologists’ work, and interesting answers can only be found in a truly inter-disciplinary approach.

“In postulating a typology of Electronic Philology, we must take into account the data, the procedures, and the results.” [7, p. 16] The majority of computational analysis of texts are quantitative (e.g. word count), therefore the lowest extent in which the computer can serve the philologist is by providing him/her with a great amount of data and by extracting other data from it. Any list of words, sorted in any order, can be a good example. Features such as the file duration or the signal average amplitude could be the audio equivalents.

These first, basic, useful results constitute a set from which secondary results can be obtained. In particular, secondary results can be selected from primary, using complex information retrieval systems and rich query languages that have been developed to exploit huge textual resources. An example is the list of selected words obtained from a whole list of words. Selecting all files with a maximum peak over −3 dB could be the audio equivalent. Finally, tertiary results are obtained from the selected (secondary), following an exact pattern. Human interpretation is crucial. And example for texts: using a concordance or an index to build a dictionary, or the results of the collatio to prepare a critical edition [7, p. 16].

Fig. 1.

The scheme summarizes the main steps involved in the process of preservation of audio documents according to the methodology adopted at the Centro di Sonologia Computazionale in Padova.

3.2 Towards a Philologically Informed Methodology for Preservation

IPEM at Ghent University and the Centro di Sonologia Computazionale (CSC) in Padova have been active in the field of experimental music since the 1960s and 1970s, and both own a fine collection of audio tapes with finished works, sketches, samples and interviews. Along the decades, the CSC has developed research in audio preservation and restoration, building on a strong scientific background in sound synthesis and electronic music. It houses a laboratory with the equipment required to create digital preservation copies (see [11, 12]) that meet the requirements of accuracy, reliability and authenticity presented in Sect. 3. A protocol for re-mediation (Fig. 1) and a controlled environment are the key to quality control along the workflow (for more details on the protocol see [16]). This goal is also achieved thanks to original open-source software developed on purpose at the CSC [12], as well as with the multidisciplinary approach that distinguishes the methodology for audio preservation of the CSC (mainly information engineering, music[ology] and chemistry). “The creation of digital objects has to meet the standards of the various disciplines involved, and [...] is a crucial part of humanities research. It is more than just preparation for research.” [1, p. 11].

The laboratory at CSC features two working stations equipped with Apple desktop machines. The main station is dedicated to the analog-to-digital transfers (A/D) and uses an A/D-D/A Converter (PRISM ADA-8XR) that supports 96 kHz/24 bit audio quality. Audio and video (Fig. 3) are acquired on separate machines, and the entire re-mediation system is connected for safety to uninterruptible power supplies (UPS). A professional STUDER A-810 with stereo heads is used to read most tapes 1/4 in. wide. Digitized audio is exported in a non-compressed open format and stored on three different locations; the metadata is automatically ingested into the database by a software developed by the authors [12]. The same software completes the preservation copy by processing the remaining data (checksums, images, video, etc.).

The laboratory also features a photographical working station for the production of the contextual information (photographical documentation of the source carrier). It was designed for short and frequent photographical sessions, maximizing the quality of the picture with the minimum effort to (i) adjust the positioning of the camera and its parameters for each session; and to (ii) transfer the new files to the desktop working station without dismantling the photographical setting nor moving too many things around. The functionality of the photographical working station was first based on the requirements reported by the Istituto Centrale per il Catalogo e la Documentazione (ICCD) and by the Italian Ministry of Culture in [17].

For the physical recovery of magnetic tapes, a precision incubator is used (Memmert INP 400). Thermal treatment, performed with the incubator, consists in applying consistent heat to the tapes over a specified period of time; it is aimed at reverting the effects of the so-called Soft Binder Syndrome - Sticky Shed Syndrome (SBS-SSS) [18]. Among the effects of this condition are sticking, squealing, and abnormal shedding of the magnetic coating. It should be noted that not all tapes are suitable for thermal treatment. In order to find alternate recovery methods, chemical analyses and experiments are currently being performed by the CSC in collaboration with the Department of Industrial Engineering – chemical sector, of the University of Padova [13].

Fig. 2.

The BilliArT experimental setup at IPEM at Ghent University, Belgium, in December 2015. The dark room aims to re-create the atmosphere of a billiard hall, at the same time favouring the detection of the special reflective balls by the motion capture system. The trajectories of the balls are processed in real-time by a software patch (Pure Data) and create a jazz-inspired sound composition.

The IPEM laboratory in Belgium represents the ideal environment to carry out research on how people engage with artistic installation and to observe their modes of interaction with technological systems. On-going research at IPEM involves bio-sensors and motion capture systems, allowing for quantitative measurements of parameters (mainly related to body movements and gestures) that relate to the aesthetic experience of interaction. The creation of “preservation (or archive) copies” of multimedia interactive installations requires the definition of paradigms and concepts to deal with the complexity of the technology involved in art making, as well as a deeper understanding of how interaction really happens and how it becomes a meaningful (aesthetic) experience for people. In order to get this understanding, the authors re-installed an artwork called BilliArT that was first presented in a public fair in Ghent in 2013 [19]. The installation is a dynamic system in which generative music and visual textures emerge from the interaction of the participants with a standard carom billiard table (Fig. 2). The importance of the experiment carried out at IPEM (December 2015, data analysis currently in process) transcends the specific results obtained form the data analysis⁴, because it is the first attempt to introduce a systematic approach into the field of installation art preservation. This field is basically unexplored and the role of academia is of paramount important in organizing and guiding the research activities. The complexity of multimedia interactive art amplifies the risks of falsifying the [documentation about the] original work, because the methods and the techniques to capture the experience have not been formalized yet. A reflection around the trustworthiness of the archival documentation for multimedia interactive installations can definitely benefit from what has been said about audio recordings, and in turn can contribute to devise better preservation models and strategies for non-artistic complex systems such as robots for automatic music expressive performance [20] and other computer-based systems involved in scientific research.

Fig. 3.

Test for video shooting. On the left, a correct setting of the camera: each frame is clear and the text is readable. On the right, a wrong setting of the camera. The aim is to detect physical defects of the tape (due to aging or joints) that may correspond to a specific noise in the audio signal. Audio is directly recorded from a secondary line-out of the STUDER A-810.

3.3 A Searchable Collection of Interactive Multimedia Installations

Current cataloguing standards provide that documents are classified by homogeneous types and, accordingly, that multimedia works are dismembered and their components grouped by category. Multimedia installations come as a multidimensional “assembly of artefacts” [21], i.e. they consist in the combination of several partial documents, mainly sonic/musical and visual documents representing intangible contents with a cultural and/or social signification. The documentary unity must be (temporarily) violated for cataloguing, and subsequently restored to re-create the aesthetic experience as a whole. This approach leads to a variety of information systems using different formats for data storage, and the low or absent interoperability among repositories makes the reconstruction of the documentary unity a problematic or impossible task.

The collaboration between the IPEM center at Ghent University and the CSC in Padova aims to provide a breakthrough solution for the problem of describing, classifying and preserving interactive multimedia installations. A complete overview of the state of the art in the extra-academic field of preservation (mainly led by large museums across the world) is beyond the scope of this article. There is a number of critical points in the current preservation practices, especially in the long-term and from the viewpoint of digital philology. One critical point that is being overlooked is the lack of a searchable database of interactive multimedia installations. In a world where searching texts, images and music by keyword (or occasionally by content) is a given fact, the impossibility to browse collections of interactive multimedia installations by characterising features (degree and mode of interaction, type of sensors involved, etc.) seems anachronistic. The on-going research at IPEM and CSC aims to define what these features may be, in order to achieve a formalization that will bring about a whole new way of grouping, approaching and studying interactive multimedia installations.

4 Conclusions

Much research conducted in the field of computer science is preparatory to other scientific fields, such as (ethno-)musicology, linguistics and anthropology, mainly with methods and tools (e.g., information retrieval, signal processing, data compression). In the field of musical cultural heritage and digital libraries, computer science is instrumental in performing the re-mediation of the audio documents and in managing the digital data produced during the re-mediation. Even more crucial is the help of computer science in modelling interactive multimedia installation art for archiving and preservation, since the complexity of these artworks challenges the existing approach applied in cataloguing systems. Most importantly, the role of academia is key in systematizing and organizing the research in the future of this field.

This article has proposed a reflection around the emerging discipline of digital philology. Traditional philology can contribute to the field of the digital humanities with concepts (e.g. authorship) and activities (e.g. textual criticism). This article has tried to point out that the challenges in this field are not only of technical and technological nature, a quite common misunderstanding especially in the non-academic world. The role of academia in leading the cultural shift needed to formalize and systematize the future of research in this field will never be stressed enough. Our cultural heritage is at stake, and the threat of those who only want to make a profit out of it is real.