AR for Virtual Restoration

Abstract

Among the possibilities offered by Augmented Reality is its potential use to give new life to works of art in museums, through the implementation of integration or completion in an operation that can be called virtual restoration. Without modifying the works of art in their current state of preservation in order to maintain their evocative power and, in particular, without invasive interventions on the signs left by time, through digital visualisation tools it is possible to provide users with a new way of re-reading objects and telling stories around them. The contribution aims to present the results of virtual restoration operations carried out on some statuary works in the collection of the Museum of the Archaeological Park of Ostia Antica in Rome, where innovative technologies are compared with the immense Heritage of the Museum, animating the exhibition itinerary by offering new points of view of the works through the creation of an interactive AR application.

1 Introduction

Augmented Reality, as is well known, makes it possible to enrich, on a sensorial perception and cognitive level, a wealth of information conveyed by a tool (such as a tablet or smartphone) within a given context, in particular that of a museum exhibition. The physical boundary between the user and the surrounding space becomes a frame with increasingly permeable boundaries, allowing a continuous exchange of information between them. In fact, it is the dissemination of information itself that makes use of new potentials that are enriched in time and space, being strongly connected to the examined Cultural Heritage, in a perspective of continuous evolution [1, 2].

Analysing the ways of fruition of the traditional museum, it is evident how the visit can find its raison d'être in the moment in which it is carried out by following a path predetermined by the curator of the event. Conceptually, the museum, as a container circumscribed to a well-defined physical space, appears limited to rare possibilities of personalization by the user. Therefore, the approach to the works appears to be reactive, guided exclusively by the established sequence of the works, which, with the exception of brief informative summaries describing their origin, give the user little information about the very essence of the artwork. In contrast to this approach, the contemporary museum seeks to break down these barriers, subverting real space, proposing more interactive and participatory fruition solutions, transforming traditional physical limits into permeable filters capable of guaranteeing articulated and customizable knowledge paths. In this type of context, it is therefore evident how the use of tools capable of favoring visitation is desirable, almost necessary, having as a nodal point the centrality of the role of the user with respect to the fruition of the artwork [3]. A museum visit organized according to a proactive approach to Cultural Heritage cannot but be equipped with all the knowledge aids capable of narrating the story behind the exhibits. This need appears even more relevant when considering archaeological artifacts, which, despite their beauty, require a necessary contextualization in order to be correctly assimilated by users [4]. Therefore, from this point of view, it is evident how augmented reality can be an important aid that can give new life to the artworks through the creation of a narrative, which becomes dynamic and transforms the user into a director of his or her own visit.

In this context, the contribution analyses one of the possible ways to implement this process of interactive dynamic fruition, integrating the visit process of the physical work, with new representations of it, in an attempt to integrate history and knowledge, showing what time has defaced. Going beyond the cultural limits of a direct action to be carried out digitally on the original artifacts, the contribution proposes an integration model of selected works, in particular through the virtual construction of models to complete the gaps present, in an action that can be defined as virtual restoration [5]. It is necessary to clarify what the concept of virtual restoration means: i.e. an opportunity to develop and then examine different intervention hypotheses with a focus on shared experimentation, a valuable analysis opportunity for experts in the field of cultural heritage (architects, archaeologists, restorers, art historians). Among the basic objectives of the activity is undoubtedly the recovery of the formal unity of the object studied, and its quality of historical testimony, since they are often silent works, with the aim of having, at a perceptive level, a correct interpretation of them as they originally appeared, starting from the systematic study of the traces still present; a further objective is the reinsertion of the object within a broader narrative, through the creation of dynamic paths that place the work within its historical context [6]. The realization of these objectives takes place in a digital environment, without in any way modifying the artworks in their current state of preservation, in order to keep their evocative power unchanged and, in particular, without making invasive interventions on the signs left by time.

In this research framework, aimed in particular at sensitively involving users in the field of Cultural Heritage fruition, bringing the user emotionally closer [7] and amplifying the experience [8], AR constitutes a privileged tool. Emotion is generated by the meaning we give to reality: through the new technologies, it is possible to transfer the sensations we feel into the digital world, actuating in the user a sense of wonderment that transcends his or her classic mental schemes, realising a greater emotional involvement, thus enabling an almost natural interaction between human being and digital content. It is precisely this experiential and emotional approach that guarantees an improvement in the enjoyment of the Cultural Heritage under investigation. The interaction between museum, collection and visitor redefines the museum space, placing the user at the center of the experience.

2 Case Study

Within this theoretical framework is the OSTIA 3D Project, which proposes new digitisation solutions for the protection and enhancement of cultural heritage in the field of archaeology. The project includes several case studies, declined at different scales, and making use of technological solutions built on the basis of specific objectives. The research is carried out on the works kept in the Ostia Antica Archaeological Park, in particular in the Antiquarium Hall, and sees the active participation of several members of the Park's staff. The contribution aims to present the results of virtual restoration operations carried out on some works of ancient statuary in the collection of the Museum of the Archaeological Park of Ostia Antica in Rome, where innovative technologies are confronted with the immense Heritage of the Museum, with the aim of animating the exhibition itinerary and offering new points of view of the works through the creation of an interactive AR application. Specifically, the study activity proposed here, while analysing various works of statuary on a medium and small scale, focuses on the integration of a Head of Augustus datable to the first century BC (Fig. 1).

Fig. 1

On the left two pictures of the Head of Augustus during the acquisition phase (Image: F. Porfiri, L. J. Senatore). On the right the Augustus of Prima Porta, which constitutes the main element of comparison (Image: Joel Bellyiure, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=129823838)

The reasons for this choice, in addition to the historical importance of the artefact, lie in its state of preservation. The Head of Augustus, in fact, is in an excellent state of preservation, except for the presence of two important lesions: the first affects the entire facial area and part of the chin, and the second is found in the area of the nose. Both lesions, particularly the one on the nose, disfigure the face of the character depicted and contribute to making his identity difficult to recognize. It will be precisely the digital elimination of the disfigurement on the face and the integration aimed at completing the missing element that will be the subject of the study, aimed at restoring formal unity to the work, i.e. to allow, in parallel with the vision in its real version, a reconstructed version based on historical sources and where, with a few interactive elements, it will be possible to reveal the story behind the artwork [9].

The process carried out so far has a prototype character, but the project is currently being developed to prepare the final model that will find its place in the new museum display next to the original.

3 Data Acquisition Workflow

In order to prepare the necessary model for the construction of the AR application, a specific operational workflow was put in place to obtain scientifically reliable data. The workflow includes well-structured work steps shared by the experts in the field. The following steps were then implemented:

• Study of the artworks under examination, aimed at identifying and classifying the object of study.

• Data acquisition (both of the object of study and of the artwork taken as reference for the reconstruction of the additions).

• Data processing for real time use within an AR application.

• AR application and communication construction.

The study was conducted under the supervision and with the support of the Park's archaeologists. The analysis of the work showed that it can be traced back to the first century B.C., thus coeval with the Augustus of Prima Porta, which constitutes the main element of comparison. The dimensions of the object of study, the Head of Augustus, are approximately 30 × 20 × 20 cm. For the data capture phase, it was decided to use different acquisition methodologies, necessary to validate the metric reliability and to verify the best solution capable of returning surfaces with less noise, to be subsequently subjected to the modelling phase, and the low-res transformation phase. Considering the use of data for interactive applications, and therefore greatly simplified in order to reduce its weight in terms of bytes, the opportunity offered by the research made it possible to verify some of the most commonly used techniques, and in particular those based on the logic of Structure from motion and those based on the use of LIDAR systems.

Two specific data acquisition sets were therefore developed:

• The first aimed at capturing photographs to be used for the construction of models on the basis of Sfm algorithms.

• The second aimed at capturing geometric and RBG data with Structured Light scanner.

For the realisation of the photographic images, with the aim of having maximum coverage and chromatic detail of the surveyed surfaces, a photographic set was organised trying to limit the presence of direct light to a minimum. With a series of opaque panels, the presence of direct reflections on the work was eliminated. Considering the illumination of the environment, realised with skylights, the positioning of artificial completion lights was not necessary. Once the photographic set was created, the Head was photographed with a Full Frame camera with a 26 mpx sensor and a 50 mm fixed focal length lens. The good ambient lighting made it possible to take shots with iso 100 (native), focal length f 8 and shutter speed 1/250 s. Given the shutter speed, it was possible to use the camera without the aid of a tripod, thus limiting the presence of micro-blur to a minimum. A total of 73 shots were required to cover the entire head area, ensuring an overlap of approximately 50%.

In order to metrically consolidate the model generated via SFM, we proceeded by inserting a number of targets into the photographic set, which were collimated with a total station (Leica, TCR 805) by setting the station points on a closed polygonal, to check the reliability of the acquired data. Once the data had been acquired, the models were processed using two different Sfm software, specifically: Zephir and Metashape. For both software, the different operational steps (sparse cloud—dense cloud—surface generation) were carried out at the highest quality level. The two elaborations allowed the construction of two models of good density both at the numerical level of the point cloud and at the geometric level of the mesh. The surface analysis revealed the fact that, although they were metrically comparable with insignificant deviations (standard deviation calculated over the entire surface of approximately 1.5 mm), the presence of noise, particularly in areas where the surface of the original was smooth, was particularly evident. In order to solve this specific problem, the models were processed with specific software through smoothing operations with obvious positive effects on the quality of the surface and negative effects on the metric validity of the model itself.

For the structured light laser acquisition, a dedicated set was again set up. The scanner used, Scantech iReal, was set to acquire a mesh of points of 0.2 × 0.2 mm, with the smallest possible distance from the instrument. The acquisition covered both the high-detail geometry and the colour aspect of the work. Particular attention was paid to all those areas that would later have to be integrated in order to ensure the best possible quality of the geometric data. Once the point cloud was acquired, it was processed to create a mesh surface that was subsequently textured with the RGB data acquired by the instrument. The analysis of the surface showed good quality for both the originally smooth areas and those with rough surfaces (Fig. 2).

Fig. 2

Image of the Head of Augustus, textured mesh model obtained from structured light scanner acquisition and orthogonal projection of the same with metric scale (Modelling: F. Porfiri, L. J. Senatore)

A critical comparison between the different surfaces obtained, realised with Sfm and Lidar techniques, showed that: while on a geometric level the differences are essentially negligible, the quality of the surface realised with Laser acquisition does not in fact require manual intervention to eliminate noise. For this reason, it was decided to set the model realised with Structured Light Laser as the reference model.

Using the same technology, in addition to the Head of Augustus, it was decided temporarily to acquire the nose of a statue preserved in the Park, which was formally compatible with the subsequent integration process. The instrumental acquisition settings were kept the same as those used for the study object, again obtaining a surface of high geometric definition and excellent surface quality.

4 Data Processing Workflow

Once the two models had been obtained, both of the original and of the work used as a reference for the integration of the missing parts, we proceeded to the actual data integration operation, using different techniques: regarding the deep lesion that crosses transversally the face and part of the chin, the missing portion of the object was reconstructed by working on the geometry of the statue's surface, going on to integrate the missing parts through a digital sculpture operation. In this case, the geometric analysis of the lesion edges constituted the main reference, and the sculptural operation was limited to reconnecting the edges of the surface discontinuity, keeping the curvature of the two portions unchanged. The modelling and curvature control tools are used to verify the formal correctness of the entire process, highlighting any sudden changes in curvature, which would have rendered the intervention unrealistic and geometrically incorrect (Fig. 3). This operation involved both the facial area, where the surface appears smooth, and also the area behind the hair, by locally analysing the surface patterns in order to ensure a new surface compatible with the surrounding areas.

Fig. 3

The process of integration of the missing part on the three-dimensional model of Augustus’ Head (Modelling: F. Porfiri, L. J. Senatore)

With regard to the integration of the missing element, the nose, the operation required further processing of the head chosen for the typological comparison, i.e., the extraction of the nose area from the acquired surface. Once this last operation had been carried out, the surfaces were put into relation, initially by superimposing the geometries and subsequently identifying the contact points necessary for the correct positioning of the nose portion on the Head of Augustus.

The integration process was carried out by bringing together a first phase of analysis and intervention on a visual and empirical basis with a supporting geometric analysis.

Going into the details of the procedure used, the original model with lesions and gaps was subjected to the following workflow:

• Restoration of major discontinuities using a digital sculpting process.

• Integration of missing parts with digital portions acquired from other statues.

• Verifying the geometric correctness of the work carried out on a geometric basis and in particular by verifying curvatures.

The first phase, the restoration of the surfaces, was carried out using digital sculpting software (Meshmixer) by acting directly on the mesh surface of the statue, which underwent numerical simplification following acquisition, in order to make the work more agile for the available computing power. The elimination of a considerable number of meshes at the preliminary phase, and not at the end of the operation, took into consideration the specific objective of the project (application in AR) and above all the possibility of considerably reducing (approximately 50%) the meshes produced after the acquisition, without the result in any way affecting the understanding of the object and its component surfaces. Acting with modifiers capable of filling in the gaps, reconstructing the traces left by the various lacunae, an empirical approach was taken, attempting to reconstruct the shape of the face and the missing portions of hair where, in the original statue, an obvious fracture is visible.

In the phase of integrating the missing parts, the nose was reconstructed. In this case, as it was not possible to have access to a statue with the same iconography, it was decided to select, from the statues in the Park's deposits, one that corresponded proportionally to the original statue and, based on the study of images of the iconography of the original, that had a nose similar in type and size. Once the statue with the required characteristics had been identified, it was scanned with particular attention to the nose. The acquisition was carried out with a Scantech iReal 2S HandHeld Laser Scanner with a scanning step of 0.2 × 0.2 mm. Once the point cloud had been obtained and the mesh created, it was processed by going on to identify, through a verification of dimensional compatibility with the missing portion of the nose, the few points characterising the nose, still present on the original. Having carried out various operations concerning the measuring of the component portion, the boundary contour of the fractures on the original was identified, which was used to cut, only for the necessary portion, the face area of the reference statue. Once this operation had been carried out, the result was optimised and some excess parts eliminated to obtain a model composed of two elements: the missing portion of the nose and the face itself. Once this phase was completed, it was necessary to reunite the two models by welding them together to recreate a single continuous and coherent surface. To carry out this operation, particular attention was paid to the points of junction between the surfaces which, once identified, had to be regularised and made coherent with an organic subject, i.e., without obvious edges or points of discontinuity. This was also performed through a digital sculpting operation by smoothing the ridges formed in the merging operation.

After the construction phase of the new face configuration had been completed, the surface was analysed in order to identify potential errors of a geometric nature and, in particular, to identify areas where it was necessary to focus on the elimination of ridges. The mesh surface was then exported to a Reverse Engineering software (Geomagic Design X) for the necessary verifications. In this case, too, the analysis was carried out in two ways, basing it first directly on the mesh surface and then, through a reverse modelling process, on the Nurbs transposition of the mesh surface for further verification. With regard to mesh analysis, the surface was analysed using visual verification tools with the creation of environment maps (Neon) and Zebra maps, and it was checked in detail how these maps rendered the surface. This type of solution, and in particular the Neon environment map, is able to show with a good approximation how the simulated effects of virtual refraction on the object change, offering an agile and effective tool for identifying the possible presence of discontinuities or ridges on the surface (Fig. 4). Once the result had been empirically verified, the mesh surface was reverse engineered to create a Nurbs surface in order to also carry out analyses relating to the Gaussian Curvature of the surface. In fact, the continuous Nurbs surface can be subjected to a Gaussian curvature analysis, returning a result that shows the punctual trend of the surface, i.e. whether the curvature for different points on the surface is positive, negative or zero. This type of analysis appears very useful in particular for the detection of sudden and continuous changes in the surface, such as the presence of ripples that are not compatible with the object of study but, above all, not evident from a simple visual inspection of the digital object (Fig. 4).

Fig. 4

a On the left: the surface analysis with visual verification tools: the creation of Zebra maps and environment maps (Neon); b On the right: the reverse engineering process: the creation of a Nurbs surface that can be subjected to a Gaussian curvature analysis (Modelling: F. Porfiri, L. J. Senatore)

After this analysis had been carried out and it had been verified that the result that was visually acceptable was also geometrically acceptable, it was possible to create the file in a format compatible with the AR software (Fbx) for the development of the application. Considering the approach used, it was possible to keep the models separate and thus allow for a segmentation that was later made evident in the augmented reality application, within which it is possible to activate the vision of a model where the parts added compared to the original ones appear easily recognizable. Considering some problems of a logistical nature, namely the fact that there is no statue of the same period at the Ostia Park to be used for the acquisition of the portions of the surface to be used for integration, the experimentation carried out here was purely prototypical in nature, but contributed to the definition of the operational protocol.

In view of the excellent results obtained from the application of the workflow (Fig. 5), activities are currently underway for the acquisition of the Statue of Augustus of Prima Porta, preserved in the National Roman Museum in Rome, which constitutes the reference model for the iconography of the case study. The result of the acquisition process will be subjected to the same workflow in the future in order to be able to include the experimentation in the communication program of the museum visit.

Fig. 5

Comparison between the original three-dimensional model and its reconstruction after the virtual restoration (Modelling: F.Porfiri, L. J. Senatore)

Finally, a brief note must be made regarding the appropriateness of verifying the difference between the two models, point cloud and digitally restored object. In view of the objective, although easily achievable, it was decided not to carry out verifications to quantitatively establish the standard deviation of the two models. The choice was due in particular to the fact that this information could be misleading with respect to the quality of the model obtained by virtue of its use within an AR application.

In fact, precisely due to the communicative aspect of the model created, and having operated exclusively in a punctual manner in the areas where the lesions were present, without in any way intervening on the non-damaged areas, it was decided to prioritise the image returned by the model that was as realistic as possible and capable of returning a coherent image of a human face [10].

5 The Application of AR

For the construction of the AR application, the open-source software Unity was chosen. The models, of the current state, the state with additions and the segmented model with the added areas highlighted, were loaded into the AR platform and formed the basis for the entire interactive experience, aimed at narrating the work.

The 3 models were classified and linked to buttons to make them appear on the viewer alternately, following the interaction with the visitor user. Once the models were set up, a Target image was created, using the logo of the Ostia Antica Park, to activate the visualisation of the first model. By clicking on the buttons on the screen, each user can access the documentation used as the basis of the reconstruction, comparing the reality according to his or her own purposes, integrating the visit in real time with new information, guaranteeing a better understanding of the perceptive aspect of the work. Through the standard visualisation functions, from the moment the chosen Cultural Heritage is visited, each model (original—reconstructed) appears on the screen of the device in the configuration recalled by the user.

The visualisation not only allows a dynamic vision of the digital object, making use of the gyroscope of the viewing device, but can also be managed by the user who, with his or her touch, is able to interact with the digital object to make it rotate in the digital space, thus visually investigating the element in every detail that constitutes it (Fig. 6).

Fig. 6

Below are the different steps of interaction between the visitor and the artwork: the original model, the reconstructed model and the information linked to it (Image: F. Porfiri, L. Senatore)

The interaction between the artwork and its spectator becomes active and participatory, thus more emotionally involving for the viewer. In addition to visualisation, particular attention was paid to narration using AR tools. In order to improve knowledge of the artefact, a number of buttons have been inserted which, once touched by the user, can refer to different types of information such as text, reference images, videos, functional to a greater understanding of the work. Among these, in a prototype version, the app allows the user to view an image of the iconographic reference, the Augustus of Prima Porta, with some data on its current location, as well as textual data to allow the user to delve into the story behind the object.

6 Conclusions

In conclusion, it is possible to assert that AR within a museum setting enhances the concept of democratising art, redefining the museum space, allowing each visitor to personally interact with the work at 360 degrees, both in the physical and conceptual sense of the term [11]. The visitor can choose what and how to get to know a work within a multiple exhibition space, grasping even intangible information, with the aim of enriching his or her experience and cultural background. In this way, a new way of exchange between the archaeological cultural asset and the general public is offered, where knowledge of a work is not precluded to the uninitiated. The use of AR, declined in its meaning as a tool for the virtual restoration of the work, can be functional precisely in bringing users closer to archaeological heritage that is often not adequately narrated. In fact, it is capable of explaining in a visual and immediately comprehensible manner all that is hidden behind a work, making the user participate in its history and the various transformations it has undergone over time [12].

Virtual restoration is a non-invasive intervention and it returns the user's perception of the original object, any additional information comes from a continuous dialogue and exchange with experts, it is a critical interpretation of the object aimed at its communication. In this context, AR, as a privileged tool for the creation of a narrative experience, constitutes its appropriate fulfilment.