Fusion of VR and AR Strategies for Immersion in Lost Historical-Architectural Heritage

Abstract

The Santa Cruz Monastery in Coimbra, of the Canons Regular of St. Augustine, founded in 1131, was one of Portugal’s main religious houses before the religious orders were extinguished in 1834. The acritical use of the building and its subdivision and attribution to various entities during the following years, contributed to the progressive destruction of important artistic and architectural legacies (Dias and Coutinho 2003; Craveiro 2011).

Today the former scale and richness of the monastic complex is difficult, if not impossible, to grasp, in the way that contemporary observers have a drastically changed perception of the original architectonic structure. At the same time, artistic pieces commissioned by the monastery are today scattered over a panoply of museums and other institutions. This was the main reason why we decided to establish a research project combining architecture and history of art with new technologies, more specifically with technologies related to Virtual and Augmented reality. Therefore, we became interested in the integration of these two emergent fields of development that gave rise to what has been designated as Mixed Reality (Milgram and Kishino 1994, pp. 1321–1329). In this way, tridimensional digital recreations of architectural and sculptural components are being produced to enhance the remaining context with photorealistic elements and data from other areas of knowledge, providing visual, acoustic and even tactile immersion.

1 Introduction and Background

During the late 19th and early 20th centuries, important parts of the Santa Cruz monastery, in downtown Coimbra, were destroyed. Among the most significant are the monastic façade, the porter’s cloister (replaced by the town hall building, 1876-79), the Renaissance dormitory (128 m long) and a prominent medieval/baroque belfry (Couto 2014).

In order to enable a renewed understanding of the erased past of the Santa Cruz monastery we decided to put up a research project proposal aimed at producing virtual reconstructions of this lost architectural heritage. This proposal was selected for funding in an open call, by FCT, Fundação para a Ciência e a Tecnologia, the Portuguese scientific and technological agency. The project we are currently developing relies on the work, information and expertise of various researchers from various fields of knowledge, such as art historians, photography historians, architects and computer scientists, who collaborate to interpret the physical state of the monastic complex in its mid-19th century state, before the main destructive actions took place (Fig. 1).

Fig. 1.

Façade of Santa Cruz Monastery, drawing by José Carlos Magne (detail), 1796. Source: Machado de Castro National Museum, Coimbra. // Laser scan of current facades of the Coimbra town hall building and Santa Cruz church. Source: Santa Cruz research project. //Tridimensional model of the previous façade of the Santa Cruz Monastery (before 1834). Source: Santa Cruz research project (3D reconstitution by Antonio Monteiro).

The virtual reconstructions of the lost architectural heritage we are producing, can be perceived through Virtual and Augmented Reality platforms and devices. These contents will be available in a prototype state for visitors of the studied sites. However, our prospect (given the technological state of the art of the dissemination platforms and devices) is a more immediate diffusion of the produced contents using smartphones. This almost universally adopted technology will allow visitors to download and view on the site, virtual contents related to the missing heritage, combined with subsisting architectural and sculptural elements.

2 Argument

We are now at the beginning of a new revolution defined by the way in which we assimilate knowledge. This latest accelerated transformation, based on the intimate connection between digital information and our perceptual abilities, connects us with realities that go far beyond our natural senses, giving way to new prolific fields as Augmented and Virtual reality, characterized by augmentations of cognitive functions that are anchored in our sensory system. These creative technologies provide an interactive experience in contexts where the elements of reality are nurtured with digital information, in order to emphasize certain aspects of contemporary or past realities. This connection between intrinsic and extrinsic elements of the virtual and real worlds enables computer systems to help us observe and think in line with the way our nervous system was developed. It also allows us to communicate symbolically in diverse cultural contexts.

It should also be taken into consideration that the cultural and creative industries (OECD designation) are recognized worldwide as one of the most important economic factors for growth and job creation. According to the TERA Consultants (2014), it is of vital importance to study and promote the uniqueness of cultural heritage, given that these industries represent approximately 5% of the European Union’s GDP. The spread of digitization and 3D representation, as well as the change associated with how people perceive reality, forces cultural and creative industries to develop new growth strategies.

One of the questions that many scholars have to address about 21^st century economics is the usefulness of people in a world where both physical and intellectual work can be provided by intelligent machines. It is very likely that these robots will do almost everything better and in greater numbers, questioning the efficacy of humans. Although there is much speculation, no one really knows what the job market will be like in half a century and what humans will be doing or what their existence will be like. It is almost impossible to ensure that the education we are promoting today will be valid within 20 or 30 years (Harari 2015). We can only suspect that the interaction between human beings and digital systems will be increasingly intimate. The creative industries associated with digital development, including biotechnology, will probably pose the greatest challenges and opportunities in the coming decades. It is possible that our physical, cognitive and emotional capacities will be transformed, creating profound changes in the way we perceive the world around us and ourselves.

As mentioned, in the several studies done so far, the intention was to create distinct contexts into a Mixed Reality environment, where the fusion of digital and analog elements can be sustained through visual, auditory and sensory outcomes. In other words, one attempts to supplant Turing’s proof for Virtual Reality. Renshaw et al. (2016, pp. 2113–2117) proposed the ground rules for the future development of a Turing Virtual Reality Test to pursue the goals of the imitation game created by Alan Turing in 1950. This challenge, initially focused on aspects related to artificial intelligence, currently promotes studies on the way humans conclude about the veracity of routine observations. Consequently, Turing’s proof for Virtual Reality was taken as a reference in this project in order to realize contents that reduce the observer’s experiential and contextual distance from the objects of study.

Observers undergoing the Turing test were primarily visitors of various interactive installations and workshops where digital versions of the human scale sculptures of the Last Supper ensemble by Hodart (executed between 1530 and 1534 and currently in the Machado de Castro National Museum in Coimbra) were abruptly inserted into architectural spaces unrelated either to their actual placing (the Museum) or to their primordial origin (the Santa Cruz monastery). Observers’ reactions were recorded to verify the momentary accomplishment of the Turing’s test for Virtual Reality. These experiences prepared the team for the research project’s main tasks. One of these tasks is to virtually place the Last Supper sculptural ensemble in its original setting, a small elevated space, now walled, that presided over the monastic refectory, which still exists today as an exhibition room.

These achievements were made possible through the elaboration and interaction of different processes of capture and analysis of architectural and sculptural elements. In the first case study, digital sculptures are integrated in their original context (the monastic refectory) captured stereoscopically. These sculptures were digitized at the Museum using a laser scanner that contains an integrated high definition photo camera. In addition, the digital sculptures have been scaled down to 1 to 20 and printed on a 3D printer to be tangibly tested on their interaction and on possible scale compositions of the scene (Figs. 2 and 3).

Fig. 2.

Photograph of three of Hodart’s sculptures (Saint Peter, Jesus Christ and Saint John) currently at the Machado de Castro National Museum. Source: Santa Cruz research project.

Fig. 3.

Primary exercise of virtual composition of Hodart’s Last Supper in its original 3D reconstructed context. Figures modeled on the basis of laser scans of the original statues currently in the Machado de Castro Museum. Source: Santa Cruz research project. // Accomplishment of the A2D2A (digital-to-analog-to-digital) process, very persistent in studies where analog elements inform digital representations that became physical models again to validate theoretical proposals. Arrangement tests made with 3D printed statues.

The way we develop Mixed Reality in our research project is not yet in real time. Nevertheless, all the components of mixed reality are there and can be experimented by the observers in the same way we will do it in a few years, with the new generation of Mixed Reality Glasses, or even with innovative smartphones capable to provide a virtual fusion of real time renders with imagery from two stereoscopic cameras.

To recreate the environments stereoscopically, we might have to take several spherical photographs according to the directions and places where we want to intensify the observer’s parallax experience. Another process that takes place in parallel is the 3D laser scanning of the spaces where we want to deploy the sculptures. Finally, the photographic captures of environments and the models based on laser scans, are creatively fused to produce high fidelity sensory immersions (Fig. 4).

Fig. 4.

Workshop “Dinner at the Table”. 7th eCAADe Regional International Symposium Aalborg, Denmark May 2nd 2019.

Among the main Mixed Reality contents envisioned within the scope of the project, most of them related with the insertion of several lost architectural elements of the Santa Cruz Monastery in the middle of the contemporary settings, we must highlight the cloister around the Manga fountain (a rare Renaissance architectural piece) along with the medieval/baroque belfry (demolished in 1935), which was visible from inside the cloister. Various versions of these architectural elements are being tested in workshops and exhibitions, where they are inserted in the most diverse contexts (Figs. 5, 6 and 7).

Fig. 5.

Graphical presentation of the workshop “A space and time travel based in AR+VR technologies: a renaissance fountain and a baroque bell-tower from Coimbra to FAUP” developed for the eCAADe/SIGRADI 2019 Conference. The final composition was constituted by a stereoscopically captured architectural element (the renaissance still standing Manga tempietto and fountain) and the 3D modelled 45 m high monastic baroque belfry (demolished in 1935), which were both abruptly mixed with Alvaro Siza’s Porto Faculty of Architecture buildings and surrounding environment.

In addition to introducing these architectural elements in different contexts in order to test their veracity and reliability, we have also begun new approaches of interaction with the observer, where the space exploration and interactivity play a relevant role. Unlike basic Augmented Reality systems, where the visual interaction with the real world is made by overlapping both virtual and real elements, modeling the most significant elements (based on 3D laser surveys) for the participant’s interaction allows us to introduce a semantic understanding of space into the computer. This process is now increasingly automated, to the point that some systems recognize in real time the hands of the VR system’s user. The current precision of these systems allows for the articulated reading of each finger and, therefore, the grabbing of objects and other haptic approaches.

Fig. 6.

Encircling the Manga tempietto and fountain with the Porto Faculty of Architecture buildings, designed by Alvaro Siza // Insertion of the medieval/baroque belfry form the Santa Cruz Monastery between two blocks of Alvaro Siza’s Porto Faculty of Architecture.

Fig. 7.

Workshop participants verifying the results in loco, using RV glasses and smartphone headset adapters. The real environment - Álvaro’s Siza FAUP building and gardens - is mixed with intertwined elements from Coimbra’s Santa Cruz Monastery, some of them transposed, others virtually recreated.

3 Forthcoming and Foreseeing

As an example, and using a much more pragmatic reference, we can perceive that one of the current difficulties encountered in Mixed Reality is the introduction of haptic sensations related to the Virtual Reality being observed. In our research project, the exposure of the observer to haptic sensations draws on the introduction of physical coincidences between what is visualized on VR devices and the physical space in which the visualization occurs. The simplest method is to ask the observer to seat in the same position in which the stereoscopic photographic survey and the laser scan were done. Likewise, other close elements such as walls, columns or tables, become elements of physical interaction.

Although our approach is much more conservative, taking advantage of the existing environment and of the possibilities of haptic sensations, we are aware that in the near future the introduction of virtual optical sensations can greatly help the immersion process in Mixed Reality. As an example, we can mention that recently a new form of haptic interaction was invented. There is still no expectation of when this technology will be used in commercial devices although prototypes denounce a new level of realism. According to an article recently published by the Soft Robotics Magazine, scientists at the Swiss Federal Institute of Technology of Lausanne in Switzerland (Harshal et al. 2019) developed a material with flexibility similar to that of the human skin that, when superimposed on a user’s body, simulates the sensation of touch in a more realistic way than previous technologies of haptic interaction. Pressure and frequency changes in the air pumping system are monitored by an electrode sensor that tracks the skin’s response to induced vibration and sends the data obtained to a microcontroller where haptic sensing can be regulated. This vibratory stimulation technology that allows to equalize the applied sensations could be the next step within Mixed Reality.

4 Latest Contributions

Observing the environment around us is a fundamental condition for feeling to be in a given environment. In addition, the sensation of being in a given environment is reinforced by the freedom of movement, interaction and exploration. The majority of current VR devices only allow for head rotation with a fixed point as its axis. Consequently, the rotation goes right, left, up or down. Although this technology allows some engagement, observers who attempt a translation realize the illusion and inaccuracy of the framework (Figs. 8, 9 and 10).

Fig. 8.

Exhibition of author’s project with Unreal 4.23 - a real time rendering platform that allows to explore the space interactively embracing time as a 4^th dimension to appreciate routes and paths. European Researchers’ Night held in around 300 cities simultaneously across Europe, September 27^th, 2019.

Fig. 9.

Freedom of movement, interaction and exploration with Unreal 4.23, a real time rendering engine. Sequence of images produced for the European Researchers’ Night (Miguel Alberto and Pedro Andrade).

Fig. 10.

Current photograph (2019) of the Manga fountain used to test the insertion of the three-dimensional model of the ancient cloister and medieval/baroque tower. Source: Santa Cruz research project. // Planning of a spherical image for virtual reality platforms with a photographic insertion test of the Manga fountain in the 3D digital model of the old cloister and medieval/baroque tower. Source: Santa Cruz research project.

5 Conclusion

Vernor Vinge is a mathematician who, in 1993 wrote an article where he predicted the appearance of a machine with a level of intelligence equal to that of a human being. This artificial intelligence could have, among other capabilities, a consciousness of its own. He called this phenomenon Singularity (Vinge 1993). Upon further investigation and going back in time, we can see that Vernor Vinge’s predictions are supported by Gordon Moore’s studies published in 1965, when the processing power exponential growth of computers was documented (Moore 1965). Up until today, Moore’s law has been verified, leading us to believe that in the coming decades, virtual reality simulations will be impossible to distinguish from reality itself. In a future technological context with these characteristics, tacit knowledge is likely to supplant explicit knowledge in all fields of research. On the other hand, as a symbolic and kinesthetic being that we are, the notion of reality itself may undergo major changes.

Envisioning future technological developments, we began the first experiments with real time rendering platforms that allowed us to explore the space interactively embracing the time as a 4^th dimension, letting observers to choose and appreciate routes and pathways by themselves. Therefore, we might say that the main value of Mixed Reality technologies applied to our research project relates to the integration of digital elements in our perception of the real world. More than a simple visualization of data, the sensation of immersion is acquired through physical interaction with the surrounding elements and surrounding acoustics. In this way, it is intended to create instruments capable of providing knowledge and wisdom (knowledge with emotions) around the lost architectural and cultural heritage.

This research may still yield other results in the near future. Along with the recreation of immersive experiments of the most emblematic architectural and artistic aspects of the Monastery of Santa Cruz before 1834, in the field of architectural archeology, we may go back further in time to the original Romanesque church and cloister, which were replaced in the early 16th century by the current church and cloister.