Hello and welcome to the NVIDIA Joint-Lab on Mixed Reality.

My name is Eugene Ch’ng, I’m a professor of Cultural Computing here at the University of Nottingham’s China campus. I am also director of the NVIDIA Joint-Lab on Mixed Reality, an NVIDIA Technology Center located here and supported by NVIDIA.

If you turn around and follow me, you can see that we have several tools available for VR research. We are a development center for VR and AR, Virtual Reality and Augmented Reality, and we deal with different areas of research, but in particular those related to cultural heritage: monuments, sites, objects, text and behaviors.

I will first introduce you to my laboratory. My speech focuses on the sharing of cultural heritage through VR technologies. I will therefore not only talk about VR, but also about the technologies that make it possible to create all sorts of VR and AR applications.

I am presently co-editor-in-chief of Presence: Teleoperators & Virtual Environments of the MIT Press Journal and for this Journal we accept articles related to the application, and art and science of VR and virtual environments. If you have original articles with good contents and that meet publication standards do send them in, they will be welcomed.

The vision of my laboratory concerns Virtual Reality and Augmented Reality, but within the spectrum between the two realities we focus on Mixed Reality, on the coexistence between the real and the virtual.

With a few exceptions, we build our machines so that they are suitable for the GPUs supported by NVIDIA. We have several professional tools that we use for processing big data, from social networks and VR applications. We have several funded projects, one of which is an “agent-based modeling” project modeling ancient ecosystems located in the North Sea submerged 10,000 years ago due to climate change. It is therefore an archaeological project which concerns cultural heritage. I also collaborate with the research group that has mapped the entire landscapes of Stonehenge in a digital format and with which we have discovered 17 new monuments. We also used the digital map, which we presented at the Royal Society Summer Science Exhibition 2015 to crowdsource people’s behaviors and to understand how they would use their surroundings for burying their dead relatives. The study was published in “Virtual Systems and Multimedia” conference two years ago. For other projects, visit the “Curious Travelers: Visualising Heritage” website. In China I have a funded project for building a 3D objects database of the entire Silk Road from both the land and the sea.

What is important is a global platform that is capable of reproducing cultural heritage. When reproducing, documenting or recording cultural heritage, we are not talking only about images, videos or 360° videos, but we increasingly refer to high quality three-dimensional images. 3D scanning techniques are becoming more widespread, less expensive and more accessible. In the past, laser scanning was very expensive and were not accessible to many laboratories or even museums, but photogrammetric techniques can use almost any digital cameras, so you can simply use a high-definition camera from a smartphone and get high quality results.

ReACH (Reproduction of Art and Cultural Heritage) is an initiative supported by UNESCO, led by the Victoria & Albert Museum in collaboration with the Peri Charitable Foundation. This initiative has reviewed and redrafted Henry Cole’s 1867 Convention on the Reproduction of Art. At the time, works of art were in fact reproduced through physical and analogue instruments. But now, 150 years have passed and a review was necessary. I was invited as one of their global consultant and an expert on digital heritage to contribute to the program which has taken me to Beijing for roundtable discussions, to London for the V&A ReACH declaration, to Abu Dhabi for the making of the technical policy. Just last week I gave a speech in Paris, at the UNESCO headquarters on the theme of sustainable digital heritage, on the challenges of storage and sharing. A book titled “Copy Culture” has been published containing several case studies and articles on the reproduction of art, of which a digital version is also available. The ReACH community, to which I belonged helps support individuals and organizations which do not have access to resources and knowledge for such works.

Why is cultural heritage so important? It is important because it allows us to preserve our identity, our social and national values. A plaque outside the Museum of Afghanistan states that: “A nation stays alive when its culture is alive”. The historian Nancy Dupree said that “a nation maintains its integrity only when its members preserve the values of its history and its heritage.”

Then there is also value and economic impact – Stonehenge visitors pay not only for the entrance fee, but also for the trip, the hotels, as well as various purchases. As you can see from the graph, the national cultural heritage contributes to a turnover of about 26.37 billion according to a 2008 study. In the past, about 10–20 years ago, digital technology has transformed documentation and analysis of cultural heritage, made significant progresses but focusing on sites that are very old. For example, Stonehenge has a history of thousands of years, or this other site at Geldeston, of which we have a video in which we see archaeologists pulling out an Iron Age stake which in the past lines the banks of a river which brings residences into the village. Apart from the physical labor there is also the need for laser scanning which accompanies the work. This is a video from 2011/2012 of which you are able see laser scans producing very high quality models. However, laser scanning is a very expensive activity which requires tools and expertise to process models. Things have changed with photogrammetry approaches.

After my visit to London, and as I was returning to China to participate in a TEDx talk, I downloaded data of the UNESCO World Heritage Sites and generated a chart demonstrating that the oldest site registered in the World Heritage list is millions of years old. There were several concentrated sites in the middle segment that are hundreds of thousands of years old and finally we have the youngest site at 29 years old – the Sydney Opera House. You can see that great world organizations and museums focus on that part of the heritage that is very dated and are ancient. But the problem of ancient sites is that the memories have disappeared, and there aren’t written documents, and therefore analyzing and interpreting these monuments are very difficult. Looking at this chart you can see that sites that are less than a hundred years old have live memories and I think these sites are important. Our laboratory focuses on sites that are still alive.

It could happen to you in Rome, even if your hometown is, for example, Bressanone. Returning to your hometown for the Christmas holidays, you would begin to think about the feelings you will have as you return home and the sense of warmth which may arise. Even looking at old photos will trigger memories that could be sweet or bitter. I therefore think that it is important in capturing living memories. When I look at this picture when I was around 3–4 years old – I and my brother sitting on the Mercedes-Benz, I was immediately transported back to that moment. I know that there is a tree here which emits a fragrant smell and it is about 7.00 pm in the evening. Going home there is my grandfather who falls asleep in front of the TV and snores very loudly, and so on. These images are therefore able to bring us into the spaces of our memories. I call all these things, the images, videos and objects “Anchors of memory” or “memory anchors”. Even physical structures are anchors of memory. When we access our past memories, we get a good feeling and this is important because they represent our roots and our identity. Yet all these memory anchors reside only in our mind. When we share stories, our friends and family can only imagine them. We could tell a good story to bring them into this memory, but to them they are only imagination, they will never be in the same memory space.

What virtual reality allows us to do is not only to share stories, but also to bring the audience, the members of our family into our memory spaces. Virtual Reality has been in development for a long time. It goes back to Ivan Sutherland who had a vision that one day we would have had a window into the virtual world and the images would have improved to such an extent as to seem real. Nowadays in the 21st century, we have achieved his VR prediction.

Virtual Reality is based on the “3 Is”: Immersion, Interaction and Imagination. These three collectively simulate the sense of presence. As compared to 10–20 years ago, Virtual Reality has become accessible, inexpensive and powerful. We can not only record the movement of our head, but also our orientation and our position, including the position of our body if we use the controllers. We have controllers that allow us to interact with the surrounding virtual environment.

Virtual Reality has become a very powerful tool and it is how it is used for creating value. In our laboratory we do a lot of research, in particular studies on users. Several articles will be published soon in this regard. We essentially look at human behavior, the sense of presence, nostalgia, memory and imagination. Presence can be induced in different ways and we simulate virtual ‘reality’ using different techniques. From the point of view of VR devices, they are now very accessible – for £1,500 you can own a good workstation and probably some of the tools shown in my slides.

The video shows a rendering in real-time of a character that is real (in 3D), not just a 2D image. All the details such as the translucency of the skin are entirely simulated with extremely powerful graphics cards. Looking at this other scene, no one would ever imagine that it is a real-time simulation in which one can actually walk inside. If we can simulate something so real, can we also store our memories in that space? This is the question we ask. If we are able to walk through this scene, apart from the feeling of the dampness on the floor, we have the sensation of being actually inside it. This shows that reality can be simulated with Virtual Reality.

In the video, I was at a site in Japan and capturing a scene using only my iPhone and photogrammetry techniques that have become very common among the virtual environments community, or to Hawaii or in China using a drone for capturing landscapes. I found that I am able to reconstruct reality in virtual reality. It takes only a few minutes to capture a series of images for 3D reconstruction. This is the digitization of spaces, structures, monuments and objects of our memories. We also need to visualize them and this requires much more than simply acquiring data. Visualization allows us to add value to the data we have acquired.

I have begun acquiring several memory anchors. I was in a beach in Australia with my children a year ago and whilst they were playing with sand I decided that maybe I should capture that moment so that I can later share it with the other family members. With only photos one could only imagine the location, and one could not be at the same place we were at where we have taken the photos. At the NVIDIA Joint-Lab on Mixed Reality we have several tools that can allow us, through the networks to bring more people into the same place. Beyond the photos and videos that we were able to share with friends and relatives, we are now able to further access the spaces in which our memories reside with VR.

20 to 30 years from now, Virtual Reality will probably be the main technology for sharing memories. Today we share photos and videos through social media, but in the future we will be able to share our memories through 3D spaces.

We recently visited an old village in Ningbo, China where, with a drone we acquired 3D data of the entire village. Here, you can see a little house in which a couple lives, the woman who is blind walks around the house, cuts vegetables, cooks, brought condiments to the table, and etc. You can observe the quality of the data acquisition, which takes time to process. Once completed, the virtual environment allows you to interact with the objects. This is a scene outside the kitchen. We see an old tofu mold, with which the 80 years old couple has prepared tofu for us. In this other scene as you can see, one can also build 3D models and not only acquire them through 3D capture that which has ceased to exist. See also an old basket here.

Creating Virtual Reality from the reality around us is a simple process. We took several images and using photogrammetry that I have mentioned before, the 3D scene is created. This is followed by a processing phase. If it is not a particularly complex interactive environment, no VR programming will be necessary. After the simple process it will be possible to share the spaces created for the future.

This is the tofu maker scene with my PhD student Shengdan Cai exploring it. Both for her, myself and my other students, entering that scene brought the same sensation experienced when we were actually in that village. With the experience gained through this particular project, we think therefore that memories can be inserted and transmitted with Virtual Reality. All objects in the scene have been acquired in 3D, but the environment and some parts of the scene are rebuilt, with visual effects added. Thus a scene is reconstructed which can give particular sensations. Here, I took a plate with coins on it and here I dropped the coins. In this other scene we combined the Oculus Rift with the controllers, here are some museum workers supporting our work.

Here, we have a much more elaborate project concerning the creation of an entire ancient village. In our portfolio we currently have a project combining Virtual Reality with Augmented Reality.

This concludes and summarizes my speech. I think that in the future there will be a shorter distance between the real and the virtual thanks to the ability to share 3D contents and environments, within which it will be possible to enter virtual spaces rather than simple 2D photos and videos in social media. We have looked at the value of cultural heritage and how it contributed to the economy, but also to our identity, our roots, our history and our national values. I have also talked about the importance of acquiring memories, especially living ones contained within structures, architectures, monuments and objects before they are lost forever. These are living memories within a hundred years old, a range of time which are perhaps uninteresting for museums and global organizations.

Virtual Reality is accessible today, accompanied by technologies that can allow us to acquire and share our “anchors of memory”, which can take us back to where we were many years ago. I think that sharing 3D objects and environments will become as common as photos and videos in the future. This is my prediction. The ReACH Initiative, led by the Victoria & Albert Museum, which I am involved in, is a global platform that contributes to this possibility through the expertise and technical documents that people can access and learn from.

With this I conclude my speech. My final sentence is: In the future, the past will be as accessible as the present, thanks to Virtual Reality.