Virtual Cinematic Heritage for the Lost Singaporean Film Pontianak (1957)

Abstract

In 1957, Cathay-Keris Studio in Singapore released “Pontianak”, the first of a hugely popular series of horror films featuring the female vampiric ghost based on Malay folk mythology. The visual transformation of the lead actress Maria Mernado into the Pontianak creature amazed the audience in cinemas (Mustafar, 2012) [1]. The Pontianak trilogy by Cathay-Keris was ground-breaking for the genre at that time and registered several firsts in Singapore: Pontianak was the first Malay film to be dubbed into Mandarin, while Sumpah Pontianak is the first local CinemaScope (widescreen) film. The 1957 film is also considered the first depiction of the Pontianak in a film [2]. Despite their relatively wide distribution in 1957 both films are believed to be lost, with no prints or copies surviving (Barnard, 2011) [3]. Film Heritage in the conventional way of ‘restoration’ being impossible, this project investigates the novel approach of creating a Virtual Cinematic Heritage application. In this paper, we outline our current work in progress from historical film research to recreating a scene from the lost film as a Virtual Reality experience.

1 Introduction

Cinematographic works are an essential component of our Cultural Heritage and therefore deserve full protection. Film or Cinematic Heritage aims to preserve tangible cultural property in the form of film copies, photos and manuscripts. In the widest sense, film preservation assures that a movie will continue to exist in as close to its original form as possible. A film for which not one single copy can be preserved is considered lost, much like a historic site, which has been fully destroyed.

Virtual Cultural Heritage explores techniques such as laser scanning, CGI animation, 360˚ imagery and photogrammetry, aiming to create photorealistic and accurate representations of historic environments (Zara 2004) [4]. The goal of being as realistic as possible has not been fully accomplished, but considering the rate of improvement, virtual environments and augmented vision will eventually become indistinguishable from reality. Inhabiting such photorealistic virtual environments with realistic-appearing actors will enable Virtual Heritage to attract a broader audience beyond researchers, creating accessible reconstructions for museum installations and edutainment. The term ‘walk-in movie’ could be used to describe a Virtual Reality environment, in which the viewer can freely explore, with a headset in a room-scale VR setup, while a narrative with actors unfolds around them. In such a scenario, the audience of a Virtual Heritage application could be enabled not only to virtually encounter a historic site but also an important historic event happening at that site.

Considering the fundamental role of visual components in Cultural Heritage, Virtual Reality is particularly suitable (Bocci et al. 2010) [5], and extensive research has been accomplished on the application, but the implication is, that for the benefit of Cultural Heritage, developments in Virtual Reality filmmaking are highly relevant aspects to explore. Bogdanovych et al. (2010) [6] argue that populating a Virtual Heritage site with virtual agents could bring heritage preservation to a new level but found that previous attempts were unsatisfying as they “are not involved into historically authentic interactions” and “that the agents are not behaving in a way that the relevant cultural knowledge is authentically presented to the human observer”. The MIRALab mixed-reality project of Magnenat-Thalmann and Papagiannakis experiments with “animated characters acting a storytelling drama on the site of ancient Pompeii” [7] and they conclude that “dramaturgical notions” could help to “develop an exciting edutainment medium” from their Virtual Heritage application. Researchers are aware that the audience’s narrative involvement is relevant for a successful Virtual Heritage application.

The research project aims to investigate if approaches used in Virtual Heritage, such as the creation of virtual environments of historic sites, and developments in the field of Virtual Reality filmmaking, such as inhabiting virtual environments with actors and stories, can be utilised to create Virtual Cinematic Heritage for films, which are considered lost and for which Film Heritage, in form of film preservation, is not an option.

As a case study, the research has begun the process of creating a Virtual Cinematic Heritage application for the historic Malayan Pontianak films produced in Singapore, originating in 1957 and originally starring Maria Menado as the titular supernatural creature from the oral folklore of Malaya. The first two films in this hugely popular trilogy – Pontianak and Dendam Pontianak, both produced and released by the Cathay-Keris Studio in 1957, directed by B.N. Rao, and starring Maria Menado, are believed to be “lost films” despite their relatively wide distribution in 1957.

According to Barnard (2011), the first film was so well received that it “even appealed to the Indian and Chinese communities – a first for a Malay film”. Released in Malay, then later in a Chinese language version, Pontianak was screened in cinemas for over a month, an unusual occurrence for a Malay film at the time. Adrian Yuen Lee (2015) [8] argues, regarding the success of the films, that “the already existing beliefs, mythologies and superstition has allowed Malaysians and Singaporeans to develop a deep sense of ‘cultural verisimilitude’ that invokes a deep sense of plausibility, motivation, justification and belief due to familiarity with the pontianak in these films”.

The choice of the Cathay-Keris Pontianak trilogy as a case study for the Virtual Cinematic Heritage application requires the project to conduct research into the films’ history and based on those findings - to investigate the recreating of key scenes of the film as a virtual representation. While both investigations are ongoing, the project has created new knowledge by unearthing previously uncollated details of the lost films. In this paper, we share our preliminary findings and outline our process of creating a virtual reality experience for the lost film (Fig 1).

Fig. 1.

Pontianak film poster © 1957, Cathay-Keris

2 The Pontianak Film(s)

2.1 Background - Pontianak Folklore

The traditional folk culture of Malayan Archipelago is rich in superstition and belief in mythical beings, which persist in the popular imagination until today. They were first ‘recorded’ in English by English Anthropologist Walter Skeat at the end of the 19th century in his seminal volume - Malay Magic, first published in 1900.

In this work, Skeat describes a female figure called the Langsuyar or Langsuir, a woman who has died in childbirth who returns from death as an owl-like “flying demon” and a “woman of dazzling beauty” who later transforms in grief into a Pontianak, a “ghostly form” which is still feminine, wearing a robe, brandishing long nails, long dark hair, and a gaping hole in the back of her neck. This creature sucks the blood of children or pregnant women, which is why it has often viewed as analogous to the Western Vampire. Skeat goes on to explain that if captured the Pontianak can be made “tame and indistinguishable from an ordinary woman” (by stuffing the neck-hole with her hair and cutting her nails) (Skeat 1900) [9].

Skeat notes that the word “Pontianak” contains elements of “mati anak”, which means “dead child” in Bahasa Melayu, and therefore it follows that “Pon” is likely a corruption of the Malay “Puan” an honorific for Woman/Wife. So Pontianak literally means “Woman of a dead child” (Skeat 1900) [10].

What’s important to note here is the Pontianak is a feminine figure whose origins are tragic, and that she is subject to multiple transformations: from corpse to beautiful woman, to flying winged creature, grotesque long-nailed monster, and back again.

Today, the Pontianak is an iconic and instantly recognisable figure in popular culture in Malaysia, Singapore and Indonesia; she has appeared in novels, comics, collections of supernatural tales, video games, and most predominantly in films.

However, we can argue that the official ‘birth’ of the Pontianak as an icon of Southeast Asian horror occurs only in April 1957, with the premiere of the Singapore-produced, Malay language film Pontianak - the subject of our research.

2.2 History of the Film

Background to the Malay Film Industry in Singapore.

The Malay-language film industry in Malayan-era Singapore has its origins prior to the Second World War, but came to prominence in the period shortly after 1945 and had a period of great productivity, oft-referred to as a “Golden Age” from 1947 to 1965.

The films they produced were shot largely in studio interiors with additional location shooting rural and urban in Singapore; they stared ethnically Malay talent from Singapore, Malaya and Indonesia; the directors were initially imported from India, and also to a lesser extent, Indonesia and the Philippines. The films initially drew on Malay musical and theatrical traditions, such as bangsawan (a form of Malay folk opera) as well as being heavily influenced by popular Indian cinema of the time.

Pontianak was directed by B.N. Rao, a veteran Indian director who’d been brought to Singapore by the Shaw Brothers to make Malay films in 1953, but left to join Cathay-Keris in 1956.

Pontianak arises out of a creative collaboration between its lead actress - Maria Menado, and its screenwriter, her husband, Abdul Razak. Maria Menado, originally Liesbet Dotulong, was born in Dutch-colonised Indonesia and arrived in Singapore to work as a model in 1950, where she married boxing promoter Razak, before he brought her to the attention of the Malay film studios. According to Maria Menado, she helped her husband write the story for Pontianak, based on her own knowledge of the supernatural from rural Indonesia (Zieman 2007) [11].

They created a story set approximately a century earlier in rural Malaya, located in the jungle and in and around a traditional Malay village, which we will refer to by the Malay term “kampong”. This environment was deeply rooted in the world of rural Malay folklore that Walter Skeat described, although Razak and Maria Menado fabricated their own origin story for the Pontianak.

There are numerous accounts of the film’s phenomenal box office success. Film historian Mustafar A. R. provides an account of its distribution in Singapore, running from late April to mid-May and then revived again in August, screening for 41 days altogether, which was “a major achievement of the time” unrivalled by even the most popular Malay films. (Mustafar 2019) [12].

The first sequel, Dendam Pontianak was rushed into production almost immediately. By November 1957, both Cathay-Keris and its rival studio MFM/Shaw were announcing multiple Pontianak films in production.

MFM/Shaw managed to release their first rival Pontianak film, Anak Pontianak (with a story also by Abdul Razak), in February 1958, before Cathay-Keris’s third Pontianak film, Sumpah Pontianak, came out in April 1958. These two titles are the earliest Pontianak films in existence, still available in the archives. However, Pontianak and Dendam Pontianak (both 1957) are classified as ‘lost films’.

The Loss of Pontianak and Dendam Pontianak, 1957.

The question of how a film actually becomes ‘lost’ is debatable. Edmundsen and Pike offer a succinct definition of this “relative term”: “(It) Usually means one < film > that no longer exists or, at least is not identified among the holdings of a film archive, production company, distributor, or publicly known organisation or collection. In other words, it is not in one of the places where one might normally seek to find it.” (Edmundsen & Pike 1982) [13].

By this standard, the first two Pontianak films are lost. There is no definite record of them ever being screened after their theatrical runs in Singapore, Malaya, Borneo, Hong Kong or Indonesia (where it was retitled Chomel).

In 1960 Ho Ah Loke and Loke Wan Tho broke up Cathay-Keris, so that Ho could set up a new film studio in Kuala Lumpur. In the official Cathay version of the events (Lim & Yiu 1991) [14], the two men “drew lots” to determine who could keep ownership of which films, and among those that Ho kept, were the first two Pontianak films (but not the third).

In 1963 Maria Menado, who had declared her marriage to Abdul Razak void, became the wife of the Sultan of Pahang, a member of the Malay royal family. After this, by her own account, her films were not permitted to be screened, “no cinema or TV station could play them”, she said in 2007 (Zieman 2007) [15].

Then, as Lim and Yiu write: “Wherever he (Ho Ah Loke) moved, the films would be brought along and usually packed under a staircase. Then years later… perhaps in a fit of depression, he ordered removers to take the cans of film away in a lorry…. (They) are believed to be at the bottom of a lake somewhere in Malaysia.” (Lim & Yiu 1991) [16].

This speculation about Ho’s “depression” was supported by a memoir by a former Cathay-Keris employee Hamzah Hussin, published in 1997, where he posits that Ho acted out of “disappointment and defiance” (Uhde 2005) [17]. Barnard goes on to explicitly make the link between the act of disposing of the films in a lake or river with the fact that Maria Menado’s royal status made the films commercially unusable (Barnard 2011) [18].

This version of the story has since become the oft-repeated ‘legend’ of the film’s loss, supported by further details supplied by Maria Menado since the 2000s. She told an interviewer in 2005 that after the Sultan’s death in 1974 she successfully applied to the Royal Council for permission to have her films screened again and contacted Ho Ah Loke to view the first Pontianak film. He told her it had been thrown into a mining pool.

“I didn’t ask him which lombong (mining pool) it was because I was already too upset. I was speechless when he told me about it.” (Koay 2005) [19].

L. Krishnan has repeatedly debunked the notion that Ho was angry or depressed when he disposed of the films. He states that Ho needed space in his house and removed many of his films (presumably canisters containing negatives as well as distribution prints), of not only the two Pontianak films [20].

The story is further complicated by an interview that Maria Menado gave in 2013 on Malay television in Singapore [21], where she states that upon request Ho Ah Loke did supply her with a print of Pontianak, for her own personal viewing, but it was dubbed into Mandarin (the film was re-released with Mandarin dialogue in July 1958). This contradicts the earlier, and oft-told, anecdote that Ho told her they were thrown away.

Another complicating element is the assumption that Ho Ah Loke was the sole possessor of prints of his films. Given that multiple prints would have been struck and released in multiple territories (where they would have been stored by the film’s local distributors), it is certain that other prints existed elsewhere.

However, in at least three decades of believing the first two Pontianak films to be lost, there has been no account of anyone seeing the films since their initial theatrical runs, and not a single trace of a copy or print has ever been unearthed.

2.3 Unearthing the Synopses of the Lost Pontianak Films

For the purposes of our research the primary task was to find out as much as possible about the content of the two lost films, and what follows is a summary of research completed so far, although it is still ongoing.

Available sources included newspaper articles, interviews and books, essays and articles on Malay cinema. Key sources for the ‘narrative’ of the first Pontianak (1957) include:

• 1. Maria Menado herself, from an interview she gave in 2005 in which she described the film’s plot in some detail

• 2. The website of Malaysian film historian Mustafar A. R. and his recently published book, 50 Years of Film in Malaysia and Singapore, which includes his personal reminiscences of watching the film

• 3. A short, but complete synopsis of the film published as part of promotional material for the Mandarin dialogue for the film in 1958

• 4. A review of the film in English published in The Straits Times, Singapore’s main English language broadsheet, at the time of its release in 1957

• 5. References to prior aspects of the story that are present in the final film of the initial trilogy, Sumpah Pontianak, 1958

One discovery we made was that over the years, books and articles that covered the lost Pontianak films invariably focussed on the first film, so that there was almost no available information regarding the sequel, Dendam Pontianak.

Attempts to contact Maria Menado, and her Pontianak co-stars, for new interviews were unsuccessful, but are ongoing. Interviews with other individuals associated with the Malay film industry, contemporaneous to the films, led to very little information on the content of the films, although valuable context was supplied.

An attempt to canvass social media for memories of the first two films, in collaboration with the marketing of a new Pontianak film in Singapore and Malaysia - also called Dendam Pontianak (2019) - led to very little information despite a great many enthusiastic responses.

However, through the key sources referred to above, an outline of the first film’s narrative presented itself:

• The film begins with a reclusive old man, Wak Dolah, a bomoh (Malay Shaman) picking herbs in the jungle where he discovers an abandoned baby. He brings her to his home (which is either a cave or a hut - depending on the source), where he names her Chomel (translated as ‘Cute’) and raises her as his own daughter.

• Years pass and Chomel grows up and is revealed to be an “ugly” young woman with a hunch-back, who is mocked and reviled by the villagers she encounters. Wak Dolah teaches her his medicinal skills.

• Eventually, Wak Dolah dies of old age, but before that he makes Chomel promise to burn all his books of magic. She dutifully carries out his wishes, until she notices a book that contains a spell to make its possessor beautiful. Unable to resist she follows the instructions, creates a potion (to be drunk during the full moon) and becomes beautiful. The book states that after her transformation, she must never drink human blood.

• The ‘new’ unrecognisable Chomel enters the village and meets the son of the Village Chief, Othman, who falls in love with her and soon they are married, and Chomel gives birth to a daughter, Maria.

• One day, while taking a walk outside, Othman is bitten by a poisonous snake. He asks Chomel to suck the poison from his wound. Reluctantly, Chomel obeys and once she’s tasted blood continues to drink her husband’s blood until he dies. At this point, Chomel transforms again - this time into a deformed, fanged creature in a white robe with long fingernails and hair - the Pontianak!

From this point, the narrative becomes less clear:

• The appearance of the Pontianak strikes fear among the villagers.

• Chomel hides in the jungle beside the village for many years.

• Years pass and Maria grows up to be a young woman. Chomel, in her form as a beautiful woman, brings her fruits.

• She kills some men from the village.

• When she turns 18, Chomel brings Maria to a cemetery with the intention of turning her into a Pontianak too. This plan fails.

• The village bomoh tries to stop or kill the Pontianak, but this fails as well.

• The village Doctor, Tabib Razak, drives a nail into the back of her neck, and she transforms back into “ugly” Chomel.

• She flees into a hut in the village - pursued by the villagers - who set fire to the hut, burning it down and thus destroying Chomel.

Mustafar, on his website (Mustafar 2012) [22], referred to a publication related to the film, published by a company called Harmy, a Malay publisher based in Singapore in the 1950s and ‘60s that was (among other things) a publisher of film synopses (Nasri 2017) [23].

These synopses were both promotional materials and film merchandise designed to appeal to film fans, by reproducing the stories of films they’d just seen at the cinema (or were about to see), complete with film stills.

Synopses were published in written Malay and Jawi (an Arabic script form of written Malay). Knowing that a Harmy Pontianak synopsis existed, we set about searching for it. Public archives, libraries, and rare book dealers in Singapore and Malaysia did not have it, and neither did the British Library. However, we did find a blog belonging to a collector of Malay movie memorabilia that featured six images of what we realised was the Harmy synopsis, in a combination of Malay and Jawi.

Although it’s not complete, through translation we have learnt several more details that weren’t previously established, including a reference to Gunung Ledang (Mount Sophir) which locates the story between Johor and Malacca, where the herbs grow that “Chomil” (not Chomel in this first synopsis) picks to create the concoction that will make her beautiful. We also learn that Chomel’s daughter Maria is the one who discovers the corpse of her father after his blood has been drained by her mother.

We have since acquired two versions of full synopses for Dendam Pontianak and now know many details of this previously forgotten, lost and undocumented film, bridging the narrative between Pontianak and Sumpah Pontianak.

3 The Virtual Cinematic Heritage Application

3.1 Project Outline

Based on the findings described in the previous chapter, we identified several key moments of the first Pontianak (1957) film of which the “snake bite scene” is considered the most iconic moment: Mustafar describes its effect on the audience at the time: “At that moment, the cinema went absolutely silent since they knew what was going to happen next. The change from Maria Menado’s beautiful face into that of the scary Pontianak shocked the audience, even causing a slight commotion for a while. When the shock died down, silence came again.” (Mustafar 2019) [24].

Location of the Snake Bite Scene.

According to film descriptions, the scene happens in a backyard area (the Malay “tanam” describes a garden or space beside a home) and in earshot of their daughter and the head of the village who come to the scene to find the dead Othman, Chomel’s husband. It is likely that the snake bite scene happened in proximity to the jungle that surrounds the kampong.

The process stages of recreating the ‘snake bite scene’ as a virtual reality experience can be described as follows:

• Script: Writing a detailed film script based on the findings describing the scene

• Visual Concept: References from the films

• Environment: Creating the village and surrounding jungle

• Characters: Creating at least 3 characters; beautiful Chomel, Othman, the Pontianak and the snake.

• Story and Interactivity: Animation layout based on script and design of viewer’s logic

• Character Animation and Realism: Capturing actors’ performances, research and development of hair and cloth simulations, skin shaders, realistic lighting and recreating the look of the film

• Sound design: Recreating the aural atmosphere of the film

• Evaluation and refinements

In the following, we will describe key considerations and preliminary findings of some of the above processes.

3.2 Script, Story and Interactivity

A detailed film script based on preliminary findings describing the scene in question has been developed. In short, the story goes as follows.

Story.

Beautiful Chomel and her husband Othman stroll in conversation from the village into a path leading into the jungle. Othman is suddenly attacked and bitten by a poisonous snake. Othman begs Chomel suck out the poison from his wound. To save her husband Chomel sucks the poison out of Othman’s leg, but in doing so tastes his blood, and the one thing cursed Chomel is not allowed to do is drink human blood. Fulfilling the curse means something terrible is going to happen: Chomel kills her husband and transforms into the Pontianak.

Considerations.

To realise the snake bite scene as a ‘walk-in movie’ we must consider several aspects and possible issues:

• The viewer can walk and look around freely and might not be in range or within a viewing angle to see the action.

• With a pre-recorded animation we have limited control over re-positioning our actors into the field of view of the user.

• How can we be sure that certain key moments such as the transformation of Chomel into the Pontianak are not missed by the viewer?

Approach.

The viewer can walk and look around freely in the Kampong. Chomel and Othman will appear and leave one of the houses if the viewer has their house in focus for more than 3 s. Chomel and Othman will start their 90 s long walk on a predefined path to the snake bite location. After 16 s a “user tracking logic” will be activated to verify that Chomel is in the user’s field of view, or in range. If the viewer is too far away or loses unobstructed line of sight, of either one for more than 5 s, the viewer is teleported to the nearest checkpoint and forced facing Chomel. If the viewer is still far away after 75 s, the line of sight is not considered anymore and the viewer will be forced to a checkpoint to assure he/she won’t miss the snake bite scene at second 90 and following that, the transformation of Chomel into the Pontianak.

3.3 Visual Concept

In the film, the actress Maria Menado plays three characters: “Ugly” hunchback Chomel, beautiful Chomel and the scary Pontianak (see Fig. 2). In the snake bite scene beautiful Chomel transforms into the Pontianak after sucking the poison from her husband’s leg.

Fig. 2.

Maria menado in three roles © 1957, Cathay-Keris

The most accurate visual references are film stills found in the synopsis booklet (Harmy, 1957) [25]. A still shows Chomel and her husband in conversation (see Fig. 3), and assuming that this image shows the couple just moments before the snake bite, we know that the scene happens in daylight in an exterior scene in or around the kampong houses. The book Cathay 55 Years of Cinema (Lim & Yiu, 1991) [26] includes a high-resolution film still from Dendam Pontianak showing the Pontianak in action, strangling a villager. We have visual references for all three characters involved in the snake bite scene: Beautiful Chomel, her husband Othman (played by M. Amin) and the Pontianak itself (see Fig. 3).

Fig. 3.

Film stills from Pontianak (1957) and Dendam Pontianak (1957) © 1957, Cathay-Keris

Kampong Villages in the Film.

None of the available film stills provides a useful visual reference to the kampong environment.

The existing third film, Sumpah Pontianak was released in April 1958, just eight months after the second film’s release in August 1957 and 12 months after the first film, so we can assume that locations and film sets would have been re-used between all three, and it can be a key reference for our virtual kampong.

In Sumpah Pontianak, several scenes play out in studio sets of a kampong (see Fig. 4), these are mainly the longer dialogue scenes. In addition, several scenes are shot in real kampong villages; according to researcher Toh Hun Ping, the main village used was Kampong Siglap at the east coast close to today’s junction of Siglap Road and Upper East Coast Road [27]. In Sumpah Pontianak, the Satay man, a comedic character played by Wahid Satay (who also features in the first two films), strolls around singing, followed by a group of boys from one village along a pathway to a second village. During the Satay man’s stroll, we can see various Kampong houses, and, although the film editing implies continuous action over a total of 9 shots, continuously following the Satay man crossing the screen from right to left, none of the shots appear to be geographically continuous at one location but rather shot at several different locations, eventually ending at a studio set at which a longer dialogue scene begins [28].

Fig. 4.

Stills from sumpah pontianak. Left on location, right studio set © 1958, Cathay-Keris

3.4 Environment – The Kampong

Kampong in Virtual Reality.

Our virtual kampong consists of around 10 different houses modelled in Autodesk Maya (see Fig. 5). These houses take visual reference from those seen in Sumpah Pontianak but are not exact replications. As it would be common for the period, the village houses in Sumpah Pontianak have palm leaf roofs. As the film was shot in the 1950s but set in the past, there are some exceptions of buildings without palm leaf roofs hidden in the background or at the edge of the frame. Regarding the building materials of the house walls, the village houses in Sumpah Pontianak appear to have either palm leaf, vertical or horizontal wooden planks. All three options are reflected in our 3D models (Fig 6).

Fig. 5.

3D modelled houses for virtual village © 2020, B. Seide

Fig. 6.

Virtual kampong village © 2020, B. Seide

Vegetation.

Singapore is famous for its rich tropical vegetation with huge trees and high species diversity. According to the National Park Board, Singapore has “over 2000 recorded native plant species” with “two million trees planted on roadsides, in parks and nature areas” [29]. The kampong sets in Sumpah Pontianak feature an equality high species diversity, ranging from angsana, sea apple, palm, banana leaf, malayan banyan trees to ixora shrubs. The vegetation used in and around our virtual village consists of 18 species (see Fig. 7). These assets are dynamic and react to wind.

Fig. 7.

Vegetation assets used in and around our virtual village

3.5 The Characters

For the project we created 3D character models of the Pontianak, beautiful Chomel and her husband, Othman. In that process an initial approximation of beautiful Chomel was designed in Reallusion Character Creator which served as a foundation for the sculpting of the Pontianak face in Pixologic zBrush (see Fig. 8) and texturing in Adobe Substance. The model was then tested in Reallusion iClone for consistency of the facial rig before designing the costume in CLO Virtual Fashion Marvelous Designer and evaluating the entire character in Epic Unreal Engine.

Fig. 8.

Modelling stages of pontianak © 2020, B. Seide

3.6 Character Animation and Realism

Many factors contribute to the realism of a Computer-generated (CG) character, and while huge advancements have been made in recent years, creating a photorealistic CG character for a real-time environment is still a hugely challenging task. The most important contributing factors are: The skin and eye textures and shaders, the hair and cloth simulation, the character’s level of detail and the character animation in general. In 2018, Epic demonstrated with their Andy Serkis and Siren Digital Human projects, that “crossing the uncanny valley in real time” [30] is possible. For these projects, “Epic Games teamed with 3Lateral, Cubic Motion, Tencent and Vicon to take live-captured digital humans to the next level”. These demonstrations were ground-breaking in several regards, not only was the CG character convincingly realistic but in addition this was achieved in a real-time environment.

Based on these demonstrations, the project team decided to utilise facial and body motion capture techniques for all main characters to achieve a high level of realistic motion. Consequently, the motion would be pre-recorded and simply played-back in the virtual reality experience, with very limited possibilities to react to the users action, thus making the experience passive to a certain degree. As the primary aim of the project is to recreate scenes from a film, its structure is linear by design. So, while still posing uncertainties, the aforementioned limitation seemed acceptable.

Body Motion Capture.

Our research facilities offer a professional Vicon motion capture system, set-up as a circle of 14 cameras at a height of 2.5 m. This camera-based set-up can produce high quality results, but often introduces errors on ground level when parts of the feet are occluded, a limitation of the one-circle set-up, requiring a time-consuming clean-up post-process of the captured data. At the time of writing, the project team is evaluating a lower-cost system from Rokoko, which in contrast to the camera-based Vicon system, is an inertial sensor-based system. Skogstad and Nymoen analyse both concepts and conclude “If high positional precision is required, OptiTrack (a camera-based system) is preferable over Xsens (a sensor-based system), but […] Xsens provides less noisy data without occlusion problems” [31]. Our comparison of the two concepts is ongoing but our recordings indicate that for our snake bite scene, the husband falling to the ground after being bitten by the snake poses the biggest challenge for both systems. As our two main characters, Chomel and her husband walk in an area spanning 40 by 25 m, the portable sensor-based system appears more practical for our use case.

Facial Motion Capture.

The project team is currently evaluating two systems, Rokoko facial capture and the Reallusion iClone Motion LIVE software. Both are iPhone FaceID-based systems. “The iPhone tracks faces with a depth map and analyses subtle muscle movements transforming the iPhone into a powerful 3D biometric mocap camera” (Reallusion) [32].

Cloth Simulation.

Cloth and hair simulation are still huge challenges in real-time environments. Epic’s Unreal Engine is improving constantly, and in 2017 implemented NVIDIA’s NVCloth physics, a low-level clothing solver in version 4.16. The project evaluated the use of NVCloth and Marvelous Designer to simulate the long robe of the Pontianak character. In our initial tests, Marvelous Designer created fewer artefacts and a more detailed simulation, whereas NVCloth produced some intersection artefacts.

Hair Simulation.

The project team tested two different approaches to simulate the long hair of the Pontianak character. Although the Pontianak character would not act with extreme movements in our snake bite scene, a solution for the hair simulation should provide a level of flexibility for a range of movements. After analysing Epic’s Paragon game character projects which used NVCloth to simulate hair using low polygon proxy ribbons wrapped to original denser high poly hair, we evaluated to combine this technique into the photorealistic hair of Epic’s Digital Human project [33] which uses Mike Seymor as a case study. This approach could work for the Pontianak character, if the long hair would not be moving too much.

Fig. 9.

Strand-based hair rendering and simulation system, © 2020, B. Seide

With the release of version 4.24 in December 2019, Unreal Engine introduced Niagara’s new “Strand-based Hair Rendering and Simulation System”. Our initial simulation results of the long hair, generated in XGen and imported as Alembic file, are remarkable regarding the simulation accuracy in comparison to the NVcloth/proxy approach: The Strand-based Hair approach did not show artefacts or geometry intersections (see Fig. 9). Unfortunately, at the point of writing the Strand-based Hair approach is not yet usable for a virtual reality experience as it is not rendered stereoscopically for both eyes.

3.7 Film Look and Result

Film Look.

The first colour films in Singapore were shot in 1952 and 1953 by Cathay-Keris Studio. Pontianak and its sequels (1957–1958) were shot in black and white. To recreate the cinematic look as close as possible for the virtual reality experience, we’ve utilised real-time post-processing effects to simulate the quality of the film, such as rendering the experience in black and white, adding film grain and a shallow depth of field. To evaluate the cinematic look of our VR experience, our approach was to first recreate an iconic moment of the existing third film Sumpah Pontianak (1958) (see Fig. 10), which is, apart from a few still photographs, the main source of visual references for the films. This approach allowed us to adjust the lighting and depth of field to get convincingly close to the original look of the trilogy (Fig 11).

Preliminary Results.

The VR experience is compiled for a SteamVR/HTC Vive Pro setup in which the user can freely walk and teleport to explore the kampong and the path leading away from the village into the jungle; the jungle itself is restricted and can not be entered. In our draft result, the music and sound design, a recreation of the aural atmosphere of the film, has not been added yet. The story logic has been fully implemented and as such, Chomel and Othman leave one of the houses and we are free to follow them to witness the snake bite scene and Chomel’s transformation (see Fig. 12).

Fig. 10.

Sumpah Pontianak (left, © 1958 Cathay-Keris), VR Pontianak (© 2020, B. Seide)

Fig. 11.

Stills from VR experience: pontianak in environment © 2020, B. Seide

Fig. 12.

Still from VR experience: the snake bite scene © 2020, B. Seide

4 Discussion and Conclusion

This project arises out of a key question - In Film Heritage, how do you bring to life that which is apparently lost? Authentic recreation of a lost film is clearly impossible, even with anecdotal accounts of the film, synopses of the stories and scenes, detailed knowledge of its aesthetics and style, there are still far too many variables in terms of filmic construction that can’t be definitively resolved. Instead we have taken a different approach, to create a virtual immersive environment that places the viewer into the narrative world of a lost film, and in one particular key scene.

This approach is a theatrical one, in that we are drawing attention to the difference between the lost object and the new work that we are making that is inspired by the lost original. We are using the latest advancements of narrative in moving image and applying them to a film from 1957, that only exists as memory and documentation.

In shifting the media, we hope to evoke and explore the lost film rather than to attempt to strictly emulate it. We aim to create an experience that activates the imagination and will animate the lost Pontianak both for audiences familiar with the Pontianak myth (and the myth of the film), and for those who are completely new to the story.

In preparing the material that can be used for the Virtual Reality Heritage Project, we have gone further than previous researchers in assembling information about the narrative content of the Pontianak films, as well as questioning some of the assumptions about the films that have been repeated in many histories of Malay film-making of that era. There is still much work to be done to learn more about both of the lost films, and we will continue to gather material.

Our visual reconstruction in form of the Virtual Cinematic Heritage application is a unique approach and at the time of writing still a work in progress, and this paper can be considered a preliminary report of our findings. The project aims to recreate a scene of the lost film as a walk-in VR experience in the most realistic way. The technical challenges are plenty and diverse, from creating a realistic environment to the even more demanding aspect of creating realism for three individual characters, including animation and simulation. Although creating near photorealism for such advanced tasks has become de facto possibility and is in reaching distance, it still poses a tremendous challenge for a small research team with limited resources. Our creations of the kampong environment and the three CG characters can be considered as possessing a high level of detail which supports the visual fidelity of the experience but the ultimate goal of reaching near photorealism has not been achieved yet. Even without full photorealism, the ability to freely explore an iconic film scene of a lost film within a historic setting alone is a unique experience, presenting an original approach to Cinematic Heritage.

As of today, there are no other virtual kampong reconstructions in existence. Panorama photos exist, i.e. of Singapore’s Kampong Buangkok, but in regard to walk-in/room-scale Virtual Reality, our project presents the first virtual Malay-Singaporean kampong for audience exploration. As the audience’s narrative involvement is highly relevant for a successful Virtual Heritage application, the research team aims to further develop the virtual kampong reconstruction into a fully inhabited village with authentically behaving and realistic appearing virtual humans.