1 Introduction

Crowd simulation has been widely used in many areas such as urban planning, emergency training and evacuation, simulation of epic battles and traffic engineering. Crowd simulation has also already been used to bring back historical cities such as Rome [56] and Nicosia [14]. They are based on microscopic crowd simulation, which avoids the laborious process of individually animating characters [33]. Performing this specific simulation brings a number of difficulties. Kneidl et al. [28] proposed a multi-scale approach for different types of pedestrians. For simulating an old trading port such as Weld Quay, in George Town, Penang, Malaysia, several ethnic groups with distinct behaviors such as Indian, Malay, Chinese, and British colonists co-exist in this environment. There exist several crowd simulation methods that involve two distinct groups through the different levels of autonomy such as Vi-Crowd model [62] and pursuit and evasion model [45]. However, in bringing the past life into digital cultural heritage, there is a need to have heterogeneous crowd simulation involving several distinct ethnic groups.

In the task of digitally simulating past life, the concentration should be to re-create the general atmosphere of the environment. The distribution of individual activities, the interactions between agents and their surroundings, and the global motion that emerges from everyone’s actions creates this atmosphere. Although there are some manuscripts describing how these multi-ethnics of historical sites looked like, there are hardly any written first hand descriptions of the individual activities and social interactions between people of the past. Since social behaviors vary from each individual to each population, a mechanism to adjust and to adapt the parameters of each agents and each ethnic group in real-time is needed. One of the works in crowd simulation that involves high-level control in a synthetic-vision based steering approach is introduced by Ondrej et al. [37]. For each virtual agent either from the multi-ethnic or multi-group interactions, there is a need for a high-level controller that runs an action selection mechanism in order to engage with new goals and change of behaviors.

To model the interactions between these diverse populations, there are needs to have each of the population group to be associated with a set of possible states, which corresponds to specific behaviors. The changes of states are triggered by global or local interactions between populations. The global and local interactions are then used to decide the next event in daily life of the virtual agent. To enable tracking of what has been done by an individual agent in a simulation cycle, hierarchical state machine is used in the microscopic crowd simulation. This approach allows the activities that have been carried out by a specific agent belonging to a population to be re-tracked. This re-tracking is good for understanding the lifestyle of each ethnic group or group of people, which can be different from other ethnicity. This work is also partially inspired by the research in crowd simulation for interactive virtual environments and VR training systems [59]. There is a need to re-create past life where the global atmosphere becomes prominent from a detailed modeling of individual activities and interactions between multi-ethnic groups using hierarchical state machine.

As the heterogeneous population of say an old trading port of Weld Quay in George Town is highly diverse in ethnicity, culture, language, and religion, the interaction model of these populations demonstrates the social activities in a specific environment. For instance, the trading activities include interactions such as escorting the clients, daily greetings with different social classes in the population or even supervising the work of their workers. One of the good solutions to establish these interactions is to support more complex behaviors like various kinds of group formations. There exist some works on modeling such group formation, namely the automated generation of freestyle group formation through user input [19] and the spectral-based group formation control [53]. However, these works are based on key framing and the simulation needs a flexible and automatic reformation and dispersion of group members in real-time based on their interactions. Therefore, there is a need of microscopic based crowd simulation technique to model the group formation realistically and find a reasonable way of handling different kind of group formations.

2 Related work

Crowd modeling is usually done to predict crowd movement and behaviors, with all the pressures, risks, delays in actions and reactions generated from a large group of people. The basic agent models are the microscopic models, detailing crowds at individual level and behavioral modeling. In computer graphics and simulation of crowd, the design of algorithms is taken deeply into consideration. Flocking behaviors tends to use force-based models that are able to perform collision avoidance and local interactions [1]. The classical flocking behaviors are simulated through boid algorithm, as it is well known for expressing motion of the group especially for crowded animals. Craig Reynold [42,43,44] has proposed numerous behavioral models with regard to flocking algorithm to drive the interactions between autonomous characters. To improve the realism of these steering behaviors, additional dynamic conditions are devised into the flocking computation. The conditions include external forces that improve boid path guidance [51] [4] [40] [63], movement of the crowd [13] and micro and macro interactions [30] [15]. Despite of the inclusion of different additional forces into the simulation of crowds, the demand for having variety and heterogeneous crowd is rocketing. Concerning about the heterogeneity of the crowd, there is a need to go beyond the limit of conventionally simulating the crowd.

In order to have a high-level behavior, the graphic designer has to manually annotate the trajectories and goals that are associated with their local and global interactions. One good solution is to have keyframe-based interactive interface for manipulating the local and global agents’ behaviors. Contrary to the goal of the research, an inspector that allows low level parameters to map and adapt with agent properties can be much more intuitive. Ondrej et al. [37] have also aimed to obtain a higher-level of control and to extend simulation abilities so that the interactions between agents are not limited to collision avoidance.

State machine has always appeared to be beneficial in determining complex behaviour using state transition as it is simple [46]. Zheng et al. [66] aimed is to have behavior control that involves path-planning of many agents, yet still sufficiently complex to achieve interesting behaviors. Therefore, Lemercier and Thalmann [31] proposed a model based on the combination of rules and FSM [8, 9, 12, 27] for controlling agent’s behavior using layered approach. Besides, most traditional finite state machines are considered as semi-automated tracker. Navarro et al. [36] modified the state graph and the transition matrix at run time on top of the hierarchical finite state machine for provoking certain event occurrences. Contrary to our goal, through hierarchical state machine for heterogeneous crowd that is fully automated, desired agent behaviors could be reproduced individually for re-tracking the past activities.

There exist also some works on modeling group of peoples with different formations. Many methods have been explored for group formation namely the social-force model [64], repulsion and tangential social forces [2], rule-based model [61] and cognitive planning [60]. There is also a comprehensive review of crowd simulation techniques by Thalmann and Musse [56]. Besides, O’Neill [38] have proposed to control a flocking simulation through scripted behaviours, procedural actors, key framing and user sketching respectively. User sketching has also been improved to control the motion transition intuitively by Gu and Deng [18] and Gu and Deng [19]. Other than that, research for group formation is continued with edited group motion that focused on minimizing the distortion of relative arrangements among adjacent agents [29]. Takahashi et al. [53] combined heuristic rules with explicit hard constraints to produce and control sophisticated group formations. Patil et al. [39] also proposed a comprehensive framework to guide moving agents with navigation fields. Despite of many proposed models, the approach in this research focuses more on the microscopic perspective of the influence of local and global interactions in order to form the crowd formation’s overall shape in real-time with rule-based model.

There are a few application areas associated to the simulation of heterogeneous crowd. One of them is in the preservation of digital cultural heritage [23]. In Rome, Italy, Namaz prayer in the Sokullu Mosque [56], virtual pompeii [34] and virtual Romans in old cities [55] [58] are presented. Other similar digital heritage projects that include crowd simulations are in Nicosia [14], Angkor Wat Cambodia [7], Hajj rituals in Tawaf [16], mummies in Egypt [3], terracotta in Xian (Magnenat-[57]) and virtual Imam in Hagia Sophia Ottoman Mosque in Istanbul [17]. There are also simulation done based on the case study in Pennsylvania Station in New York City [47], City of Uruk [5, 6], Roman Odeon in Aphrodisias, Turkey [10] and urban area in United Kingdom [20]. Our proposed work is an extension from Lim et al. [33], where social and rule-based inter-ethnic are simulated at Pier Swettenham, George Town but in a larger scale of about 20,000 crowds that is able to switch alternately with real-time simulation and crowd patches method depending on the distinct activities by the ethnic groups.

3 Methodology

For reconstructing past life, the crowd is simulated and rendered in an old trading port. In order to bring past life, heterogeneous crowd simulation method is proposed together with a high-level controller scheme that allows adaptation of agent parameters in real-time, and a hierarchical state machine that allows re-tracking of agent history of activities. A group formation model is also proposed to ensure smooth group maneuvers depicting the social behaviors of the past life. Figure 1 shows the steps involved in the methodology of this research in a more detailed way and they are as follows:

Fig. 1
figure 1

Proposed Methodology

Full size image
  1. 1.

    Preliminary analysis and review of related work: Related works are reviewed in order to identify the state of the art, the contributions, the technologies and the platforms of the proposed work.

  2. 2.

    Data collection and gathering of the past data: Old postcards, maps and data of a heritage site are compiled for the rendering of virtual environment. Sample motions and movements from some old videos are referenced for crowd simulation and rendering. Basic information on traditional garments is also collected from some books for the rendering of crowd costumes.

  1. 3.

    Bringing the past life:

  • Heterogeneous crowd simulation method: This step is concerned with developing crowd simulation method and integrating other behavior information. The proposed method adopts particle-based steering behaviors that allow multiple interactions with different groups or inter-ethnic populations for digital cultural heritage simulation. Each of the agents is constructed in the sense that they have a particular set of data, representing the characteristic of its group.

  • High-level controller scheme: The scheme is proposed in order to allow changing and adaptation of the parameters of the agent to produce different behavior in real-time.

  • Hierarchical state machine: In digital cultural heritage preservation, it is also better if the history of the agent could be re-tracked in order to understand the daily activities that can be different from one ethnic group or group of people to other groups.

  • Group formation model: A model is proposed for formation of group in the heterogeneous crowd simulation. The local and global interactions drive a group of agents to group together and behave similarly to form a smooth microscopic-based group maneuvers.

  1. 4.

    Assessments and validations: The above-mentioned models, schemes and methods are evaluated through scenario- based visual observation and goal-oriented experiments

  • Scenario-based visual observation: Visual observations and inspections are carried out to evaluate the effectiveness of heterogeneous crowd model and group formation model. The simulation is run for various scenarios and the judgment is carried out based on careful observation on the output of the graphical effects.

  • Goal-oriented experiments: This experiment is carried out through accomplishment of the goals and decision-makings for evaluating the proposed higher level control method and hierarchical state machine respectively. The goal-oriented experiments are also used to demonstrate the efficacy of the featured schemes.

The heterogeneous crowd simulation for bringing past life is evaluated through the scenario-based visual observation of global and local interactions. The high-level controller scheme is evaluated using goal-oriented experiments through waypoint accomplishment of a particular agent. The hierarchical state-machine is also evaluated using the goal-oriented experiment by re-tracking the history of activities of an agent through decision-makings. The group formation model uses the scenario-based visual observation as the assessment method. It is evaluated through the validation of the shape of group formation and smoothness, and as realistic as possible the maneuvers with reasonable number of followers. It is also important to note that there are various techniques for examining the different aspects of the crowds as in Jiang et al. [24] who had proposed a local evaluation approach for the navigating the multi-agents in dynamic scenarios and as in Courty et al. [11] who had assessed the quality of the crowd using the Agoraset dataset, however, in most cases the quality of the simulation is measured by examining its “look-and-feel” [32]. Besides, there is no standard way of evaluating or comparing the quality of steering solutions [50]. Therefore, in evaluating the crowd model and simulation method in this research, scenario-based visual observation and goal-oriented experiments are used. Also, no data-driven evaluation as the past life has no real video data captured for comparison.

4 Bringing the past life

This section discusses the proposed heterogeneous crowd simulation method that aims to realize the multi-group or heterogeneous crowd simulation. Besides, the implementation of the heterogeneous crowd simulation method is also described based on the algorithm formalized for each behavior to cater the pre-defined social rules. In the simulation, other schemes are proposed namely a high-level controller for parameter adaptations, a hierarchical state machine where the history of the agents’ activities can be retraced as well as a group formation model for the case of microscopic-based group manoeuver.

4.1 Heterogeneous crowd simulation method

Before crowd is simulated, there are a few pre-processing that have to be taken into consideration. First, an environment mimicking the port is setup through the reconstruction of the virtual environment. Other considerations such as the narration planned for each of the population to display their social behaviors, taking the ships as another type of population to be managed like a group of people and finally the human templates that are distinguished through their traditional costumes to determine their ethnicity or group in the simulation are also vital. However, this section explains only how the crowd is modeled as autonomous agents in a standard homogeneous way and how these people or groups of people are animated.

The motion of idealized agents consists of action selection, steering and flocking, and locomotion processes. Some of the action selections include having the goal to flee away, seeking a target, avoiding an obstacle, and following a path or flock with the agents’ neighbors. In this research, the focus is on the types of steering behavior and they are combined to cater the complex system of having four different types of population simulated with their specific behaviors to propose heterogeneous crowd simulation method. Some of the steering behaviors include repulsive bound, following wall, containment, alignment, cohesion, and separation. Other more complex steering behaviors would include obstacle avoidance, seeking or pursuing, fleeing, arriving, path following, queuing, wandering and evading [49]. The mathematical principles behind these steering behaviors can be further explained by Shiffman [48]. The locomotion concentrates on animation of the virtual agent. In order to implement all these behaviors, Reynolds ([42, 43] has developed the boid model based on forces that are used to determine the changes of velocity and direction. From this idea, a few basic behaviors are modified for the proposed heterogeneous crowd simulation method. In order to first simulate the standard homogeneous crowd and the crowd is then simulated in heterogeneous manner.

The implementation of flocking behaviors started off with simplified animation in 2D that involves the position, velocity and acceleration. The first model was done based on OpenGL where the crowd is represented by particles that is able to perform collision avoidance behavior. Then the crowd model is ported into Unity 3D using 3D circles and finally replaced with the Rocket Box Characters. Figure 2 shows the illustration of the virtual agents first done in OpenGL, simulation of virtual agents in Unity 3D and the transformations of circles to virtual agents.

Fig. 2
figure 2

Illustrations of the Virtual Agent (white) Colliding with Rounded Obstacles (yellow) in OpenGL (left), Simulation of Virtual Agents in Unity 3D (middle) and Porting the Circles to be Virtual Characters in Unity 3D (yellow)

Full size image

The skeleton of the Rocket Box Characters was then used and the materials of the skin colors, eye colors and hair colors were changed. The fabrication pipeline is done in Maya, which renders the .fbx, and .fbm formats so that the materials attached to the characters can be imported into Unity3D. The modelisation, texture and shading, setting up the lighting and layout in parallel, animating the character FX with special effects and finally rendering and compositing the virtual characters are done. The special effects include the hair and clothing, the movement and the trajectories, the timing and spacing and finally applying the Unity3D shadow effect on the virtual character. Besides, the mesh of the character is deformed to give the bulges that look more muscular and finally smooth skinning is applied to generate good result for the character representations. Other than that, accessories are added like lacey dress and hats to the character of different ethnic groups. Finally, the colors and designs of certain materials were manipulated to increase the varieties of the characters being designed. Therefore, each character has some dedicated components and scripts attached to them, giving them individualized identity.

Animation is basically needed for creating motions to the crowds. In term of transition of elementary motions, the motions of the crowd are not visually represented at the early stage of the simulation. By using motion capture, there might be incomplete point of trajectories where certain agents are occluded as if they are not within the camera view. To improve the representation of crowd motion, a study of the kinematics of basic human locomotion is crucial [30]. In this paper, their trading behavior is simply shown and a transition form of actions provided by Unity3D is carried out. These are run in the state machine. Agents can switch their motions according to the transition model as illustrated in Fig. 3. This general transition of elementary motions is applied to all the ethnic groups.

Fig. 3
figure 3

Transition of Elementary Motions

Full size image

To simplify the processing of the crowd behaviors, these animations are solely relied on the animation algorithm and thus the motions can be generated procedurally and choreographed automatically. This research adopted the steering behaviors algorithm of Craig Reynolds and formalized the algorithm that at each time step, the new state is computed from the local and global interactions and each state is assigned with different priority weights. In the simulation, there are two assumptions that are used at first. The acceleration is given by a summation of forces and the mass is always assumed as 1, thus the force is equal to the acceleration. Later on, the mass parameter is introduced starting from the Indian model with Force = Mass * Acceleration. The formalization of the different behaviors using boid algorithm is as follows:

For each particle (i), each force is weighted to give priority to some behaviors over other behaviors with weight, w:

At each time step,

  1. 1.

    Re-compute the force by the summing up the forces due to the different behaviors as follows:

$$ {\sum}^F={w}_1\ast {F}_1+\dots +{w}_n\ast {F}_n $$

where F is Force and w is weight from 1 to n.

  1. 2.

    Re-compute the acceleration from the forces as follows:

$$ {a}_t+{d}_t=\frac{\sum^F}{mass} $$

where a is acceleration, t is time and dt is delta time (difference between current time and previous time).

  1. 3.

    Re-compute the speed from the acceleration as follows:

$$ {\mathrm{Velocity}}_{t+1}={\mathrm{velocity}}_t+\mathrm{Acceleration}\ast \updelta \mathrm{Time} $$

where

$$ {\mathrm{Velocity}}_t=\mathrm{Direction}\ast \mathrm{Speed} $$
  1. 4.

    Re-compute the position from the speed as follows:

$$ {\mathrm{Position}}_{t+1}={\mathrm{position}}_t+\mathrm{Velocity}\ast \updelta \mathrm{Time} $$

where d is delta time of the velocity changes.

  1. 5.

    Display the virtual agent at the computed new position.

4.2 High-level controller

A high-level controller scheme proposed in this research is to build a simulation platform that enables parameter adaptations of specific agents’ based on the incorporation of Reynolds’ and Helbing’s models. This research is not concerned with scientifically life-like or realistic crowd simulation, as no motion-captured data are available. It is about believable production of scenes where the behaviors are shown through the interaction models. The interaction models are scripted as a rule-based interaction between the agents. The communications among the agents were animated. By applying the classical models, it resulted in the velocity- and force-based interactions. These interactions can be changed and adapted into virtual agents’ behaviors by a set of parameters through the high-level controller in real-time to influence the events. The global events that triggered the changes of local and global behaviors of the virtual characters have been shown to be successful.

In the Helbing and Molnár [21], there are social-force model, attractive and repulsive model as well as prisoner theory that are mainly used to display evacuation, panic and turbulence scenarios. In this research, a modified neighborhood, vision and density-speed models have been included into the particle-based steering behaviors to show the trading scenes and the activities involved. The models are the neighborhood model which includes the separation, alignment and cohesion, the vision models which uses the vision angle and vision distance to control the wandering behavior, the density-speed control model in twigging the local and global responses and the rest of the controls are based on parameter adaptation through the inspector of the simulation. Within the neighborhood model, it is speculated that there exists cooperation behavior among the English soldiers and captains while the competition behaviors exist among the Malay sellers who are trying to approach their targeted potential customers on the nearest distance basis. The non-linearity solution by having zero and one bits as triggered and non-triggered path-points is also introduced to control the trajectories of the crowds.

Crowd consists of many individuals. If the virtual agents are all trying to achieve the same target, the route will be exploited and deadlock will occur. Depending on the complexity of the environment, it is necessary to understand the problems associated particularly with the speed, mass and force relationship of the virtual agents. By altering the parameters of the model and adding an alternative route, the model may improve both congestion and circulation problems when the agents are heading to the same waypoint. By having random element, it will affect the outcome of the behavior. For instance, a minimum travel distance is required from one point to another to slow down the agent. The simulation is capable of rapid modification of all parameters, such as behavioral dynamics, random variables, and event triggering in order to adapt to their surroundings using simple rules.

A higher level controller adopts a situation-based distributed control mechanism that gives each agent in a crowd specific details about how to react at any given moment based on its local environment [52]. In this case, more rules have to be pre-scripted to predict all the possible reactions for the heterogeneous crowd behaviors. In this simulation, a higher level controller scheme is proposed by standardizing a set of parameters that can take immediate effects in changing the behaviors of the virtual agents to be altered and adapted into the situation in real-time without terminating the simulation over and over again. Figure 4 shows a sample of controller used by the FlockManagerChinese. With Unity3D Gizmoz, the position of the virtual agents is obtained in x, y and z axes. The rotation and scaling of the characters are also provided. The FlockManagerChinese is scripted with some public parameters such as alignment weight, avoidance distance, the wander radius, the maximum force, the number of flockers and so on for rapid changes in the simulation.

Fig. 4
figure 4

Controller of Flock Manager Chinese

Full size image

With the proposed high-level controller scheme, the parameter adaptions for each of the virtual agents are automated in real-time. By altering the parameters during the simulation runs, the graphics designers can achieve the ideal behaviors according to the semantics and narration pre-defined at the early stage of simulating the crowd in digital cultural heritage [33].

In this simulation, particle system is not perfectly suitable in overcoming a simulation with high-density (from 200 to 20,000 agents) crowds that are all interacting naturally and simultaneously in a largely populated dynamic environment [67]. Therefore, boid algorithm is re-formalized with additional rules to match the changing goals. One good example is with the Indians, which are simulated based on a set of specific rules. The Indians have three stages namely Carrying-Goods, Wandering and Reaching-Boat. Instead of having the direction multiply with the maximum speed to obtain the velocity in general, the current target speed is manipulated and applied. When there is an event where the ship reaches the pier, the virtual agents who are in the Carrying-Goods and Wandering state will travel with a quarter of their maximum speed. As for the boat, when the path point is triggered, the next target distance is 2.0 while it is 3.0 when the path point is not triggered. Otherwise when the next path point is not null, in the case of Reaching-Boat for the coolies where they have a task of loading the goods, they will travel with their maximum speed.

For the Indians, an assumption is also made where the original mass is in the ratio of 1 before the Indians carry the goods and more than double of the actual mass, which is 2.25 when they are having the goods on their shoulders. Therefore, the goods that a coolie is able to carry is at least more than 200% of their weight. Besides the current target speed were change from 0.25 for loaded and 250 when they are unloaded. All the forces will have less impact when the mass of the virtual agents is increased where they will eventually reach their maximum speed. The virtual agents will thus have more difficulty in accelerating when the mass is increased and more time will be taken for them to reach their actual target speed as illustrated in Fig. 5. The simulation adapts the real ratio of the actual walking speed of 1.5 m per seconds and actual running speed of 1.8 m per seconds.

Fig. 5
figure 5

Example of Parameter Adaptations by Boat Sailing

Full size image

To exemplify the interaction between the states and the local and global behaviors of the crowds through the proposed high-level controller scheme, the parameters are adapted to optimize the simulation based on various scenarios. The observed simulation speculated that only certain forces are applied to guarantee the behaviors to be both steering and collision-free.

4.3 Hierarchical state machine

At this point, the heterogeneous crowd model and the higher-level controller scheme for parameter adaptations have been explained. Next, in this simulation, it is found that there is a need for hierarchical state machine. The semantic flows of bringing the crowd has been proposed and developed based on traditional state machine. However, the graphics designer will face the problems of tracking the history of the virtual agents to validate their correct behaviors for different scenarios. Therefore, a hierarchical state machine consisting of four different groups of peoples could be a solution in tracing the history of the activities of the virtual agents individually. For delivering the state machine in a hierarchical manner, the states and events for all the groups of peoples need to be standardized.

The current simulation is based on state machine. Each population (Indian, Malay, English and Chinese) has their own state machine. The transition states, the interaction types and the condition cases govern each state machine. The association between two states then triggers the events. To make the event occurs, the agent is driven by a set of combination forces. The trading interactions between the Indian start with wandering. When the boat is at the pier, the Indians will switch to the reaching-boat state provided that they are not engaged with other activities with other agents. Then they will upload the goods and perform path-following behavior until they reach the container. Other agents that are engage with other activities will procrastinate and complete the previous interactions before they switch to the reaching-boat state provided that by the time they end the previous interactions, the number of goods is still not null. So in this case, they are obliged to upload and download the goods until the load in the ship is empty as their static occupation is unskilled worker.

For Malay, some of them are initially assign to be as static seller at their stalls while some are being the mobile seller at the marketplace. The mobile seller will approach the potential customers who are not the agents from their own community. Their potential customers are Indian and Chinese who are in the wandering state and English who are in the monitoring state. When there exists some potential customer, the selling interaction occurs and both the agents who are interacting will leave after certain duration. During the one-to-one interaction between two agents, the agents can choose to perform the trading in the proper manner or they can choose to rob the local vendors. If the Malay vendor is being robbed, the particular vendor will alarm the police (English) and then continue approaching the next target customer.

The English will start with being in the monitoring state. They will consistently check if there is any Malay asking for help. If there is, they will switch to the robbery state provided they are within the distance where the robbery occurs. The police will execute the action of start chasing and when he managed to catch the thief, the police will call for another two colleagues to help escorting the thief to the prison. When four of the characters eventually arrive at the prison, the three policemen will switch back to monitoring state while the thief character will be suppressed and appear as a new character. There is also another possibility where the thief is running too fast and too far from the police, the police will decide to drop the arrestment and switch the state to stop-chasing. When the English is in the monitoring state, they also have to be alarmed if the boat is at the pier. When the boat arrived at the pier and stop sailing, the English is expecting some Chinese traders from the sea-route. If their identity is a captain, they will switch their state to reaching-boat. The captain also calls for other few followers to follow behind him to escort their customer.

The Chinese starts with the initial state of standing-on-the-boat. If the boat arrives at the pier, and if among the Chinese, he is a Kapitan (Chinese head), he will be having the meeting event where the English captain and the Chinese Kapitan will interact. After certain greeting duration, all of them will establish path-following behavior until the entire group reaches the fort. After reaching the fort, they switch to the wandering state where they can become potential customers to the Malay vendor. If he is not a Kapitan, he will wait for the leader to call before they move together to another goal. If he is a supervisor, he will ignore the current event and mainly reaching the container to supervisor the good downloading transition. Then the supervisor also switch to wandering state after certain duration of being at the container. At each time-step, we also interested in checking if the number of goods is null. As the goods are completely downloaded, the Chinese will drop all their current interactions and rush back to the boat to sail away.

Global interactions cause the changes of state for all the agents belonging to the target population [33]. Local interactions are detected between specific agents and resulting in the changes of state and/or of steering parameters for the specified agents. In our formalism, interactions between different populations are used as event-driven switches of states. These interactions can be either global or local, depending if they apply to the entire populations or to specific individuals as illustrated in the Fig. 6.

Fig. 6
figure 6

Example of Interacting Populations based on Event-driven Switches of States

Full size image

An agent can be in the initial state of selling, wandering, monitoring or standing, depending on which ethnic group the agent belongs to. In an environment that involves the marketplaces, the obstacles, the containers, the fort, the prison, the pier and the sea, an agent can be influenced by the global and local interactions. The global behavior of being aware of the boat arrival will result in the local behavior of Indian coolies of uploading and downloading the goods. Another example could be when the potential customers are available in the global behavior, the Malay will automatically inspired to approach the target customer.

In each transition model, the states of each ethnic group are formulated. The formulation involves identifying the states of which are linked to their potential changes of actions and describing the transitions of one goal to another goal. Each transition models is governed by a set of state events, the types of interactions and the game objects. The game objects refer to the ethnic group or individual agent of an ethnic group that propagates the events. Since the virtual agents all have different characteristics depending on when they first appear, what event they will be taking part, the reaction of individual agents given who they are interacting with, where they are supposed to go and the level of degree of freedom that each ethnic has given to their virtual agents, the temporal and spatial information associated with a set of algorithms as in Ju et al. [25].

Currently we already have at least 18 actions, 18 states and 11 testing conditions as depicted in Appendix 1. However, the outcome is non-deterministic as the parameters and states used changes at each-time step depending on the state machine. There are possibilities of bad actions, suicides and unreachable episodes. Unreachable episode could be we are expecting an agent to switch from one state to the another in the order of our pre-defined states, but some changes might occur, skipping some unnecessary states to reach his sub or ultimate goals. Agents have their own story line. So for the entire agent to switch from one state to another, there are a few possibilities of the next state. The possibility to the next state is highly dependent on the test condition. We give three examples here:

  • Whether a person is to steal or not it depends on whether there is any police around them or not.

  • Whether a person stop-by to buy something, it depends on if there are available and are consider as a potential buyer or not.

  • Whether the Chinese goes to the container to supervise the work or not, it depends on whether there exist an English troop to come and discuss with the Chinese group.

4.4 Group formation

In bringing crowd in an old trading port, a few specific group formations are needed for portraying the social activities. Therefore, there is a need to propose a group formation model based on the heterogeneous crowd model that could demonstrate adequately superiority between the leaders and the followers. Unlike the previous works, in the simulation, the number of followers for each group formation is also assessed to ensure that the formation is smooth in handling the maneuvers. This section explains how group formation is guided by the leader- following and queuing behaviors and how it influences the search for nearest way-point and practice smooth turning between the leader and the followers.

In the proposed group formation model, the leader-following and queuing behaviors are focused and re-formalized to behave according to the predefined social rules. Figure 9 shows the leader-following behavior where the neighborhood region is decided by the condition set in the function inSight(). Two optimal conditions need to be computed in order to decide if the virtual agents should perform the leader-following behavior. The conditions are as follows:

  • The horizontal vector must be <5.0f

  • The vertical vector must be <3.0f

The horizontal vector is obtained through the dot product between the forward vector of the leader and the vector between the leader and other virtual agents. The vertical vector is obtained through the dot product between the right vector of the leader and the vector between the leader and other virtual agents. If both of the conditions are fulfilled, the virtual agent is considered in the neighborhood region and thus the specific virtual agent is assigned to obey the leader-following algorithm. The algorithm used for leader-following behavior among the heterogeneous crowd is formalized as shown in Fig. 7.

Fig. 7
figure 7

Algorithm for Leader-Following

Full size image

Queuing force is used especially on the Indians that upload the goods at the pier and download the goods at the container. The separation force ensures that it is not big so that they can reach the indicated way-point as the animation of picking up, carrying the goods on the back of the shoulder and putting them down are not visually represented. Figure 8 shows the algorithm for queuing behaviour.

Fig. 8
figure 8

Algorithm for Queuing Behavior

Full size image

Waypoints are specific route in a trajectory in terms of the movement of the virtual agents. Figure 9 shows the algorithm for path-following behavior that is used to search for nearest waypoint. As in the simulation, the virtual agents have specific task such as going to the prison, uploading the goods at the pier and downloading them at the container. The options are also given for the virtual agents to skip the navigated route by computing their nearest way- points based on distance. The algorithm for determining the nearest waypoint is as shown in Fig. 10. Figure 11 illustrates two flockers that search for the smallest distance d, as the nearest waypoint.

Fig. 9
figure 9

Algorithm for Path-Following Behavior

Full size image
Fig. 10
figure 10

Algorithm for Determining the Nearest Waypoint

Full size image
Fig. 11
figure 11

Flockers Search for Nearest Waypoint

Full size image

Consider the intersection and the angle to the next waypoint. If the virtual agents are all trying to achieve the same target, the route will be exploited and deadlock will occur. Depending on the complexity of the environment, it is necessary to understand the problems associated with the speed, mass and force relationship of the virtual agents. By altering the parameters of the model and adding an alternative route, both congestion and circulation problems may improve when the agents are heading to the same waypoint. The proposed rule-based and social force-based simulation is capable of allowing rapid modification of all parameters, such as behavioral dynamics, random variables and event triggering in order to adapt to their surroundings using simple rules. Another scenario is when the smooth turning is applied to the ship sailing away. The ship has the path-following behavior where it is driven by a list of waypoints that will determine the next target that results in the smooth sailing motion of the boat in a circular way. The algorithm in handling and selecting the next target is shown in Fig. 12.

Fig. 12
figure 12

Algorithm for Handling Waypoint

Full size image

GameObjects are the most important object in Unity3D. It needs special properties before it can become a character, an environment, or a special effect. In the case here, gameObject is a component that is carrying a specific behavior depending on which script it is attached to. The components that are possible are Indian, Malay, Chinese or English, which have different roles. On the other hand, the interactionType is designed in such a way that two or more objects are involved in different kinds of events being scripted.

Smooth turning can be useful to solve congestion and handle collision avoidance in the group. As the simulation concentrates at the pier, the size of the bridge is fixed but the number of virtual agents is increasing over time especially when they are called for duty. Through the path-following algorithm, the virtual agents move from shoulder-to-shoulder according to the designated waypoints. However, the movements will be solitary and the maneuvering will be statically horizontal and vertical at the intersection as illustrated in Fig. 13(a). A steering approach is proposed based on angle prediction to overcome the clogging and deadlocks caused by the turning in real-time. With this approach, virtual agenti will head towards the way-pointi and continues with way-pointj while virtual agentj will by-pass way-pointi and straight away perform smooth turning at way-pointj as depicted in Fig. 13(b). This will give a more maneuvering space of virtual agents that are walking closely together and waiting for their turn to pass a single way-point and the algorithm is as shown in Fig. 14.

Fig. 13
figure 13

Waypoints Setup For Smooth Turning. a Static Maneuvering b Skipping Lookahead Waypoint and Moving to Next Waypoint a Angle-based Steering Prediction d Conversion from Angled Turning in Unity3D. Note that (c) and (d) shows that 135° is equivalent to −0.2 in Unity3D

Full size image
Fig. 14
figure 14

Algorithm for Smooth Manoeuver

Full size image

5 Results and discussion

For the results and discussion for evaluating the reconstruction of past life, the evaluation of heterogeneous crowd simulation using scenario-based visual observations is described. Nonetheless, the evaluation of the scheme for higher-level control through accomplishments of waypoints is also described. Besides, the validation of hierarchical state-machine using goal-oriented through decision-makings and the evaluation of group formation model using scenario-based visual observations are described.

5.1 Evaluation of heterogeneous crowd simulation using scenario-based visual observations

The simple scenario presented in this research in bringing the past life into digital cultural heritage is the first attempt in heterogeneous crowd simulation to model multi-cultural trading port in Weld Quay, Penang in Malaysia, back in the nineteenth century [26]. The implementation was done using the Unity3D SDK [54]. Introducing different congregations for the boats and for the four ethnic groups, together with global and local interactions between them to drive changes of states, enables animations to be performed coherently, conforming to the scenario. The proposed heterogeneous crowd simulation method was evaluated through thirteen meaningful scenarios. The simulation was run for one hour with 40 frames per second. Each frame is being captured with ten virtual cameras and thus at each minutes 2400 images were collected. The images are then put into the right sequence to form the animations of the heterogeneous crowd. The results of the images that could represent the thirteen meaningful scenarios are selected and compiled for the evaluation purpose. Table 1, Table 2 and Table 3 show some simple validation scenarios where these scenarios are designed to test the very basic and fundamental abilities of steering behaviors of heterogeneous crowd. Table 4 shows basic one-on-one interactions where these scenarios test the ability of two homogeneous agents to be able to interact while steering to avoid static and dynamic obstacles.

Table 1 Simple Validation Scenarios - Indian and Malay Groups. These Scenarios are Designed to Test the Very Basic and Fundamental Abilities of Steering Behaviors of Heterogeneous Crowd
Full size table
Table 2 Simple Validation Scenarios - English Group. These Scenarios are Designed to Test the Very Basic and Fundamental Abilities of Steering Behaviors of Heterogeneous Crowd
Full size table
Table 3 Simple Validation Scenarios - Chinese Group. These Scenarios are Designed to Test the Very Basic and Fundamental Abilities of Steering Behaviors of Heterogeneous Crowd
Full size table
Table 4 Basic One-on-one Interactions. These Scenarios Test the Ability of Two Homogeneous Agents to be able to Interact while Steering to Avoid Static and Dynamic Obstacles
Full size table

Based on the results of the evaluation, it can be seen that the agents from different groups behaved heterogeneously with individualized parameters such as updated speeds, velocities and positions. This simulation of the heterogeneous crowd model has overcome the typical problem of homogeneity among the virtual agents from various groups. Besides, the proposed model could cater four different groups of crowd and a congregation of boats in this simulation and thus, higher number of groups of crowd can also be attainable through this proposed heterogeneous crowd model.

5.2 Evaluation of the scheme for higher-level control through accomplishments of waypoints

This research has proposed a heterogeneous crowd simulation method by simulating a local trading port in the 1800s, where various populations with very different social rules interacted with each other to bring the past life into digital cultural heritage. These populations include Indian coolies, Malay vendors, British colonists and Chinese traders. The challenge is to model these ethnic groups as autonomous agents, and to capture the changes of behavior due to inter-ethnic interactions and to the arrival of boats at the pier. Agents from each population are equipped with a specific set of steering methods that are selected and parameterized according to predefined behavioral patterns (graphs of states).

The current model simulates the scene of Chinese traders arriving with the boats and trading goods with the English, customer of various ethnic groups at the market place, and complex interactions such as soldiers arresting a thief. For the evaluation of the scheme for higher-level control, the experiment was set up with six different waypoints as the goals. In the simulated scenes there are goals for the agents to reach depending on the changes of states through the indication of the triggered waypoints as illustrated in Table 5. Each waypoint is indicated with colors to represent the ultimate goals and a visual observation is carried out to ensure that the agents fulfill their goals as indicated by different waypoints. In order to obtain a fair result showing the effectiveness of higher-level control in heterogeneous crowd simulation, the simulation was left running until the first agent outraged the other agents in accomplishing each goals.

Table 5 Indications of Various Waypoints for the Accomplishment of Specific Goals
Full size table

There are some results of the dedicated agents taking waypoints as their goals and the responses are influenced by the changes of global and local interactions. Table 6, Table 7 and Table 8 show the agent and agent interactions where the agents meet their goals while avoiding the obstacles and navigating the possible routes.

Table 6 Agent-agent Interactions - Goods Uploading and Goods Downloading. These Scenarios Test the Ability of an Agent in Achieving Their Designated Goals while Avoiding the Obstacles and Navigating the Possible Routes
Full size table
Table 7 Agent-agent Interactions - Fort Arrival and Prison Arrival. These Scenarios Test the Ability of an Agent in Achieving Their Designated Goals while Avoiding the Obstacles and Navigating the Possible Routes
Full size table
Table 8 Agent-agent Interactions - Re-appear and Supervising. These Scenarios Test the Ability of an Agent in Achieving Their Designated Goals while Avoiding the Obstacles and Navigating the Possible Routes
Full size table

Based on the results captured during the simulation, it can be seen that by manipulating the parameters of the agents in real-time, the agents can quickly adapt to achieve their goals through the indications of the pre-defined waypoints. For instance, certain Indian workers reach the good uploading waypoint (Goods Uploading) while the rest is still engaging with other activities in the trading port. Besides, the Indian workers who have already uploaded the goods will continuously reach the downloading good waypoint (Goods Downloading) first as compared to other Indian workers. It is also observed that only the English, Indians and Chinese are involved in arriving at the fort waypoint (Fort Arrival) while the Malay vendors remain in the trading port. Besides, only the English and Indian groups arrived at the prison way- point (Prison Arrival). Other good example is that only the virtual agent with its parameters changed on purpose turned into a thief and thus after reaching the prison waypoint, only that particular virtual agent re-appeared at the re-appear waypoint (Re-appear). Other than this, one of the Chinese traders is turned with the parameter changed to a supervisor, left the group when the English is escorting them back to the fort and thus only that particular virtual agent whose parameters are twigged in real-time reached the supervising waypoint (Supervising). Then it can be concluded that the scheme for the higher-level control has been successfully incorporated into parameter tuning and adaptation in heterogeneous crowd simulation in real-time. This would improved the quality of the results of the simulation and aid the designers in reaching a desired level of realism.

5.3 Validation of hierarchical state-machine using goal-oriented through decision makings

In this research, a new formalism is proposed for heterogeneous crowd simulation where interactions between the different ethnics groups and with the boats can be either activated globally or locally. Global interactions cause changes of states for all the agents belonging to the target population, while local interactions only take place between specific agents, and result in changes of states for these agents only. One of the main contributions of the proposed crowd simulation method and rendering is to apply microscopic crowd simulation to the complex case of heterogeneous crowd simulation by simulating a multi-ethnic trading port which involves different behavioral patterns and then introduce a transition model through state-machine to formalize this inter-ethnic or inter-groups interactions.

A validation of the proposed hierarchical state machine is carried out through the assessment of the log file in order to trace the transition states of each agent. This aim is to re-track the past history of each agent belonging to the ethnic group. This is done through the assignment of the selected agent in each ethnic group. Figure 15, 16, 17 and 18 show the crowd events of the Malay, Indian, English and Chinese groups respectively. Note that every individual agent who is interacting has the potential in stealing the goods from the vendors provided that there is no police in the vicinity. This action is illustrated in the crowd events of Indians.

Fig. 15
figure 15

Crowd Events of the Malays Group

Full size image
Fig. 16
figure 16

Crowd Events of Indians Group

Full size image
Fig. 17
figure 17

Crowd Events of English Group

Full size image
Fig. 18
figure 18

Crowd Events of Chinese Group

Full size image

The simulation test was run for one and a half minutes with 60 frames per seconds. Four agents act as the indicators so that their history of their activities can be traced after the simulation is stopped. The four agents are the Malay_001 from the Malay ethnic group, Indian_001 from the Indian ethnic group, English_001 from the English ethnic group and Chinese_001 from the Chinese ethnic group. The history of the activities of each agent is read from the log file and compiled into the standard state machine diagram based on four different cases. In Fig. 15, a robbery is captured where the transformation of the state of the assigned Malay agent is from selling to interacting and back to selling after the robbery has occurred. In Fig. 16, the incident of arriving at the prison is captured. The log history shows that the assigned Indian agent was from the state of Wandering to Reaching-Boat and Carrying-Goods. After the job of uploading and downloading the goods, the agent went into the Interacting state and then Stealing. The agent is then caught and finally has to follow the police to the prison. In Fig. 17, the escorting client incident was captured. The log history shows that the assigned English agent is transforming from the state of Monitoring to Reaching-Boat to Greeting and finally Escorting their client back to the fort. In Fig. 18, the supervising incident was captured. The assigned Chinese agent was changed from the state of Standing to the state of Entering, to the state of Wandering to the state of Leader-Following and finally to the state of Supervising at the container.

The validation of the state-machine, is not exhaustively done but one conclusion can be drawn based on the four sample cases that is the virtual agents are making unpredictable decisions from time to time to improve the realism of the social behaviors among the different groups of crowd. In most of the interactions, both individual agents and a group of agents of the same ethnic or group have to make a decision that will influence either their local or the entire global interactions. Besides, the decision made is the most impactful on the other virtual agents within their determined neighborhood. For instance, a role of the thief is to steal something from the seller. However, they have decisions to make before stealing as illustrated in Fig. 19. This is evaluated through visual observation and validated through the compilation of summarized log file that displays the history of the activities of the agents. When a seller persuades the potential buyer to interact, the thief will most likely answer the interaction so that they stand a chance to steal when both agents come near together. However, they can also refuse to interact sometimes especially when there is a police within their Field of View (FoV). The normal interactions between a seller and a potential buyer will stop eventually and soon after they will leave the spot. However, for the thief, the probability of them stealing is based on the availability of the nearby police, they will flee from the seller. The seller on the other hand will demand for the nearest police from the scene to chase after the thief. Therefore, the decision making of stealing determined by the will to steal is important in computing the next interactions, and this is possible through the proposed hierarchical state-machine used by heterogeneous crowd. The semantic of the responses is programmed in such a way that when a virtual agent decides to steal at this time-step of simulation and decides not to steal at the next time-step, the interactions between agents will differ at each simulation and thus, giving the simulation a more interesting interaction scenes.

Fig. 19
figure 19

Decision Making of Will to Steal

Full size image

5.4 Evaluation of group formation model using scenario-based visual observations

In crowd simulation and rendering, various types of steering behavior are deployed such as repulsive bound, wall following, containment which is a target behavior that restricts the flockers to stay within the bound, alignment which is a cooperation behavior based on average velocity, cohesion which is also a cooperation behavior based on average position, separation which is a competition behavior based on safe distance, obstacle avoidance of agent with other agent and agent with object which consider the neighborhood, path-following based on waypoint where dot product is utilized to seek for the waypoints, group behavior of queuing and arriving where the ratio between the distance of two agents with the arrival distance is used to determine the strength of the force, seeking which is a steering behavior of taking pursuit as the target, fleeing behavior which is steering away from its pursuit, other individual behavior like wandering where the angle and radius of wander is determined based on different situations, leader following which utilizes the right and forward offset for placing the followers, evading which is the behavior of rapidly escape and avoid the predator and lastly the escorting behavior where the followers are side-by-side with the leader.

Table 9 shows the group formation resulting from the group interactions that are composed of several agents from different hierarchy levels and group of populations. These group interactions are intended to test the proposed leader-following algorithm’s ability to handle some common situations in a trading port. Besides, a testing is also carried out to find out the realistic and reasonable number of followers for the cases of social behaviors in the simulation. It is known that a group of 2–3 people is the most suitable and has reasonable degree of freedom between each agent [35]. In the simulation, there is also a complex group formation from the group of English colonists and Chinese traders. In the simulation, the group formation of the Chinese group is best with eight followers while the English troop is best with only four followers, considering the scale of the terrain rendered. Besides, each leader has its own way- point side-by-side until they reach the fort. This is inspired by the work of [22, 65]. By having multiple waypoints, it can be a simple solution to deadlock for a big group of 14 peoples. Table 10 shows the large-scale scenarios that are designed to highlight the ability of the proposed group formation model where the algorithm is able to handle situations with large number of agents corresponding to the size of the terrain.

Table 9: Group Interactions. Group Formations that are Composed of Several Agents from Different Hierarchy Levels and Group of Populations.

Table 10: Large Scale Scenarios. These Scenarios are Used to Predict and Test the Reasonable Number of Agents Handling Macroscopic Situation.

Table 9 Group Interactions. Group Formations that are Composed of Several Agents from Different Hierarchy Levels and Group of Populations
Full size table
Table 10 Large Scale Scenarios. These Scenarios are Used to Predict and Test the Reasonable Number of Agents Handling Macroscopic Situation
Full size table

6 Conclusion

In bringing past life into digital cultural heritage, this research has proposed heterogeneous crowd simulation method with high-level parameter controller, hierarchical state machine and group formation. Parameterized agents are used to adapt the social behaviors of the trading processes. The scenario presented in this paper is the first attempt in modeling multi-cultural trading in Weld Quay in Penang, Malaysia, back in the nineteenth century. It involves also the modeling of heterogeneous crowd that has more than two groups of complex populations. There were four ethnic groups and different congregations of boats being introduced in the simulation. The examples of simulated scenes are presented based on four meaningful scenarios: working, trading, robbery and other interactions such as conversation, argument and negotiation.

A high-level controller scheme proposed in this research is to build a simulation platform that enables parameter adaptations of specific agents’ based on the incorporation of Reynolds’ and Helbing’s models. This research is not concerned with scientifically life-like or realistic crowd simulation, as no motion-captured data are available. It is about believable production of scenes where the behaviors are shown through the interaction models. The interaction models are scripted as a rule-based interaction between the agents. The communications among the agents were animated. By applying the classical models, it resulted in the velocity- and force-based interactions. These interactions can be changed and adapted into virtual agents’ behaviors by a set of parameters through the high-level controller in real-time to influence the events. The global events that triggered the changes of local and global behaviors of the virtual characters have been shown to be successful.

In the Helbing [21] model, there are social-force model, attractive and repulsive model as well as prisoner theory that are mainly used to display evacuation, panic and turbulence scenarios. In this research, a modified neighborhood, vision and density-speed models have been included into the particle-based steering behaviors to show the trading scenes and the activities involved. The models are the neighborhood model which includes the separation, alignment and cohesion, the vision models which uses the vision angle and vision distance to control the wandering behavior, the density-speed control model in twigging the local and global responses and the rest of the controls are based on parameter adaptation through the inspector of the simulation. Within the neighborhood model, it is speculated that there exists cooperation behavior among the English soldiers and captains while the competition behaviors exist among the Malay sellers who are trying to approach their targeted potential customers on the nearest distance basis. The non-linearity solution by having zero and one bits as triggered and non-triggered path-points is also introduced to control the trajectories of the crowds.

Besides, the proposed hierarchical state machine has eased the job of the graphics designer in re-tracking the history of the activities of each virtual agent. Moreover, the proposed group formation model has displayed the ideal result of the escorting behavior in the past at an old trading port. In term of motion of an idealized agent, the simulated autonomous agents have been confirmed to be able to perform action selection, steering and flocking as well as locomotion. The computation by having more than one behavior to be computed at each frame has been also optimized to give smooth effects to the behavior of the proposed particle-based models.

7 Future work

For future work of bringing past life in general, it is good to include a more complex system such as chaotic system and economic model where feedback from the virtual agents are collected to determine their emergent behaviors. For instance, if the demand of the customers increases, the sellers will become more aggressive in searching for the potential customers. Besides, there can also be exhaustion model where the worker moves slower over times. The current model is simulated with four populations: Indians, Malays, English colonists and Chinese. Future scenarios might include user controls. Besides, for the optimization of real-time simulation, it can be switched alternately with the crowd patches techniques where the less deterministic scenarios are simulated in real-time, while the more deterministic scenarios have its predetermined trajectories through crowd patches such as in Ramirez et al. [41].