1 Introduction

Animated virtual characters play an important role in numerous applications including video games, film and virtual reality. The motion of these virtual characters is as important in determining the perceived quality of an application as the model fidelity or rendering quality. In film, dangerous stunts such as jumping down head first from a staircase are not suitable for motion capture, but can easily be performed by a virtual stunt man. For animators, designing motion for each character in each specific situation is a laborious task, necessitating the development of automated tools for synthesizing motion.

Synthetic motion has a long tradition in computer graphics, the most representative work being Space-Time Constraints [84], Physically-based Approaches [53], Motion Retargetting [23] and Motion Graphs [34]. Each of these approaches deals with a subset of the overall problem of motion synthesis. Motion synthesis techniques need to not only generate natural looking motion for a given character, but also exhibit a level of adaptation of the motion to changes in the character or to the environment. In recent years, this specific problem of adaptive motion synthesis (AMS) for virtual characters has been of particular interest to the research community, as exemplified by Fig. 1a.

Fig. 1
figure 1

Both plots demonstrate the increasing popularity of hybrid methods in recent years. Another observation is that more adaptations in recent works are required in a user-specified task with more complex environment context, such as in [3, 7]. We also present three different directions for future research, which are discussed in Sect. 7.2. a A plot of the publication frequency of Adaptive Motion Synthesis for virtual character in IEEE Transactions on Visualization and Computer Graphics, SIGGRAPH, EUROGRAPHICS, SIGGRAPH ASIA and ACM Transactions on Graphics, EUROGRAPHICS/SYMPOSIUM ON COMPUTER ANIMATION over the last two decades. b Selected publications in adaptive motion synthesis. Different colors represent different categories of methods (blue example-based, green simulation-based, red hybrid)

Full size image

The ultimate goal for AMS is to generate natural motion that can interactively adapt to a specific character and environment while satisfying user-specified goals. There are two distinctive types of motion adaptation:

  • Character adaptation: characters may vary in all aspects, for example body proportions and weight distributions. Characters are not restricted to being humanoid, so a motion synthesis technique accounts for changes in geometric and skeletal topology.

  • Environmental adaptation: Environmental changes include external perturbations, such as being pushed or walking in strong wind, as well as changes in the terrain. We also include the ability to specify motion targets, such as reaching for a specific object or walking along a predefined path.

Our classification will be further explained in Sect. 2.

The goal of this survey is to classify and compare existing techniques of AMS. This excludes a large body of work on motion synthesis without considering adaptation. For general discussions on motion synthesis, readers can refer to a state-of-art survey in simulation-based approaches [21] and example-based approaches [52]. With specific focus on motion adaptation, we classify techniques of adaptive motion synthesis into three categories:

  • Example-based: This includes classical constraint-based approaches and more modern approaches utilizing sophisticated statistics models, such as Bayesian Networks, to adapt synthetic motion to specific situations, or conform to characters with geometry variations. Example-based methods are dominant in industrial applications for their computational performance and naturalness but at the price of constraining their adaptation scope to motion library.

  • Simulation-based: Physics simulation is essential in motion synthesis because of the flexible level of control which can be applied to virtual characters, and real-time adaptability to environmental perturbations. To broaden the adaptation scope, abstraction of character and actuator modelling is often implemented. Meanwhile on the controller side, active adaptations are introduced to meet the requirements of challenging environments.

  • Hybrid: It is an interesting discovery that simulation-based controllers generally produce motions with good adaptation to perturbations but poor adaptation to motion styles, while example-based controllers vice versa. Therefore, researchers also experimented the possibility of integrating both to generate natural motion without sacrificing the controllability.

The aforementioned methods will be extensively discussed from Sect. 3 to 5. Each category will be assessed by its adaptivity—the capability to adapt to changes in the character and the environment, which is the core comparison in this survey, and also its naturalness—a subjective comparison of the synthetic motion against the equivalent organic dexterous action. In addition, methods to generate synthetic motion differ in the level of controllability—the possibility of direction by the user. Additionally in Sect. 6, we first provide readers with an overview of optimization strategy, which is widely implemented in existing works, and later present two different perspectives of tackling this problem—biological inspirations and industry implementation. In Sect. 7, we conclude with a summary and our assessment on future directions in this field.

2 Categories of adaptation

In this section we clarify our classification system for different types of adaptation. By adaptive we mean that the method used is capable of adjusting to unpredicted external environmental perturbations or changes in the character, while still maintaining stable and realistic synthetic motion. We classify all related works in the field according to this classification in Table 1.

Table 1 A categorization of prominent literature in the field of adaptive motion synthesis for virtual character according to the classification in Sect. 2
Full size table

2.1 Adaptation to different characters

Characters may not be humanoid, but can extend to any creature or object capable of conveying human-like expression and emotion through linguistic and body language [87]. Non-human characters are common in many applications, where characters can range from organic creatures, like animals and aliens, to man-made objects, such as cars and toys. Taking this into consideration, the adaptation to different characters can be categorized into two aspects—geometry and topology.

2.1.1 Geometry

When the character is expressed as a hierarchy link of joints, geometry variations refer to different length or mass of the skeletal links, which does not influence the overall topological structure. As the earliest form of motion adaptation, there exists a wide repertoire of techniques being able to adapt motions to characters with different geometrical properties. Example-based methods, such as motion retargetting [10, 23] and Dynamic Filtering [73], generally achieves this goal by minimizing the differences between original and synthetic motions while satisfying user-specified constraints. Simulation-based and hybrid methods are possible solutions but require retuning the control parameters or extra transformations [42, 80, 81].

2.1.2 Topology

Two characters are identified as topologically different if they do not share the same skeleton structure, for example, a biped and a quadruped. Alexander [5] believes that it is possible to deduce information about plausible modes of locomotion just by observing that animal’s musculoskeletal structure. Since characters are not limited to biped creatures, topologies may be different for characters in the real world, and even more complex for a virtual character (for example, Fig 2). Researchers also explored solutions to characters of different topology, including example-based methods [26, 59] and simulation-based methods [13, 75, 78]

Fig. 2
figure 2

The motion data captured from actors was retargetted onto a penguin and squirrel. Image from [87] used with permission

Full size image

2.2 Adaptation to different environments

Adaptation to changes in the external environment are composed of three aspects: perturbations, changing terrain and user-specified goals. This adaptation is important in game design, in which the characters should be able to recover themselves from external perturbations, react in correspondence with different and changing environments or grasp, strike or dodge specific targets.

2.2.1 Perturbations

When the character encounters an external perturbations, such as a destabilizing force, the adaptation is successful if the character could eliminate the effects caused by the perturbations and maintain the stability. It is a challenging problem to make corresponding edits on existing motion given arbitrary perturbations, while still maintaining the naturalness. Physics simulations are suitable for generating responsive and stable motion. For example, the Inverted Pendulum [45] is a popular solution for dealing with external perturbations effectively.

2.2.2 Terrain

Locomotion is the most common motion in animal activity, and adaptation to variation in terrain is essential in synthesizing a natural walking motion. Simple terrain may vary in the angle of the slope, while complex terrain includes stairs and irregular ditches. Adjusting the width of the foot stride and maintaining balance is vital for adapting to terrain variation [14]. Dynamic constraints, particularly zero momentum point (ZMP) [62], is widely used to cope with changing terrain.

2.2.3 User-specified goals

Another major input from external environment for virtual characters is from user commands. The interactions with objects and other virtual characters can be generally specified with user input. Interacting characters have been explored by [65, 71, 77] in the context of competitive gaming. Also many researchers have developed methods to allow the user to interact with objects, such as grasping or kicking targets, or avoiding dangerous targets. Recent research also explored the possibility of adapting virtual characters to concurrent tasks [7].

3 Example-based methods

Example-based techniques are dependent on an existing motion library, which ensures the naturalness of the synthetic motion, but constrains the adaptation scope. The techniques in this area can be divided into two subcategories: constraint-based and statistics-based. The former is the most classical technique in this area and still dominant in industry applications, while the latter models the motion library with statistics model, which serves as decision-making mechanism and adapts characters’ motion to the external situations.

3.1 Constraint-based

Constraints provide intuitive and high-level control over the motion of virtual character, especially the end-effectors [53, 84]. Constraints are effectively the interactions between the characters and environments, which users intend to preserve during adaptation.

Motion retargetting, as the most popular approach in this area, adapts motion from a source character to a target one [23, 39]. The technique could be summarized as inverse kinematics (IK) solvers integrated with an objective function optimization for space-time constraints. Constraints are employed to define specific features of the motion that must be maintained and the objective functions are designed to penalise unacceptable changes.

The configuration of an articulated figure is specified by a position for the root of the hierarchy and the angles of its joints, denoted by vector q. The original motion is referred to as \(\mathbf {q}_0(t)\) and the retargetted motion as \(\mathbf {q}_1(t)\). The motion displacement \(\mathbf {d}(t)\) represents the difference between two motions, e.g.

$$\begin{aligned} \mathbf {q}_1(t) = \mathbf {q}_0(t) + \mathbf {d}(t), \end{aligned}$$
(1)

For retargetting, a simple objective is “minimizing the amount of noticeable change” and the most basic objective function is the similarity of the pose in parameter space. For example,

$$\begin{aligned} g(\mathrm{q}) = \int \limits _0^t(\mathbf {q}_1(t) \mathbf {q}_0(t))^2 = \int \limits _0^t\mathbf {d}(t)^2 \end{aligned}$$
(2)

minimizes the magnitude of signal differences of the motions over time.

In constrained optimization framework, adaptations, such as point-to-point attachment, are modelled as equality constraints and adaptations, such as joint limits, are modelled as inequality constraints:

$$\begin{aligned} \mathbf {C}(\mathbf {q}, t)&= 0\end{aligned}$$
(3)
$$\begin{aligned} \mathbf {C}(\mathbf {q}, t)&> 0, \end{aligned}$$
(4)

Dynamic filtering is a per-frame Kalman filter framework which solves spacetime constraints. The framework is composed of two parts: the unscented Kalman (UK) filter and the least-squares (LS) filter [73]. The UK filter accepts the original motion data and constraints as input, and estimates an optimal pose for the current frame that conforms to the given constraints. The result is fed to the LS filter to resolve inter-frame inconsistencies.

When adapting an existing motion sequence to new characters or environments, the artefacts of foot-skating and object collisions generally exist and impairs the visual credibility of the synthetic motion [8]. Regarding the categories in Sect. 2, the introduction of constraints is especially useful for adaptations to characters with both geometrical differences [23, 55] and environments with varying terrains [3, 70], and also useful for authoring users’ objectives, such as grabbing a cup [7].

Later researchers expanded the territory to characters with different topological structures [26, 59]. A morphology-independent representation of characters’ skeleton is often required in order to adapt to varying topologies. Gaits are synthesized by authoring the constraints between abstract legs/limbs and ground. Constraint-based techniques can also help virtual characters adapt to a changing environment. [39, 70] demonstrated a gait generation technique which produces kinematic motion of human walking on a variety of different terrains. A recent study takes a sequence of motion fragments as input and “squeeze” them into a highly-cluttered environment by satisfying four different types of constraints [11]:

  • continuity constraints to ensure the motion’s continuity at two connecting sequences.

  • non-penetration constraints to avoid collision with obstacles in the virtual environment.

  • contact constraints to specify the interaction between characters and the environments.

  • motion constraints to accordingly deform each motion fragment.

Different from previous methods (handling a sparse set of positional constraints), recent work proposed more advanced representations to describe the spatial relationship between the environment and character body parts [3]. The position of body parts is described as the weighted sum of translation vectors relative to selectively sampled points on the mesh surfaces of the environment objects. Using this representation, any deformation of these objects could reconstruct the character’s pose accordingly while preserving the original context of the scene. This method is able to achieve adaptations to different environments and characters with different geometrical properties.

3.2 Statistics-based

Instead of modifying the motion by constraining specific joints, statistics-based methods model the high-level character behaviour based on a large motion database. This technique alone is generally not enough to adapt motions to either different characters or environments, but it is useful when users’ intention is to generate motions with variations [38], or it can be combined with simulation components to make runtime decisions based on situational requirements [82] (This will be further discussed in Sect. 5).

The most popular decision-making networks in adapting character animation in varying situations are Bayesian Networks (BN) and Hidden Markov Models (HMM). A Bayesian network (BN) is a directed acyclic graph that represents a joint probability distribution over a set of random variables. Each node of the graph represents a random variable while edges represent the dependency relationship between these variables. To extend to situations where the stochastic states are not observable, researchers proposed HMM. In concise, HMM is a double-layered stochastic process with an underlying and not observable stochastic process, and an upper and observable one [22]. An HMM can model a walk cycle because it can model sequential stochastic processes, or states, where the probability of a state depends on previous states [2]. Readers may refer to [22] for more detailed introduction to Bayesian Network and Hidden Markov Model.

BN is widely used to synthesize motion for virtual characters because of a number of its attractive features [56, 57]:

  • The causal relationship between variables is intuitive for understanding given the graphical representation of the Bayesian Network.

  • This graphical representation also reduces memory overhead, especially when the dependencies in the joint distribution are sparse.

  • The structure of a Bayesian Network is based on existing data and can be updated dynamically.

  • A Bayesian Network handles missing observations by calculating marginal probabilities conditional on the observed data using Bayes’ theorem .

A representative work [38] learns a BN model from a few examples of a particular type of input motion. Here the dynamic BN defines a joint probability distribution over X[0],..., X[T]:

$$\begin{aligned} P(\mathbf X [0],\ldots ,\mathbf X [T])&= P_{G_\mathrm{prior}}(\mathbf X [0], \mathbf X [1])\nonumber \\&\quad \cdot \displaystyle \prod _{t=0}^{T-2}P_{G_\mathrm{trans}}(\mathbf X [t+2]\ |\ \mathbf X [t], \mathbf X [t+1])\nonumber \\ \end{aligned}$$
(5)

The prior network \(G_\mathrm{prior}\) represents the joint distribution of the nodes in the first two time steps, \(\mathbf {X}\)[0] and \(\mathbf {X}\)[1]. The transition network \(G_\mathrm{trans}\) (Fig. 3) specifies the transition probability \(P(\mathbf {X}[t+2]|\mathbf {X}[t], \mathbf {X}[t+1])\) for all \(t\), assuming a second-order Markov property applies to this network.

Fig. 3
figure 3

The illustration of Bayesian Network for variables \(X_1,\ldots ,X_n\). The prior network refers to the first two frames, and the transition network models subsequent frames given the previous two frames, which assumes this BN as a second-order Markov Model. Image from [38] used with permission

Full size image

The most significant disadvantage of an approach involving Bayesian Networks is the fact that there is no universally accepted method for constructing a network from data. In addition, the acyclic property of the standard Bayesian Network implies that feedback effects cannot be included in the network [57].

3.3 Evaluation

  • Adaptivity: Example-based methods are capable of adapting to characters with different geometry [23, 55] and topology [26, 35, 55] properties, varying terrains [3, 70] and user-specified goals [7, 11, 38]. By using constraints, foot-sliding artefacts and unwanted object collisions can be easily cleaned up. By using a statistical model, users can generate unlimited number of similar but different motions at ease [38]. However these methods are not appropriate when adapting to external perturbations, which requires delicate or unpredictable interaction. Because of the dependency on the original motion data, example-based methods also meet challenges when adapting to characters with large topological differences. Additionally statistics models alone cannot handle dynamics problems such as foot contact, which requires an extra post-processing step [30].

  • Naturalness: Example-based approaches are currently widely used in industry, due to the motion naturalness and real-time performance. In addition, the ability to generate a diverse range of motion styles by using statistical methods is of practical application to realistic crowd simulation.

  • Controllability: It is difficult to impose control over the motion after it has been captured. Due to its dependency on the database, the virtual character is difficult to control and its skill repertoire cannot be extended without collecting further motion data. Because of the non-linearity of the statistical methods, it is difficult to generate the specific desired motion.

4 Simulation-based methods

In recent years, physics-based simulation has become important in motion synthesis research as it allows motion to interactively adapt to a changing environment. As our review focuses on adaptation to different characters and environments, readers could refer to [21] for a state-of-art survey in animating virtual characters using simulated physics.

According to [21], a simulation framework is composed of three parts: simulator, character and controller. In the following section, we would focus on character modelling and controller design, which are most related with the adaptation to different characters and environments.

4.1 Character modelling

The task of character modelling involves two parts: body and actuator modelling.

Body modelling in simulation-based methods mainly refers to designing a hierarchy of rigid bodies, each including geometry and inertial properties, the connecting joint type (such as hinge joint) and joint properties (such as joint limits) [21]. In most cases, this hierarchy is designed to follow the physiological properties of real equivalents, such as bipeds or quadrupeds. However in some cases, simplified physics model (such as Inverted Pendulum [76]) is intentionally selected to drive the motion of complex characters. The reason for this choice is:

  • The abstraction of complex physical models to simplified ones often makes the same controller adaptable on different characters, even for those with significant topological differences [13, 41].

  • The simplified models also reduce the computational burden and speed up the performance, which is of vital importance for realtime applications [76].

The designs of abstracted models are determined by the features which controller intends to manipulate. Features of characters’ motion describe a set of high-level, physically-related properties of character state, such as trajectories of Center-of-Mass or end-effectors [36]. Specifying the variation in features provides an alternative tool to author the desired motion.

Actuator modelling defines the mechanism by which controllers apply force and torque on the hierarchy of rigid bodies. As is also pointed out in [21], there are three types of actuators: muscle-based [81], servo-based [93] and virtual force [54]. Similar to the abstraction of body model, researchers also introduced abstract actuator (such as virtual model control (VMC) [54]) to model general actuators. VMC uses simulation of virtual mechanical components to transform generalized virtual forces \(\mathbf {f}\) into the actuator torques \(\varvec{\tau }\) (or forces) with a Jacobian matrix \(J\) (Eq. 6):

$$\begin{aligned} \begin{aligned} \varvec{\tau }&= J^T \mathbf {f} \\ \mathbf {f}&= \mathbf {f(q,} \dot{\mathbf {q}}\mathbf {)} \end{aligned} \end{aligned}$$
(6)

By designing the virtual component, this technique is intuitive to compute the actuator torques required by high-level control tasks, especially in controlling the relative movement of end-effectors. This is especially useful for controller designs which are capable of adapting to external environments with end-effectors interactions. More detailed examples may include balance control [54], steering [15] etc.

4.2 Controller design

For abstracted character models, the controller is able to adapt to characters even with topological differences. However for most simulation approaches, the dynamic character is modelled to mimic its real equivalents (for biped, that is human body). Therefore, adaptation for simulation methods mainly refers to its ability to adapt to varying environments, which is advantageous when producing interactive motion.

The dynamics of a set of linked bodies can be formulated as:

$$\begin{aligned} \mathbf {M}(\mathbf {q})\ddot{\mathbf {q}} + C(\mathbf {q}, \dot{\mathbf {q}}) + h(\mathbf {q}) = \mathbf {f}_{ext} \end{aligned}$$
(7)

where \(\mathbf {q}\), \(\dot{\mathbf {q}}\) and \(\ddot{\mathbf {q}}\) represent the joint angle poses, velocities, and accelerations respectively. Parameters \(M(\mathbf {q}), C(\mathbf {q}, \dot{\mathbf {q}})\) and \(h(\mathbf {q})\) are the joint space inertia matrix, centrifugal/Coriolis and gravitational forces respectively. The controller applies the joint torque (or direct force), which is included in the term \(\mathbf {f_{ext}}\), to generate the desired acceleration and trajectory.

The representative work in simulation-based techniques is Simbicon [93]. The character is modelled as a typical hierarchy of linked rigid bodies and the actuator is modelled with Proportional-Derivative (PD) servo to track target angles for individual DOF. The whole motion cycle is divided into two or four states and target angles are specified for different states with the help of Pose Graph. Priori information from motion capture data is used to set the objective value for the individual PD servo. The resulting framework is able to react against external perturbations and simple terrain variations.

The main contribution from Simbicon is its introduction of an additional balance controller which is simple and capable of adapting to changing environment and external perturbations. The target angles in PD controller for both torso and swing hip are explicitly specified in world coordinate, which ensures the character’s balance by directly controlling the torso and hip.

Inspired by Simbicon, later work pushes the adaptation boundary by employing a variety of improvements. Changes include the introduction of four additional walking states in the finite state machine (FSM) to take the preparation of lifting and striking into consideration, improving the naturalness [81]. Rather than simply setting the objective value for PD servo, motion priori provides a more sophisticated, optimization-based motion controller (discussed in Sect. 6.1). The implementation of virtual model control helps adapting to non-locomotive tasks, which can be used to adapt to simultaneous high-level control [15].

A recent development replaces the PD servo with biologically-inspired muscle unit to actuate joints [81], which improves the naturalness and stability and provides comparison against the performance driven by real muscle. While existing work only controls the lower-body joints with these muscle units, we expect to see the simulation of the full body controlled with biologically-inspired principles.

Relating to the adaptation categories in Sect. 2, simulation-based methods come with innate capability of adapting to external perturbations and environments [15, 20, 81, 93]. In order to adapt existing controllers to characters with different geometry or topology, re-optimizing parameters inside controllers is needed [20, 25, 81]. Additionally, the standard controller may only be able to adapt to terrains in a moderate sense (such as a small range of gradient). Researchers have introduced the local and global transformation operators on the dynamics equations governing the character’s gait, which is then able to make the controller adapt to more challenging terrains [42]. Pure simulation-based methods also find difficulties in authoring user-oriented tasks, since different task requires individual controller [16]. Motion examples are often useful in designing task-specific controllers, leading to our discussions on hybrid methods in Sect. 5.

4.3 Evaluation

  • Adaptivity: Simulation-based methods are able to adapt to perturbations and terrains in an automatic fashion, however a pure simulation-based method often produces rigid animation [93], which explains the main differences of techniques between character motion synthesis in computer graphics and computer simulation in robotics. Pure simulation-based methods also find difficult to adapt themselves to characters with significantly different geometries or topologies. Abstracted model (such as Inverted Pendulum) and abstracted actuator (such as Virtual Model Controller) prove to be useful [15, 76]. Because there is no universal controller that could produce every motion, new controllers are needed to add to the skill repertoire [43, 50].

  • Naturalness: Synthetic motion still looks comparatively unnatural. A simplified dynamics model facilitates controller design and accelerates computation, while helping motion to adapt to different characters and environments [25]. However, this comes at the expense of more subtle motion expression, which is important for the visual perception of natural motion.

  • Controllability: Simulation-based methods have significant advantages with respect to controllability because the character will strictly follow the scripted trajectory [21]. Integration of motion priori will be useful in designing the controller but partially reduced the controllability over the synthetic motion.

5 Hybrid methods

Researchers also explored the possibility of combining these different components in Sects. 3 and 4 to take advantage of both motion data and physics simulation. The specific recipe for this combination varies case by case, but in general there are three typical solutions which are discussed below. Relating to categories of adaptations in Sect. 2, each solution is more suitable for one kind of adaptation or another.

5.1 Tracking the reference motion

Tracking the reference motion, or trajectory tracking, is a standard technique in robotics control. The success of Simbicon relies on specifying a set of key-poses to provide input to its PD servos, however the sparsity of the key-poses requires the high frequency of simulation steps [93].

By using a large library of motion library instead of key-poses, there are two significant advantages [76]:

  • allowing low-frequency tracking with increasing number of sample points (as key-poses in Simbicon)

  • enabling quick design of tracking controller in an more automated method

These two are both crucial to design a controller which is able to adapt to different environments, especially in industry applications (more details in Sect.  6.2).

Reference motion and online simulation may often stay out of phase, especially in the situations of different characters or environments. To adapt to these cases and maintain synchronicity, the controller needs to detect foot-strike events in simulation side and compare against the reference trajectory. A solution is proposed from [40]: when the foot-strike happens earlier in a simulation process than reference trajectory \(\mathbf {M}\), the rest of \(\mathbf {M}\) is dumped and the current pose is smoothly blended towards the next reference trajectory; when the foot-strike happens later in a simulation, the current reference trajectory is extended by applying constant velocities at the end of \(\mathbf {M}\).

Relating to the adaptation categories in Sect. 2, motion tracking is useful in cases of user-specified tasks and moderate ranges of perturbations [40, 76]. By combining the simulation and reference trajectory, the character is able to perform natural movements and recover from a small perturbation. However the motion generated in this way stays close to the original motion, otherwise artefacts can be easily spotted. Because the introduction of reference trajectory requires a strict topological similarity between the animated character and original actor performing the reference motion, these methods are difficult to adapt to a different topology. In recent years, a number of works [59, 87] are able to kinematically animate non-humanoid characters with human motion data, which can be further combined with motion tracking to drive the character in a simulation-based fashion (as in [76]).

5.2 Dynamic selection of motion examples

Trajectory Tracking combines the motion example and simulation process in the same time window, while the second hybrid formulation is to switch between these two at different time windows [95, 97]. While in undisturbed cases, the character will perform the example motion in a kinematic fashion, which is straightforward and ensures the motion’s naturalness. When perturbation or control is introduced, the simulation component will be activated and the character is able to react interactively to the unexpected perturbations.

As was developed in [97], two key components are included in this system: a search engine to find the next motion clip after the simulation and a controller to drive the character state towards the target pose. Supervised learning is introduced in [97] to train the search engine, which is able to make online prediction of next motion. Blending can be done to generate the transition clip, which can be tracked with the techniques described in previous subsection. Trajectory tracking, as discussed in previous subsection, can be used here to drive the character towards the starting state of next sequence.

This technique serves as an intelligent decision-maker in the situations of emergency, and is able to adapt interactively to environmental perturbations (Sect. 2). Since the dynamic selection of motion examples do not alter the existing motion data, this technique alone can not adapt motion to other situations, such as characters with different topologies. This high level adaptation could also be further enhanced with additional constraints inside each motion clip, which may expand its adaptivity scope.

5.3 Integration with statistics model

In the work of [82], a non-linear probabilistic force field function is learned from pre-recorded motion data by Gaussian processes and combined with physical constraints in a probabilistic framework. A force field \(\mathbf {u}\) is defined as a non-linear probabilistic function that maps the kinematic state \((\mathbf {q},\dot{\mathbf {q}})\) to the generalized forces:

$$\begin{aligned} \mathbf {u}=\mathbf {g}(\mathbf {q},\dot{\mathbf {q}}) \end{aligned}$$
(8)

The force field prior is extracted from motion capture data with Newtonian dynamics:

$$\begin{aligned} \mathbf {u}=M(\mathbf {q})\ddot{\mathbf {q}}+C(\mathbf {q},\dot{\mathbf {q}})+h(\mathbf {q}) \end{aligned}$$
(9)

where the parameters notation follow the Eq. 7.

The Gaussian process model learns this force field from a training database:

$$\begin{aligned} pr(\mathbf {u}|\mathbf {q},\dot{\mathbf {q}}) = \varPsi (\mu (\mathbf {q},\dot{\mathbf {q}}), \sum (\mathbf {q},\dot{\mathbf {q}})) \end{aligned}$$
(10)

where both the means and the covariance matrix are functions of kinematic states \([\mathbf {q},\dot{\mathbf {q}}]\).

According to Bayes’s rule, the probabilistic motion model \(pr(\mathbf {x})\) is decomposed into the following three terms:

$$\begin{aligned} pr(\mathbf {x})&= \underbrace{pr(\mathbf {q}, \dot{\mathbf {q}})}_{pr_{ init }} \cdot \prod _{t} \underbrace{pr(\mathbf {u}^t|\mathbf {q}^t,\dot{\mathbf {q}}^t)}_{pr_{ forcefield }} \nonumber \\&\cdot \underbrace{pr(\mathbf {q}^{t+1},\dot{\mathbf {q}}^{t+1}|\mathbf {q}^t,\dot{\mathbf {q}}^t, \mathbf {u}^t)}_{pr_{ physics }} \end{aligned}$$
(11)

As is shown in this chain-like equation, physical and statistical motion models are complementary to each other. Statistical models ensure the naturalness of motion, while the physical components can equip the characters with the capability to react to external circumstances. Compared with previous example-based or simulation-based approaches, this hybrid formulation achieves wider adaptation in a single framework, including adapting to external force (originally from simulation side, see Fig. 4b) and performing stylized walking (normally generated with example-based methods, see Fig. 4a). More results are displayed in Fig. 4.

Fig. 4
figure 4

Results demonstrated by method in [82]: a walking with a heavy shoe; b resistance running; c stylized walking; d running\(\rightarrow \)walking\(\rightarrow \)jumping. Images from [82] used with permission

Full size image

This technique combines a sophisticated physics simulation with a statistical model. The former gives the user control over the virtual character, while the latter serves as the prediction mechanism and improves the naturalness of the synthetic motion. Thus, this technique can achieve a variety of realistic adaptations to perturbations that are not present in the training data, characters with varying properties and terrains [82]. Furthermore, due to the demand for increasing character’s behaviour repertoire, more sophisticated motion planning algorithms are introduced. For example, [71, 77] both aim to simulate the interactive behaviour between multiple virtual characters. To adapt to these highly complex situations, a hybrid formulation of both simulation-based and statistics-based modules is preferable over any single method. However, this technique still suffers from the inherited drawbacks from individual components that are only partially complemented rather than resolved.

5.4 Evaluation

  • Adaptivity: By combining the merits of individual approaches, a hybrid framework expands the adaptivity scope. It not only possesses the inherent feature of responsive adaptation to external environments from the simulation side, but also adapts to broader range of situations by using constraints to produce additional high-level stylized motion, or using existing motion to inform the complex design of simulation-based controller. Using statistical models as the underlying decision-making network enables complex user-specified tasks, including multi-player interaction [71, 77].

  • Naturalness: With minor editing on original motion data, this hybrid approach could improve the naturalness of the synthesized motion, which has been one of the drawbacks of simulation-based methods. Furthermore, constraints are employed to post-process any synthetic motion produced with statistics-based or simulation based controllers to remove any significant artefacts, such as foot-sliding.

  • Controllability: In hybrid approaches, the incorporation of dynamics enhances the flexibility of the control over the virtual character, which is difficult to achieve with example-based approaches alone, especially when the motion is generated with statistical network.

6 Discussion

6.1 Optimization

Optimization is a ubiquitous technique widely used in most of the discussed techniques, such as to adapt to constrained terrain [14], or to generalized motion skills for more complex tasks [92]. An optimization strategy is described as online or offline depending on whether the optimization is conducted at runtime or not. An online strategy is repeated to make realtime decision over an upcoming time interval and is suitable for problems where dynamic adaptation is required for unexpected environments and perturbations [86]. In this sense, the character is able to actively adapt to situational requirements, in most cases referring to the category of environmental adaptations in Sect. 2. However, an offline strategy generally runs beforehand to find an optimal set of control parameters, which minimizes or maximizes the objective functions. This aims to increase the scope that a controller can passively adapt itself and is mostly suitable for the character-related adaptations in Sect. 2.

When designing an offline strategy to optimize a controller, the user needs to carefully select the objective functions [4]. Some of the typical choices are to:

  • Minimize the difference between adapted and original motion as measured by joint angles or control parameters [14, 53]. This normally applies to the cases of adaptation where a few constraints are applied and this objective ensures the minimal deviation of the synthetic motion from the original one, by either kinematic shooting or trajectory tracking.

  • Maximize the stability of specific parts of the body, typically the center of mass (COM) and head [4, 79]. For example, the character’s heading direction is penalized when deviating from the desired direction [4]. Though stability is a general objective of controller design, it is vitally related with controller’s capability to adapt to environmental perturbations. This improves model stability, but also has a biological explanation—human physiology is designed to keep the head as stable as possible to ensure the continuity of sensory input.

  • Maximum accomplishment of a user-specified task. Contact locations between end-effectors and environments are commonly used to author the task and achieve the adaptation to the external world. As an example, in the problem of walking on stepping stones [14], a cost is assigned to stepping-location errors.

Initial conditions are crucial for the successful convergence of the optimization and motion priori is useful for this purpose [80]. A recent work presents the first method to optimize the full-body trajectory for physics-based character control without using any existing motion examples and is able to perform a wide range of motion skills [4].

Offline optimization is widely implemented in contemporary work to find the appropriate control parameters in simulation-based methods, but suffer from some significant limitations:

  • Computational cost: Optimization comes with extra computational cost. Off-line optimization [14, 53] searches the entire motion space and returns a globally optimal result, while on-line optimization [24, 33, 49] limits the search domain to local frames. A compromise is typically made to combine off-line optimization for learning or pre-processing with on-line optimization for interactive performance [90];

  • Objective function design: To cope with the expressiveness of character motion, the objective function is normally a weighted sum of several components [1, 41, 49, 80]. High complexity in the objective function will increase the possibility of finding local minima [17].

  • Local minima and convergence: Nonlinear objective functions in motion adaptation have a high probability of local minima, which can lead to failure of convergence. This problem can be alleviated by either using advanced optimizers with better robustness against local minima (for example Covariance Matrix Adaptation (CMA) [78]), or improving the initial guess from dynamics formulation [17] or motion priori [58].

Online optimization normally optimizes over a fixed time window to fit with performance requirements [85], but still suffers from the difficulty of designing objective functions and risks of falling into local minima.

In some problems of motion adaptation, the character needs to adapt to different situations simultaneously. A weighted sum of objective functions are not sufficient to rigorously meet the requirements. An alternative solution is prioritized optimization, which finds the solution minimizing each objective without conflicting with higher priority objectives. However, because each priority level repeats the optimization process, prioritized optimization is slower than weighted optimization. In addition it requires higher arithmetic precision than weighted optimization [49], which makes it less numerically stable. This approach also leads to the situation where low-level objectives are highly constrained.

6.2 Industry implementations

Character animation in industry applications, especially video games, requires all categories of adaptations presented in Sect. 2. For instance, a single set of animations is often adapted to many character rigs to improve variety in the game without significant additional authoring effort. For adaptations to different characters with different geometry, most character controllers in games rely on purely example-based methods [27]. Some adaptation may be achieved by manipulating blend weights in motion graphs [28] or transitioning between states in a state machine [50]. IK is frequently used to adapt animations to deal with uneven terrain or interact with objects. Variety may be introduced by additively blending a small amount of random motion.

Simulation-based methods are the growing force in the industry applications as it can provide runtime adaptations to perturbations and user controls. These emergent behaviours can add significantly to richness and immersion, but must be balanced with pragmatic concerns. A practical issue of implementation in game industry is that character motion generation runs alongside many other computationally demanding tasks and therefore has limited access to processing and memory resources (typically around 5–10 % Footnote 1). Many motion synthesis techniques described in this survey are only able to achieve real-time speed in isolation and have higher computational requirements, with resource utilization closer to (or even higher than) 100 % on representative hardware [36, 89], and may therefore not be immediately applicable to the game industry.

In the film industry, naturalness of the synthetic motion is prioritized over other features: characteristic of film animation is high frame rates, high resolution but no interactivity. A much higher computation and time budget in film production than game simulation also allows this pursuit for naturalness, which is against most of current techniques in producing adaptive character motions. Character animations on film screen are mostly manually created from scratch or based on motion capture data but still tuned by animators. However, with the improved performance of these aforementioned techniques, some are used in places with less strict requirements, such as previsualization [6]. Another application of adaptive motion synthesis techniques is crowd simulation, which is mostly concerned with duplicating the same behaviour across a large crowd of agents with same rig but different proportions. This is a similar as those encountered in the game industry when motion is retargetted from a general skeleton. Agents in the crowd often perform similar but slightly different animations, which could benefit from techniques like [38].

6.3 Motor control and adaptation

When it comes to the environmental adaptations from Sect. 2, we human beings or any other animals are actively performing the actual adaptation to the outside environment as we move. Motor Control is the process of organizing and executing animals’ actions, which is determined by the interaction between the nervous system and musculoskeletal system and involves sensory information as external input [37]. Motor control by its very nature is a complex process, involving electrical, chemical and mechanical changes. In spite of the complexity of this system, most motor control problems seem an easy task for animals and humans. The typical examples of adaptation are unconsciously transversing uneven terrains. Even the creatures with low-level neural network, such as the cockroach and ant, demonstrate significant flexibility in adapting to uncertain and dynamic environment. It naturally leads us to believe that the mechanism of motor control and adaptation is the potential solution to solve the bottlenecks of existing techniques. The following three models are selected to give readers insight into the question how animals achieve the goal of adaptation.

Equilibrium point hypothesis proposed that the neural system does not control the entire process, but rather only the final motion result—as long as the motion completes the specified task it does not matter how the action is executed [18, 37]. This allows the motion to freely adapt to the environment and body conditions, greatly simplifying computation while maintaining energy efficiency. A consequence of this theory is that animals do not move the way they want, but rather the way they can.

Internal models are discovered in biological systems to use learned simulations of their dynamics to accomplish feed-forward control [74]. There is also evidence that internal models adapt to environmental influences. For example, in a task involving guiding the hand to a target, hand trajectories are initially disturbed when external force fields are applied, with the disturbance diminishing over time, and equal and opposite displacements are initially observed when the external force is removed, suggesting that trajectories are planned in advance using internal models which are adaptive [60]. Similarly, test subjects were found to exhibit timing errors when catching objects in zero gravity [46]. The errors were consistent with the brain using an internal model with a second order component corresponding to gravity. The errors diminish with repetition of the experiment due to neuroplasticity—the remarkable property of the brain to self-organise based on neuronal feedback.

Reflexes, which are low level feedback mechanisms based on sensory input, are thought to be important in regulating motion and adapting to differing environments and unexpected disturbances. In common usage, the term “reflex” refers to an involuntary response to a stimulus, but the technical definition also includes more complex, modulated responses, which may be dependent on task, motion phase or stimulus intensity [94]. A variety of reflexes based on sensory input including muscle length, pressure on the skin and muscle loading are thought to contribute to stability and adaptation by stabilizing limb trajectories, regulating force production during stance, altering limb trajectories to avoid stumbling, and influencing timing of motion phases [94]. Spinal reflexes may also be important in regulating the force-length relationships of muscles in the equilibrium point hypothesis, reducing the high level control effort required.

Motor control has been employed in robotics research, where motion is generated to respond to the natural dynamics of the environment. Some examples of these include the \(\lambda \)-model [72], the central pattern generator [83] and Lie Group symmetry [69]. These approaches can generate adaptive motion for unpredictable environments without effecting the system stability. These methods are used to drive actuators of robots for use in the navigation of difficult terrain in real-time, and therefore presents a rich seam of unexplored technology which could greatly influence future directions of motion synthesis research.

7 Summary

7.1 Conclusion

This survey presents a broad range of research in the area of Adaptive Motion Synthesis for virtual characters. Motion Synthesis is the process of artificially generating natural motion for a virtual character. However, it is significantly challenging when the synthesized motion is required to be natural-looking while at the same time adapting to character variations and environmental factors. The virtual characters may vary in aspects of geometry and topology while the environmental factors come from perturbations, terrain and user-specified goals.

In this survey, three major categories of techniques are discussed in detail. Example-based methods are most prevalent in interactive applications because of the naturalness and expressiveness of the motion with the assistance of recorded data. However, this dependence restricts the flexibility, controllability and repertoire of these methods. Simulation-based methods ensure the physical plausibility of the motion and provide control over the character and synthesize responsive motion to external environments, but they have noticeable disadvantage of the poor naturalness and also limited adaptation to different motion styles. The hybrid combinations of simulation-based and statistics-based methods serve as ameliorated solution, but without a fundamental breakthrough in one of the principle approaches, possibilities for improvement may be limited. Additionally, light is cast upon two novel and interesting areas—the practical application in industry and promising solution derived from biology.

7.2 Future directions

Although much research has been conducted on the methodology and application of the Adaptive Motion Synthesis for virtual characters, several issues have not been fully addressed and as a result there are numerous directions which could potentially lead to significant advances in this field.

Currently these techniques are still developed by the academics and far from the actual industry implementation. For the state-of-the-art research, many techniques are still not running at realtime speed or require heavy offline optimization [36, 81, 89]. A suggestion is to apply dimension reduction techniques, such as aforementioned GPLVM, to make current techniques faster, which could lead to feasible industry applications. There is also a trend in this area to synthesize delicate motion, for example hand animation [32, 91]. These body movements are essential to achieve successful adaptation to environment and enhance the naturalness of synthetic animation.

Current research trends also include multi-player interaction, which can be also regarded as a category of adaptation [29, 65, 77]. Compared with static adaptations to either characters’ properties or environments, the active adaptations to other characters is more challenging. The success of multi-character interaction relies on an additional level of decision network, which is more closely related to Artificial Intelligence than the underlying mechanism of motion synthesis. However, as this will significantly enhance a user’s experience, we believe this will be another fascinating and important direction to explore.

To improve naturalness without sacrificing the interactive performance, existing methods are far from sufficient, which demands introducing techniques from related subjects, such as the Central Pattern Generator [25, 72] and Passive Dynamic Walking [69] from biology and robotics. Novel simulation models are also expected and some have already demonstrated their capability in producing motion with wider adaptation and naturalness, for example the Hill-type musculotendon units [20, 81]. We are also expected to see more sophisticated techniques from the area of Machine Learning, such as Neural Networks and Deep Learning, to process large stores of motion capture data.