Efficiency and Precise Interaction for Multibody Simulations in Augmented Reality

Abstract

In this chapter, an enhanced methodology for interactive, accurate, fast and robust multibody simulations using Augmented Reality is presented and discussed. This methodology is based on the integration of a mechanical tracker and a dedicated impulse based solver. The use of the mechanical tracker for the interaction between the user and the simulation allows to separate the processing of the data coming from the position tracking from those coming from the image collimation processing. By this way simulation results and visualization remain separated and the precision is enhanced. The use of a dedicated sequential impulse solver allows a quick and stable simulation also for a large number of bodies and overabundant constraints. The final result of this work is a software tool able to manage real time dynamic simulations and update the augmented scene accordingly. The robustness and the reliability of the system will be checked over two test cases: a ten pendula dynamic system and of a cross-lift mechanism simulation.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

During the last years, many investigations focused on the increasing trend of using Augmented Reality (AR) [1] to support a variety of engineering activities and to develop interactive tools in design. Augmented Reality has been used for supporting geometrical modeling [2], reverse engineering [3], assembly simulation [4–7], analysis [8].

The augmented reality deals with the combination of real world images and computer generated data. Most AR research is concerned with the use of live video imagery which is digitally processed and augmented by the addition of computer generated graphics. The purpose of the augmented environment is to extend the visual perception of the world, being supported by additional information and virtual objects. One of the most important feature of AR is the possibility to embed an high level of user’s interaction with the augmented scene [9].

Some recent contributions showed an increasing interest in developing environments for simulating and reviewing physical simulations based on Augmented Reality [10–14]. They focused on the interactivity between the user and the augmented environment. By this way, the user is not a mere spectator of the contents of the augmented scene, but influences them. Interaction in dynamic simulations can concern both the definition of boundary conditions and initial parameters and the real time control of the process.

In a recent paper [15], P.P. Valentini and E. Pezzuti developed a methodology for implementing, solving and reviewing multibody simulation using augmented reality. According to their results, augmented reality facilitates the interaction between the user and the simulated system and allows a more appealing visualization of simulation results. For this reason, this approach has revealed to be suitable for didactical applications and teaching purposes as well.

On the other hand, the development of multibody simulations in augmented reality requires very fast solver in order to produce a smooth animation and an effective illusion. Valentini and Pezzuti proposed to use an optical marker to track the position of the user and interact with the objects in the scene. Moreover, they developed some simple examples to introduce the methodology.

Starting from the discussed background, this chapter aims to discuss two important enhancements of the work in [15] in order to improve the accuracy of user’s interaction and to allow a robust simulation of large multibody systems. The accuracy in tracking the user is important to perform precise simulation that are required in many engineering applications. The use of optical marker is simple but in this case position tracking error highly depends on the resolution of the camera sensor and on the distance between the camera and the marker. Using standard USB cameras, this error can be some millimeters and can be unacceptable for precise applications, and unwanted flickering may occur due to the tracking algorithm limitations [16]. In order to improve the accuracy in tracking, in the present study we have included the use of a mechanical instrumented arm which is able to achieve a precision of about 0.2 mm in a working space of about 1.2 m of diameter. This enhancement is also important to separate the processing of the data coming from the position tracking from those coming from the image processing (for perspective collimation and visualization issues). By this way, higher precision in the analysis results can be achieved and only the graphical visualization is affected by optical imprecision.

The second enhancement which has been proposed and tested, is about the use of a dedicated solver for managing the integration of the equations of motion. The implemented solver makes use of the sequential impulse strategy [17] which allows a quick and stable simulation also for a large number of bodies, in presence of overabundant and unilateral constraints. According to some authors and applications (i.e. [18, 19]), this approach leads to a very fast and stable solution, but quite less accurate than global solution methods.

The chapter is organized as follows. First of all, a brief introduction about the state of the art of virtual engineering in augmented reality is presented, focusing on the aspects related to multibody simulations. Then, a description of hardware and software implementations is discussed, including the explanation of the iterative impulse solver and the details about the strategy for implementing interaction between the user and the scene. In the second part, two examples are presented and discussed.

2 Multibody Simulations in Augmented Reality Environments

The first and basic implementation of multibody simulations in augmented reality is about the possibility to project on a real scenario the results coming from a pre-computed simulation. It concerns the rendering on the augmented scene of all the objects involved in the simulation whose position is updated according to the results of the simulation.

This implementation is similar to that of the common post-processing software for visualizing graphics results. The only difference is in the introduction of the simulated system in a real context. The advantage is to perceive the interaction with the real world and check working spaces, possible interferences, etc.

Although it can be useful, this approach does not unveil all the potential of AR [15]. A more powerful way to enhance multibody simulation is to introduce an high level of interactivity. It means that the user does not only watch the augmented scene, but interacts with it. In this case, the solution of the equations of motion has to be computed synchronously to the animation in order to populate the scene with quickly updated information.

With this type of interaction, the user is active in the scene and can change the augmented contents by picking, pushing and moving objects and controlling the provided information and the environment behaves according to realistic physics laws. The interaction is carried out with advanced input/output devices involving different sensorial channels (sight, hear, touch, etc.) in an integrated way [20]. In particular, in order to interacts with digital information through the physical environment, the environment has to be provided of the so called Tangible User Interfaces (TUIs) [21–23].

In the most simple implementations, the patterned markers used for collimating real and virtual contents are used also as TUIs. In advanced implementation visual interaction is achieved by dedicated interfaces (mechanical, magnetic, optical, etc.). By this way, the image processing for the computation of the camera-world perspective transformation can be separated from the acquisition and processing of the user intent. Thanks to this split computation, it is possible to achieve a very precise interaction and simulation and a less precise (and more efficient) visual collimation. This means that the results of the simulations can be accurate and suitable for technical and engineering purposes. On the other hand, the small imprecisions in the optical collimation are limited to graphics display.

Starting from these considerations, the generic integration algorithm between multibody simulations and augmented reality presented in [15] can be modified separating the contributions of interaction and collimation. For this reason, an high-interactive generic multibody simulation in augmented reality can be implemented following five main steps:

• 1. Before the simulation starts, the geometries and topological properties (joints and connections) have to be defined (as for any multibody system);

• 2. The real scene has to contain information for collimating the real world to the virtual objects;

• 3. The real environment has to contain one or more TUIs for the acquisition of the user’s intent of interacting with the scene;

• 4. During each frame acquisition, the user’s intent has to be interpreted; the multibody equations have to be built and solved synchronously in order to compute the correct position of all the virtual bodies in the scene;

• 5. For each frame acquisition, virtual objects have to be rendered on the scene, after the numerical integration, in the correct position and attitude.

3 Implementation of the Augmented Reality Environment

3.1 Hardware Setup

For the specific purpose of this investigation, the implemented AR system (depicted in Fig. 1) includes an input video device Microsoft LifeCam VX6000 USB 2.0 camera, an Head Mounted Display (Emagin Z800) equipped with OLed displays, a Revware Microscribe GX2 mechanical tracker and a personal computer.

Fig. 1

Augmented reality hardware setup

The Revware Microscribe is an instrumented arm (digitizer) which can be grabbed and driven by the user and possesses five degrees of freedom. It is able to acquire the real-time position of its tip stylus. The operating space is a sphere of about 1.2 m of diameter and the precision of tracking is up to 0.2 mm.

3.2 Implementation and Interaction Strategy

In order to implement a AR environment suitable for multibody dynamics interactive simulation, we have chosen the following strategy (see Fig. 2). Before the simulation starts, the geometries and topological properties (joints and connections) have to be defined (as for any multibody simulation). Then, the real scene has to contain information for collimating the real world to the virtual objects. Usually this operation is performed by a patterned planar marker which is recognized by the processing units. For the scope it is necessary to perform the computation of the camera point of view and the corresponding perspective effect.

Fig. 2

Multibody simulation in augmented reality procedural scheme

During each frame acquisition, the position of the user in the scene has to be acquired as well. This task can be performed using the mechanical tracker and computing the position and attitude of its end effector. This information can be interpreted and transferred as a spatial input for the simulation (user’s intent to interact with the scene). Then, the multibody equations have to be solved in order to compute the correct position of all the virtual bodies in the scene, taking into account the user’s intent.

After this computation, all the virtual objects have to be rendered on the scene in the correct position and attitude.

All the supporting software has been implemented using C++ programming language and Microsoft Visual Studio 2003 developing suite. Routines for image processing have been developed using the open source libraries named ARToolkit¹ which has been successfully used in many previous investigations. The libraries comprise a set of numerical procedures which are able to detect and recognize planar patterned markers in a video stream in real time. Using correlation techniques, the routines are also able to compute relative position and attitude between markers and camera with good precision for visual purposes. This computation is necessary for an accurate perspective collimation between virtual entities and real scene. The details about specific implementation and about the contents of the library can be found on the Internet site of the developers.²

The Microscribe GX2 has been integrated using the Microscribe SDK library that allows the real time access to position and attitude of each link of the instrumented arm. This library interprets the output coming from the rotation sensors of the five revolute joints of the mechanical arm and computes the position of the stylus tip by solving an open kinematic chain problem.

For managing complex geometries the OpenVrml library³ has been included. All rendering tasks about virtual objects in the augmented scene have been performed using OpenGL library.

Details about the procedures for deducing and solving the equations of motion of the system under investigation have been provided in the next sections.

All these pieces of software have been integrated into a single simulation environment.

3.3 Collimation Procedure

The first step in the integration of the tracker in the augmented scene is the collimation between the information acquired by the instrumented device and that of the digital camera (see Fig. 3). The video stream acquired by the digital camera is elaborated by an image processing routine. It is able to recognize a patterned marker in the scene and to compute the corresponding transformation matrix between the camera and the real word. This matrix is used to project all the virtual contents in the augmented scene in the correct position and perspective.

Fig. 3

Collimation between optical marker and mechanical tracking system

The information acquired by the digitizer is concerned with the position and attitude of the end effector with respect to the reference frame fixed to the device itself.

In order to ensure the collimation between the data stream coming from the camera and that from the tracker, it is important to compute the relative transformation matrix between the tracker and the world (described by the marker). This calibration has to be performed only at the beginning of the application and it has to be repeated only if the relative position between the world marker and the digitizer changes.

The calibration procedure can be performed by picking with the tracker stylus a set not-aligned points (four no-coplanar points at least) at known positions with respect to the relative frame associated to the marker.

For expressing the coordinate transformation between points, it is useful to deal with homogeneous transformation matrices which include information on both rotation and translation parameters. A generic homogeneous transformation matrix can be expressed in the form:

$$ [ T ] = \left[ \begin{array}{c@{\quad}c} [ \mbox{Orientation} ]_{3\times 3} & [ \mbox{Position} ]_{3\times 1} \\ 0\quad 0\quad 0 & 1 \end{array} \right] $$

(1)

In the same way, a generic point can be expressed with the following coordinate vector:

$$ \{ P \} = \left\{ \begin{array}{c@{\quad}c@{\quad}c@{\quad}c} x & y & z & 1 \\ \end{array} \right\}^{T} $$

(2)

The coordinate transformation of a generic point P from the local coordinate system fixed to the digitizer to the world coordinate system attached to the marker can be written as:

$$ \{ P \}_{\mathit{world}} = \bigl[ T_{\mathit{world}}^{\mathit{digitizer}} \bigr]\{ P \}_{\mathit{digitizer}} $$

(3)

where:

{P} _world :

is the vector containing the coordinate of the point P expressed in the world reference frame;

{P} _digitizer :

is the vector containing the coordinate of the point P expressed in the local (tracker) reference frame.

Considering a collection of points P ₁ P ₂ … P _n , we can build two matrices as:

(4)

(5)

In order to compute the matrix $[ T_{\mathit{word}}^{\mathit{digitizer}} ]$ we have to solve the following system of equations:

$$ [ P ]_{\mathit{world}} = \bigl[ T_{\mathit{world}}^{\mathit{digitizer}} \bigr][ P ]_{\mathit{digitizer}} $$

(6)

for the unknown elements of the matrix $[ T_{\mathit{word}}^{\mathit{digitizer}} ]$.

An homogeneous transformation matrix is defined by 6 independent parameters (three for the description of the rotation and three for the translation). For this reason, the system (6) has more equations than unknowns and the solution can be computed as:

$$ \bigl[ T_{\mathit{world}}^{\mathit{digitizer}} \bigr] = [ P ]_{\mathit{world}}^{ - 1^{ +}} [ P ]_{\mathit{digitizer}} $$

(7)

where the $[ P ]_{\mathit{world}}^{ - 1^{ +}}$ is the pseudo-inverse matrix of the [P] _world matrix.

Due to numerical approximation or errors in acquisition, the orientation block of the computed matrix $[ T_{\mathit{word}}^{\mathit{digitizer}} ]$ can result not exactly orthogonal. Since it represents a rigid spatial rotation, it is important to correct this imprecision. For this purpose, we can operate a QR decomposition of this orientation block:

$$ \bigl[ \mbox{Orientation}_{\mathit{word}}^{\mathit{digitizer}} \bigr]_{3\times 3} = [ R_{1} ]_{3\times 3}[ U_{1} ]_{3\times 3} $$

(8)

where (due to the QR algorithm):

[R ₁]:

is an orthogonal matrix representing the corrected rotation;

[U ₁]:

is a matrix whose upper band contains the errors of approximation and the lower band has only zero elements. In case of a pure rotation (orientation block without errors) [U ₁]=[I].

Finally, in order to compute the transformation matrix between the digitizer and the camera $[ T_{\mathit{camera}}^{\mathit{digitizer}} ]$, useful to collimate the acquired points to the visualized ones, a matrix multiplication has to be performed:

$$ \bigl[ T_{\mathit{camera}}^{\mathit{digitizer}} \bigr] = \bigl[ T_{\mathit{word}}^{\mathit{digitizer}} \bigr]\bigl[ T_{\mathit{camera}}^{\mathit{word}} \bigr] $$

(9)

Figure 4 shows some snapshots acquired during a calibration procedure. The reference points are picked using a reference cube of known dimensions (80 mm × 80 mm × 80 mm).

Fig. 4

Four snapshots taken during calibration procedure

3.4 User’s Interaction

Once the position and the attitude of the tracker are correctly recorded, we have to define the methods to interact with the simulation.

A possible solution (see Fig. 5) is the use of a fictitious spring-damper element connected between the digitizer stylus tip and a point belonging to one of the rigid bodies in the simulation [24]. By this way, if the tip is moved in the scene, it affects the dynamics of that point that moves accordingly.

Fig. 5

The implementation of a fictitious spring for managing the user’s interaction to the simulated bodies

Mathematically, the use of a fictitious spring between the tracker stylus tip and a spline control point adds a term to the external force vector {F} _e in equations of motion:

$$ \{ F \}_{e} = \{ F \}_{e} - k_{f\_\mathit{spring}}\bigl\{ d_{\mathit{virtual}\_\mathit{body}\_\mathit{point}}^{\mathit{tracker}\_\mathit{tip}} \bigr\} - c_{f\_\mathit{spring}}\bigl\{ \dot{d}_{\mathit{virtual}\_\mathit{body}\_\mathit{point}}^{\mathit{tracker}\_\mathit{tip}} \bigr\} $$

(10)

where:

$\{ d_{\mathit{virtual}\_\mathit{body}\_\mathit{point}}^{\mathit{tracker}\_\mathit{tip}}\}$ :

is the distance between the connected control point and the digitizer tip;

$\{ \dot{d}_{\mathit{virtual}\_\mathit{body}\_\mathit{point}}^{\mathit{tracker}\_\mathit{tip}} \}$ :

is the derivative of $\{ d_{\mathit{virtual}\_\mathit{body}\_\mathit{point}}^{\mathit{tracker}\_\mathit{tip}} \}$with respect to time;

k _{f_spring} :

is the stiffness coefficient of the fictitious spring;

c _{f_spring} :

is the damping coefficient of the fictitious spring.

In order to prevent the dynamics of the system to be affected by the presence of an external (and not physical) elastic component the value of the stiffness parameter has to be chosen very stiff. Moreover, the use of a damping coefficient improves the stability of the system and prevents jittering in simulation.

4 Implementing the Sequential Impulse Solver

Given a collection of n _body rigid bodies, constrained by n _joint kinematic joints and subjected to a set of external forces, the equation of motion can be deduced following different approaches.

One of the most used multibody dynamics formulation is based on the Lagrange equations, obtaining a differential-algebraic system:

$$ \left\{ \begin{array}{l} [ M ]\{ \ddot{q} \} - [ \psi_{q} ]^{T}\{ \lambda\} = \{ F_{e} \} \\ \{ \psi\} = \{ 0 \} \end{array} \right. $$

(11)

where:

[M]:

is the inertia matrix of the collection of rigid bodies;

{ψ}:

is the vector containing all the constraint equations;

$\{ q \}, \{ \dot{q} \}\mbox{ and }\{ \ddot{q} \}$ :

are the vectors of absolute generalized coordinates, velocities and accelerations, respectively;

[ψ _q ]:

is the Jacobian matrix of the constraint equations (differentiated with respect to the generalized coordinates);

{λ}:

is the vector of Lagrange multipliers associated to the constraint equations;

{F} _e :

is the vector containing all the elastic and external forces (including the fictitious spring contribution).

In order to reduce the complexity of the solution, the constraint equations are often differentiated two times with respect to time and Eq. (11) is rearranged as:

$$ \left[ \begin{array}{c@{\quad}c} [ M ] & [ \psi_{q} ]^{T} \\ {[ \psi_{q} ]} & [0] \end{array} \right]\left\{ \begin{array}{c} \{ \ddot{q} \} \\ \{ \lambda\} \end{array} \right\} = \left\{ \begin{array}{c} \{ F_{e} \} \\ \{ \gamma\} \end{array} \right\} $$

(12)

where

$$ \{ \gamma\} = - \bigl( [ \psi_{q} ]\{ \dot{q} \} \bigr)_{q}\{ \dot{q} \} - 2[ \psi_{qt} ]\{ \dot{q} \} - \{ \psi_{tt} \} $$

(13)

and the subscripts “q” and “t” denote the differentiation with respect to the generalized coordinates and time, respectively.

Both Eq. (11) and Eq. (12) allow to solve for the unknown generalized acceleration, velocities and positions taking into account all the constraint equations simultaneously. Of course, this approach can achieve accurate results with suitable DAE solver. On the other hand, a dynamic system with a lot of constraints increases the complexity of the problem and the computational effort to solve it. For this reason, the system in (12) can be rearranged for being suitable for the sequential impulse solution strategy.

There are two main steps in the impulse-based methodology. Firstly, the equations of motion are tentatively solved considering elastic and external forces but neglecting all the kinematic constraints. This produces a solution that is only approximated because the constraint equations are not satisfied. In a second step, a sequence of impulses are applied to each body in the system in order to correct its velocity according to the limitation imposed by the constraint. This second step is iterative. It means that a series of impulse is applied to the bodies until the constraint equations are fulfilled within a specific tolerance. It is important to underline that each impulse is applied independent from the others. By this way the constraint equations are not solved globally, but iteratively.

4.1 Solving the Equations of Motion

Following the approach introduced in the previous section, the sequential impulse formulation can be split into two main steps. The first one is about the solution of the equations of motion in (11) neglecting the constraint equations and constraint forces:

$$ [ M ]\{ \ddot{q} \}_{\mathit{approx}} = \{ F_{e} \} $$

(14)

By this way, Eq. (14) can be solved for $\{ \ddot{q} \}_{\mathit{approx}}$ that represent the vector of approximated generalized accelerations.

The values of the corresponding approximated generalized velocities and positions can be computed by linear approximation:

(15)

(16)

where h is the integration time step.

In order to correct the $\{ \dot{q} \}_{\mathit{approx}}$ and fulfill the constraint equations, a series of impulses {P} _constraint has to be applied to the bodies. Each impulse is computed imposing the fulfillment of the constraint equations written in terms of generalized coordinates. As well-known from the Newton’s law, the application of the impulses causes a variation of momentum:

$$ [ M ]\bigl( \{ \dot{q} \}_{\mathit{corrected}} - \{ \dot{q} \}_{\mathit{approx}} \bigr) = \{ P \}_{\mathit{constraint}} $$

(17)

where $\{ \dot{q} \}_{\mathit{corrected}}$ is the vector of generalized velocities after the application of impulses {P} _constraint .

The corrected velocities can be computed from Eq. (17) as:

$$ \{ \dot{q} \}_{\mathit{corrected}} = \{ \dot{q} \}_{\mathit{approx}} + [ M ]^{ - 1}\{ P \}_{\mathit{constraint}} $$

(18)

Considering that the impulses are related to the constraint equations, they can be computed as

$$ \{ P \}_{\mathit{constraint}} = [ \psi_{q} ]^{T}\{ \delta\} $$

(19)

where {δ} is the vector of Lagrange multipliers associated to the impulses.

Since the effect of the impulses is to correct the generalized velocities and fulfill the kinematic constraints, the $\{ \dot{q} \}_{\mathit{corrected}}$ has to satisfy the constraint equations written in terms of velocities:

(20)

(21)

Inserting Eq. (19) into Eq. (18) and substituting $\{ \dot{q} \}_{\mathit{corrected}}$ into Eq. (21) we can obtain:

$$ [ \psi_{q} ]\bigl( \{ \dot{q}_{\mathit{approx}} \} + [ M ]^{ - 1}[ \psi_{q} ]^{T}\{ \delta\} \bigr) + \{ \psi_{t} \} = \{ 0 \} $$

(22)

Equation (22) can be solved for {δ}obtaining:

$$ \{ \delta\} = \bigl( [ \psi_{q} ] [ M ]^{ - 1}[ \psi_{q} ]^{T} \bigr)^{ - 1}\bigl( [ \psi_{q} ]\{ \dot{q} \}_{\mathit{approx}} + \{ \psi_{t} \} \bigr) $$

(23)

Then, the impulses can be computed using Eq. (19) and the corrected values of generalized velocities using Eq. (18).

Since the impulses are computed sequentially, the global fulfillment of the constraint equations cannot be directly achieved. Some iterations are required. The computation of {δ}, {P} _constraint and $\{ \dot{q} \}_{\mathit{corrected}}$ can be repeated till a tolerance on the fulfillment of Eq. (21) is reached or for a maximum number of time. Experience [17] shows that four or five iterations are sufficient to achieve an adequate tolerance. Since the constraints are imposed at velocity level, a stabilized formulation is required to control the constraints fulfillment at the position level. Details are provided in Sect. 4.3.

4.2 Computation of Reaction Forces

When simulating multibody dynamics, one of the most interesting results for engineers is the knowledge of the reaction forces, i.e. the forces exerted by the joints.

Using the sequential impulse formulation, the reaction forces cannot be computed directly but a preliminary computation is required. The problem is that impulses are applied sequentially, it means that each joint exerts more than one impulse during each time step and the various impulses have to be recollected.

In order to deduce a methodology for evaluating the reaction forces of the joints we have to introduce the concept of the accumulated impulse. It can be defined as the resultant impulse of each joint produced each time step and it can be computed as the sum of all the impulses exerted by the joint over the iteration.

In the solution of the equations of motion, the joint impulses are evaluated using Eq. (23) and Eq. (19). This computation is performed iteratively in order to reach a set of velocities congruent to kinematic joints. It means that at each iteration a new impulse vector {P} _constraint is computed.

The accumulated Lagrange multipliers {Δ} of the impulses for each time step can be evaluated as:

$$ \{ \varDelta\} = \sum_{\mathit{iterations}} \{ \delta\} $$

(24)

The accumulated impulse {P _tot } _constraint can be computed using Eq. (19) obtaining:

$$ \{ P_{\mathit{tot}} \}_{\mathit{constraint}} = [ \psi_{q} ]^{T}\{ \varDelta\} $$

(25)

The reaction forces {F} _constraint exerted by the joints can be computed using the general relation between forces and impulses:

$$ \{ F \}_{\mathit{constraint}} = \frac{\{ P_{\mathit{tot}} \}_{\mathit{constraint}}}{h} $$

(26)

4.3 Stability Issues

The use of the sequential impulse strategy is subjected to the use of constraint equations expressed in terms of generalized velocity. It means that the exact information about the kinematic joints may be lost during the integration process. In this case a position drift can be observed and stability problems may occur.

In order to enforce the constraints on position a stabilized formulation can be adopted. In this case, the constraint equations in Eq. (21) can be modified as:

$$ [ \psi_{q} ]\{ \dot{q}_{\mathit{corrected}} \} + \{ \psi_{t} \} - \frac{\beta}{h} \{ \psi\} = \{ 0 \} $$

(27)

where β is a scalar chosen in the range 0–1.

5 Examples of Simulation

In order to test the proposed methodology and both hardware and software integration, in this section two examples of implementation are presented and discussed.

5.1 Ten Pendula Dynamic System

The first simulated scenario is about a collection of 10 rigid pendula that move under the effect of gravity. All the pendula have the same geometry and inertial properties which have been summarized in Table 1.

Table 1 Geometrical and inertial parameters of the first example

The first pendulum is pivoted to a fixed frame by means of a point-to-point 3 d.o.f. joint. The other pendula are sequentially connected by mean of revolute joints (hinges). The user can interact with the scene by connecting a fictitious spring between the tracker stylus and the free end of the last pendulum. In particular, the user can decide when the connection between the tracker and the last pendulum has to be activated (simulating the clipping) or deactivated (simulating the release). This scenario is also important to test the methodology with event based changes in the equations of motion.

In order to achieve stable and correct results, the solution strategy has to be able to manage rapid changes in force definition. The simulation has been performed with a fixed time step of 0.01 s.

Per each video frame, 4 integration steps are computed and the augmented scene is updated accordingly.

Figure 6 reports a series of four snapshots taken during the run of the simulation. The rigid body collection is real-time rendered along with the simulation. In the first part of the simulation (snapshot A of Fig. 6) the pendula are free to move and they are in an equilibrium position (aligned along the vertical direction).

Fig. 6

Snapshots of the simulation of the first example

Then, the user locates the tracker near the last pendulum tip and activates the fictitious spring connection (snapshot B). From this moment, the tip of the last pendulum moves subjected to this connection.

When the user moves the tracker, the tip of the pendulum follows it. It is important to notice that the motion of all the rigid body collection is continuously simulated according to the external action of the gravity and the driving force due to the user’s presence.

When the user decides to release the fictitious spring connection (snapshot C), the collection of pendula moves subjected to gravity force only and it oscillates around the equilibrium position (snapshot D).

The accuracy and the stability of the simulation have been checked computing the error in the fulfillment of the constraint equations and the overall energy of the system (potential, kinetic and elastic). The norm of the constraint equations expressed using position variable is always lower than 10⁻⁶ m except for ten time steps when it reaches 1.6⋅10⁻⁵ m. The variation of the overall energy of the systems (energy loss) is below 1 %. In the computation of the overall energy, the contribution of the external action of the user has been taken into account by evaluating the corresponding power as the dot product between the reaction force of the fictitious spring and the velocity of the digitizer stylus tip as:

$$ \mathit{Power}_{\mathit{user}\_\mathit{action}} = \{ F_{\mathit{fictitious}\_\mathit{spring}} \} \cdot\{ v_{\mathit{stylus}\_\mathit{tip}} \} $$

(28)

where

Power _{user_action} :

is the power associated to the reaction force of the user’s action;

{F _{fictitious_spring} }:

is the reaction force of the fictitious spring computed as in Eq. (10);

{v _{stylus_tip} }:

is the absolute velocity of the stylus tip.

5.2 Cross-Lift Dynamic Simulation

The second simulated scenario is about the dynamic motion of a cross-lift device which is comprised of 11 rigid bodies connected by 16 hinges and 4 slider joints (see Fig. 7). The links are connected by means of the following constraints:

• Two revolute joints between the frame and the first link on both side A and B of the mechanism;

• Two slider joints between the frame and the second link on both side A and B of the mechanism;

• Four revolute joints between adjacent links on each side of the mechanism (8 in total);

• A revolute joint between the third link and the upper plate on both side A and B of the mechanism;

• A slider joint between the fourth link and the upper plate on both side A and B of the mechanism;

• Four revolute joints connecting the two horizontal rods between the two side of the mechanism.

Fig. 7

Cross-lift device of the second example

Two linear spring-damper elements act horizontally between the frame and the slider joint location of the second link on both side A and B of the mechanism. The gravity field acts downward along the vertical direction.

The example has been chosen in order to test the capabilities of the solver to deal with many overabundant constraints. Geometrical, inertial and elastic properties of the simulation have been summarized in Table 2.

Table 2 Geometrical, inertial and elastic parameters of the second example

The user can interact with the scene by imposing a fictitious spring between the stylus tip and the middle point of the upper plate. In particular, the user can decide when the connection between the tracker and the upper plate has to be activated (simulating the clipping) or deactivated (simulating the release).

In the same way of the first example, the simulation has been performed with a fixed time step of 0.01 s. Per each video frame, 4 integration steps are computed and the augmented scene is updated accordingly.

Figure 8 reports a series of four snapshots taken during the run of the simulation. The cross-lift mechanism is real-time rendered along with the simulation.

Fig. 8

Snapshots of the simulation of the second example

In the first part of the simulation (snapshot A of Fig. 8) the mechanism is free to move and it is in an equilibrium position. Then, the user locates the tracker in the middle of the upper plate and activates the fictitious spring connection (snapshot B). From this moment, the upper plate vertical coordinates is controlled by the tracker and the mechanism moves subjected to this connection.

When the user moves the tracker, the upper plate follows it in the vertical direction (snapshot C), preserving the right connection with the other rigid bodies. It is important to notice that the motion of all the rigid body collection is continuously simulated according to the external action of the gravity, the springs and the driving force due to the user’s presence.

When the user decides to release the fictitious spring connection, the lifting device moves freely subjected to gravity force and internal springs and it oscillates around the equilibrium position (snapshot D).

Also for this example, the accuracy and the stability of the simulation have been checked. Due to the presence of a lot of overabundant constraints, the norm of the constraint equations is higher than in the previous example. During the free motion (when the simulation run without the interaction of the user) it is lower than 10⁻⁶ m. When the user interacts by pushing and pulling the upper plate, the norm increases and reaches 6.1⋅10⁻⁵ m. The variation of the overall energy of the systems is always below 2 %. As in the previous example, in the computation of the overall energy, the contribution of the external action of the user has been taken into account by evaluating the corresponding power as the dot product between the reaction force of the fictitious spring and the velocity of the digitizer stylus tip using Eq. (28).

6 Conclusions

In this chapter, an enhanced methodology for interactive, accurate, fast and robust multibody simulations of mechanical systems using Augmented Reality has been presented and discussed. This methodology is based on the integration of a mechanical tracker and a dedicated impulse based solver.

In this context, the simulation of movement of mechanical systems in an Augmented Reality environment can be useful for projecting virtual animated contents into a real world. By this way, it is possible to build comprehensive and appealing representations of interactive simulations including pictorial view and accurate numerical results.

In particular, two important enhancements have been presented with respect to a previous implementations. First of all, it has been possible to improve the precision of the interaction between the user and the scene by means of a precise mechanical tracking instrumentation. This constitutes an important improvement if compared with the use of simple optical markers for tracking the user in the scene. In the latter case, the precision in tracking was affected by the resolution of the camera, while with a mechanical device, it is possible to separate the processing of the data coming from the position tracking, from those coming from the image collimation processing. By this way, the simulation input is independent from the visualization input and output.

The second important enhancement is the use of a dedicated solver based on the sequential impulse strategy in order to perform a fast and robust simulation.

According to this approach, the solution is based on the less computational demanding solution strategy. Following the implemented algorithm, the equations of motion are firstly tentatively solved considering elastic and external forces but neglecting all the kinematic constraints. This produces a solution that is only approximated because the constraint equations are not satisfied. In a second step, a sequence of impulses are applied to each body in the collection in order to correct its velocity according to the limitation imposed by the constraint. This second step is iterative and involves the application of a series of impulses to the bodies until the constraint equations are fulfilled within a specific tolerance.

The final result of this work is a tool able to manage real time dynamic simulation and to update the augmented scene accordingly. The robustness and the reliability of the system have been checked over two test cases: a ten pendula dynamic system and the dynamics of a cross-lift mechanism.

According to the proposed methodology, the user can directly control the simulation by a smooth visualization on the head mounted display.

The integration among Augmented Reality, dedicated solver and precise input tracker can be considered an advantage for the future development of a new class of multibody simulation software. Moreover, this integrated simulation environment can be useful for both didactical purposes and engineering assessments of mechanical systems.