Keywords

1 Introduction

The technological advances in both the mobile and the transportation industries are remarkable. This has made the development of ADAS an interesting topic [1]. However, even though most high-end cars nowadays ship with built-in embedded systems, most of the older cars do not have such devices. This brings about an interesting research opportunity, which is to develop and test ADAS that run on low-cost devices, such as an Android tablet or smartphone.

The main goal of this paper is to describe the methodology of our Multi Agent System (MAS) based driving simulator, integrating SUMO microscopic simulator with driving simulators. We also intend to describe our implementation of a test-bed to easily develop ADAS using the system, simulating their use in a low-cost and controlled environment.

There are several benefits to testing ADAS in a simulated environment rather than in real-world scenarios. As the tests are not conducted in a real physical location, they are not subjected to travel times, traffic or other adverse conditions which could render them mute. This, as well as being able to deploy the simulator in low-cost computers, and therefore reaching more test subjects, leads to time compression of the tests. Noticeably, cost reduction is another significant benefit as the electrical cost of running a simulator is dismissive when compared to fuel costs of real-world testing. Besides preventing the safety risks inherent to driving, simulation allows us to control the test environment and manipulate it according to the specificities of the ADAS being tested.

The objective of our work was to develop the MAS and include a test-bed that was easy to implement and replicate in low-cost environments. We aimed to combine SUMO microscopic simulator with IC-DEEP, which is a driving simulator developed at LIACC [2], with the Geostream framework developed at SI and CG, as well as to enhance them with logs of simulated GPS positions in a mobile device. To achieve so, a mobile application/service was developed in order to receive this communication from the simulator and override the default GPS sensor of the device. We wanted to make it easy to extend the communication between the simulator and a mobile device, providing the latter with more information such as the current speed limit, semaphoric information, or other data from the network.

This work aims to contribute with a novel multi-faceted methodology to simulate and research multiple human factors in Intelligent Transportation Systems (ITS) and, particularly, with a novel approach to test ADAS that will enable developers to validate and test their applications more easily and efficiently while reducing costs.

In the following sections we describe the development and results of our implementation. In Sect. 3 we introduce some related state-of-the-art works on the subject of ITS, focusing on simulation, on the integration of different scope simulators and also on the topic of serious games. We then describe our approach, architecture and development details. Finally we present our preliminary verification as well as their analysis in Sect. 5. We finish this paper with a set of conclusions and interesting future work.

2 Background and Related Work

The Artificial Transportation Systems (ATS) [3, 4] concept has been one of the main research topics in the IEEE ITS Society [5]. A typical approach to ATS modeling and development is the MAS metaphor. Another potentially concomitant approach is the HLA concept, which is based on the idea of distributed simulation so as to meet the requirements of all usages and users rather than of a single simulation model and analysis perspective [6].

In [7], authors propose to integrate a driving simulator and a traffic microsimulator, in an attempt to tackle the mutual-dependence between the driver’s behavior and traffic conditions.

Combining SUMO microscopic traffic simulation [8], using MAS capabilities, with other simulators has also been researched [9]. Authors in [6] have studied an HLA-based approach to simulate electric vehicles in Simulink and SUMO. Driver-centric simulation has been researched by authors in [10], where they have developed a simulation tool that provides feedback to the network based on the driver’s behaviour.

Driving simulators are no doubt an important tool when researching ATS, specially so when studying the influence of human factors in driving faults [2]. These faults often occur in direct consequence of performing secondary tasks while driving [11]. In [12] authors introduce a game-engine-based modeling and computing platform for ATS. They describe the artificial population both in their macroscopic and microscopic aspects.

Regarding driving simulators with ADAS testing capabilities, authors in [13] propose a reconfigurable driving simulator with several components to accommodate different ADAS testing or training. A framework for ADAS assessment and benchmarking has been developed in [14], with configurable scenarios and 3D scenes and multiple sensors input.

Authors in [15] developed a Full Speed Range Adaptive Cruise Control with their platform for ADAS prototyping and evaluation, SiVIC. The platform is capable of reproducing vehicle and sensor behaviors in a realistic fashion, according to the configured environment in the simulator. The developed platform also simulates noised and imperfect data.

A system comprising a large scale driving simulator, built in a 360° full dome with 3D scenes from real city area has been developed in [16]. The system contains a multitude of features such as real-time hardware-in-the-loop, wireless communication devices and bio signal analysis and is used to develop and test ADAS as well as Advanced Safety Vehicle, ITS infrastructure and others.

3 Methodological Approach

The proposed system architecture is as described in Fig. 1. The main module of the system is the SUMO simulator, which is responsible for the network’s multi-agent microscopic simulation, and has multiple driving agents. This module provides an overview of the whole MAS and can be manipulated directly.

Fig. 1
figure 1

Overview of the system’s architecture

Full size image

The SUMO module also acts as a “central server”, providing all the essential information for both IC-DEEP and the High Fidelity Simulator. This information consists of the network infrastructure and the agents in the system, whereas terrain morphology and road or building geometry are provided by the Geostream framework.

Both of the driving simulators have a local representation of the whole MAS and are capable of controlling any driving agent. The simulators are also able to connect to an Android device and pass along all the information deemed necessary, such as the GPS coordinates of the current driving agent being controlled. The Android device is running a service that receives the incoming connections from the simulator and also the ADAS being tested. The dotted area in Fig. 1 corresponds to the developed components as of the writing of this paper.

3.1 Simulators and Framework

The proposed system architecture has two simulators; however, as of the writing of this paper, only the IC-DEEP simulator has been enhanced and integrated into the framework. The latter is implemented in Unity3D and has the Geostream framework embedded directly. The Geostream framework connects with Open Street Maps, Google Geolocation API, Google Altitude API and other data providers in order to fetch the required geographical information of a given location and remodel it in a fashion which can be interpreted by all the simulators in a coherent and consistent way. This is especially important to the SUMO microscopic simulator as the raw network data imported from Open Street Maps, typically, generates unrealistic ways and intersections. The information generated by Geostream framework is then parsed into both simulators to generate a 3D scene that is representative of the chosen test location. The Geostream framework is explained in further detail in the following section.

3.2 The Geostream Framework

The Geostream Framework was originally designed to aggregate location-based data from multiple sources and recreate urban and rural environments for use in location-based games. One of the issues in location-based games is the need to accurately determine the user’s current context. As such, the Geostream provides a foundation on top of which location-based games can be developed. However, since the information is of use to other areas (simulation, procedural modeling), the framework was further altered so that it can be used in non-game development.

The framework accesses the following web services, using them as providers for the respective context-sensitive data (Table 1):

Table 1 Geostream currently used services and their data
Full size table

The data retrieved by these sources is then combined as a means to recreate the current location context of the player (or in this case, the user). Certain sources may be optionally not used if their data adds little to the problem at hand. After loading data from the above sources, the scene graph looks similar to that of Fig. 2.

As Fig. 2 depicts, the typical scene graph of a Geostream based application consists of a World Map gameobject, and other several Geo-gameobjects (objects not based on the external sources of data, but that are placed in the game world, such as a player, enemies, cars, etc.). The World Map game object consists of several Map chunks. Each Map chunk holds the information of a specific part of a location, such as the geometry of the terrain of that part (its elevation) and its aspect (the corresponding part of an aerial photo).

Fig. 2
figure 2

Example of a Geostream scene graph

Full size image
Fig. 3
figure 3

Screenshot of the reconstruction of the Mount St. Helens crater

Full size image
Fig. 4
figure 4

Depiction of the loading behavior of a World Chunk

Full size image

As Fig. 3 shows, a World Map is comprised of several World Chunks. In the above image, the grid mesh that is drawn with a blue color, represents the limits of a given World Chunk. In this case, the World Map consists of 16 World Chunks, each responsible for computing the information that belongs to itself (not only its geometry or texture, but those of the structures that belong to it: POIs, roads, buildings, natural structures, etc.). The behaviour of each World Chunk is summarized in the fluxogram of Fig. 4.

When a World Map is created, using a certain coordinate or address as a center reference, it will procede to create NxN World Chunks, each with a predefined width and height of fractions of degrees. Then, each World Chunk will proceed to look at the information it needs to reconstruct itself. Additionally, a World Map can have a Geo-Gameobject to “follow”. This means that as the referenced Geo-Gameobject moves around, new World Chunks may be dynamically loaded, while others are unloaded to preserve memory.

New sources of information can be added to a World Chunk for it to make use of. Additionally, it is possible to change, in design time, how certain types of meta data are processed. For instance, in Unity, in the editor view, one can visualize what current tags of Open Street Maps or Google Places are being processed and how (as it can be seen in Fig. 5).

As seen in Fig. 5, a dictionary is created in design time, pairing tags with the script classes responsible for treating information related to a specific tag. When the world object is created, it will instantiate, through C# Reflection feature, the necessary scripts (all derived from the Structure Script super-class). So, whenever a structure’s information with a certain tag is compiled after consulting the multiple sources, it is passed on to the respective script for treatment. In Fig. 5, we can see that “residential”, “motorway”, “bridge” and “highway” are all treated by the Road Script, a script responsible for generating roads with different features. This allows for further expanding the possibilities or procedurally generating other real-world objects based on the information compiled from external sources. For instance, one could add the tag “traffic light”, present in the information gathered via Open Street Maps, coupled with a Traffic Light Script that would be capable of treating that specific data. It could, for example, instantiate a semaphore prefab in that structure’s position. Same thing could be done with trees, lamps, and other simple and repeatable structures.

Fig. 5
figure 5

Subsection of the World Map behaviour script

Full size image

The biggest drawback of Geostream Framework, is that it is limited by the availability of the context-related sources, and the quality (or existence) of information. Performance wise, as most operations are threaded, loading of World Chunks is made in a smooth manner, albeit for areas with many structure-related information available, it can take some time to generate all the needed geometry (even so, the FPS of the simulation rarely becomes affected). In locations with abundant information and with the needed scripts performing procedural generation of structures, the results are both visually appealing and accurate (note that generated structures fit with the structures captured by the aerial photos).

Notice how in Fig. 6, some buildings are red, while some are not. Red buildings are those that have all their geometry’s height defined at the source, while textured buildings have their height generated based on the height of neighboring buildings and those whose heights are known from the source. Buildings with extra detail can also be computed by further developing the Building Script, associated with the creation of those structures.

Fig. 6
figure 6

Geostream procedural generation of Monaco (above) and New York (below)

Full size image

3.3 Mobile Device

The Android service sets the current GPS location using Mock Location API to override the default location provider. All applications running on the mobile device that use or perceive the current location will also be affected by the running service.

The new GPS coordinates are sent from the simulator every second; however, this value is a parameter of the simulator and so can be adjusted to the specific needs of each scenario. The developed service can be run as a standalone application, and thus testing the ADAS mobile applications independently, as shown in the left side of Figs. 1 and 2. There is also the option to use the service as a library in any Android application, as long as it matches API level 19, as shown in the right side of Fig. 7.

Fig. 7
figure 7

Standalone mobile application (left) and mobile application with included library (right)

Full size image

3.4 Interaction of Driving Simulators and Android

A typical interaction between the simulators and the mobile devices is shown in Fig. 8. The modules are connected via TCP-IP sockets due to implementation simplicity. The communication messages are formatted in JSON and therefore the message contents can be easily changed to add different kind of data.

Fig. 8
figure 8

Typical interaction between IC-DEEP simulator and an ADAS

Full size image

The basic message template contains two compulsory fields, which are latitude and longitude. Other optional fields are the current speed, the GPS accuracy, the message timestamp or even the speed limit from the current location. A specific instantiation of this interaction is discussed in the next section.

The coupling of SUMO microscopic simulator with the driving simulators, namely with IC-Deep, uses the same methodology as implemented and described elsewhere [17]. However, this raises some issues regarding the communication channel, as the SUMO TraCI protocol uses sockets and currently does not support more than one active socket. This is obviously a bottleneck when controlling multiple driving agents and a possible SUMO extension to support parallelism in terms of communication is in study.

4 Preliminary Verification

The preliminary verification to assess the proof of concept and also the efficiency of the developed architecture focused on the modules in the dotted area of Fig. 1, the remainder of the system will be developed later on, as mentioned in the next section. We have divided the verification into two independent tests. Both of the tests were performed in the same geographical location, which was downtown Porto, on Avenida dos Aliados, as seen in Fig. 9.

Fig. 9
figure 9

Generated 3D Scene ortographic view

Full size image

In the first experiment we test the simulator accuracy to represent real-world scenarios using the Geostream framework. The other test aims to emulate the GPS signal on the mobile device.

4.1 Simulator Accuracy

To test the simulator accuracy we have collected multiple GPS trace logs while driving a real car in the selected geographical location. We have then overlayered a visual representation of the obtained traces on the simulator and on Google Earth, both results can be seen in Fig. 10. The results show that the generated 3D scene is highly representative of the real-world location. In one of the trace logs we have noticed an error and highlighted it in Fig. 10 (see labels 1 and 2), this error happens due to the data being collected as raw, untreated GPS, where the road matching algorithm [18] has not been applied. This is also an interesting result, as the error can been seen in both the simulator and Google Earth alike.

Fig. 10
figure 10

GPS traces on the simulator (left) and Google Earth (right)

Full size image

Apart from testing the fidelity of the simulation with GPS trace logs, it is also noticeable that the orthographic view of the generated 3D scene very much resembles the satellite image of the same location, as it can be seen in Fig. 11.

Fig. 11
figure 11

Satellite image of Av. dos Aliados

Full size image

4.2 Mobile Device ADAS

To test the communication and emulation in the mobile device the setup consisted of a basic usage scenario, using a simple ADAS that shows the user his current and average speed, the total kilometers traveled, and, most importantly, warns him when he exceeds the speed limit of the current location as shown in Fig. 12.

Fig. 12
figure 12

Developed ADAS

Full size image

The interaction between the simulator and the mobile application followed that of Fig. 8. To run the simulation the mobile application must be started and the device’s IP address, which is shown in the application initial screen, must be entered in the simulator configuration screen. After this the simulation can start and the simulator internally updates the geographical coordinates as the driver traverses the network. These coordinates are passed on to the mobile device as described above, every second and via a JSON formatted message over a TCP-IP socket.

In this particular simulation the information sent to the mobile device consists of the current GPS coordinates and the speed limit of the current location. The heading of the vehicle is calculated internally by the mobile application using a simple algorithm that computes the bearing with the last two known locations and thus, even though this can be done easily, there is no need to pass on an orientation variable from the simulator.

The main goal of this experimentation was to understand whether or not the mobile device simulated GPS position and calculated speed matched those of the simulator. We have used the driving simulator and Google Maps application to compare the marked position while driving. To compare the driving speed we have used the developed ADAS.

Even though the results from the simulator and the mobile device were not recorded, any inaccuracies were not noticed when testing. The only possible minor differences would be due to the fact that the Google Location API on the Android device automatically adjusts the current position to the nearest road, which is, as mentioned above, a technique called road matching.

5 Conclusion

In this paper we presented a multi-faceted MAS-based driving simulator methodology. The presented framework can be used to simulate and test multiple aspects in human factors in ITS, generally. Among others we identify some that we consider more expressive of the system’s spread, such as supporting a Serious Game [19] to test driving behaviors and ergonomics, simulating driver’s idiosyncrasies effects on the ATS with peer-designed agents, and also prototyping and validating Advanced Driver Assistance Systems.

The preliminary verification has illustrated the system efficiency and usability, as well as its ability to accurately represent real-world scenarios without the need of extensive 3D modeling or expensive hardware setups. This ability allows researchers to conduct studies regarding singularities of the different geographical locations.

As a great advantage over other systems we point out the fact that our system is always up to date in terms of real-world mapping, and also that there is no need to waste any time creating a scene when the sole purpose is to test an ADAS or do any other kind of simulation.

In addition to implementing the remaining components of the proposed methodology, there is an ambitious workload of further developments. We would like to point out some that we consider proprietary and more challenging. We believe it would be interesting to support batch simulations, in order to collect significant data and extract more elaborate conclusions. There are also improvements specific to driving simulators that we envisage, such as more detailed scenarios and improved physics.

It would also be interesting to develop cache servers that could store the responses from external services, and thus improve loading times. Another interesting enhancement would be to allow multiple agents to connect to multiple ADAS, simulating distributed ADAS applications while extending SUMO capabilities. There are also refinements to be done in the Geostream framework, especially regarding road generation and also importing models and textures for different buildings.