PEGASUS—First Steps for the Safe Introduction of Automated Driving

Abstract

PEGASUS (Project for the Establishment of Generally Accepted Quality Criteria, Tools and Methods as well as Scenarios and Situations for the Release of Highly Automated Driving Functions) is a joint project funded by the German Federal Ministry for Economic Affairs and Energy (BMWi) which seeks to close the gaps in the testing and release of automated vehicles (see Fig. 1) and supports the rapid transfer of existing functions and prototypes into series production (Fig. 2). PEGASUS intends to answer these central questions. What is the minimum performance level for an automated vehicle? How do human beings perform (as a reference value)? What can and must automation deliver (and what not)? How can it be demonstrated that the automated vehicle performs reliably?

1 Overview

17 partners from science and industry are working together on PEGASUS (see Fig. 3) to develop a complete toolchain to include criteria and measures for the evaluation of functions and for quality levels, with test catalogues, central methods, and processes for establishing the safety and make it possible to release highly automated driving functions (Level 3 according [1]). The toolchain will be developed as a prototype within the project and demonstrated in practice. The result will be a new state-of-the-art in technology and engineering across all manufacturers to ensure the safety of highly automated driving functions, preparing the way for subsequent release and approval. Implementation and validation will take place using the Highway Chauffeur as application example in order to develop the real-world functionality of the toolchain.

Fig. 3

PEGASUS key-facts

The project operates as four subprojects looking more closely at further, more detailed issues (see Fig. 4).

Fig. 4

Central issues of the PEGASUS project

Subproject 1 “Scenario Analysis and Quality Measures” defines the Highway Chauffeur example application, human and machine performance, and the criteria and measures used for evaluation. Subproject 2 “Implementation Processes” analyses safety processes currently established within the automobile industry, transfers them into new or extended process methodologies for highly automated driving functions and works up the process specification for actual testing. Subproject 3 “Testing” develops methods and tools for carrying out tests in the laboratory, at the test site and in real traffic situations. The objective here is to cover as many tests as possible in simulation. Ensuring transferability and embedding of results into industrial processes is the task of Subproject 4 “Result Reflection and Embedding” by means of a proof of concept, amongst other tools.

2 Scenarios and Quality Measures for Automated Driving

In order to ensure efficient testing of (highly) automated systems, a large volume of diverse information is both useful and necessary. This information includes scenarios, suitable criteria and measures which permit an evaluation of the performance and quality of the system. Alongside the technical system, it is essential that we quantify human performance, particularly for use as a benchmark, in order to determine the requirements of a (highly) automated system and to ensure that quality of traffic at least remains the same.

The Highway Chauffeur example application is taken as a realistic basis for all investigations and developments for all subprojects. Based on Highway Chauffeur, tools will be created for determining critical traffic situations, human and machine capacities. The dataset is composed of existing sources of information such as accident databases (e.g. GIDAS [2], ZIDATU [3]), Naturalistic Driving Studies (NDS), Field Operational Tests (FOT) or driving simulator studies.

These data will be efficiently and automatically evaluated for critical situations using metrics developed within the project. Relevant conclusions regarding human performance in this situation can be derived in turn from the results and compared with machine performance. The result is a determination of the effectiveness of the automated driving function which can be equated with an accident avoidance potential. From the subsequent evaluation of the probability of occurrence and the ability of the automated system to control the critical situations identified, the required safety level can be determined and an accepted quality measure and system requirements can be specified (see Fig. 5).

Fig. 5

Scenario analysis and quality measures

In order to make efficient use of the results acquired, they will be entered into the test specification database [4]. The other subprojects also have access to this database, make use of the data, further add to it and similarly save data here, e.g. the results of tests. All of the project results directly required for testing are thus available from a central source.

The fundamental metric perspective [5] in Fig. 6 details this approach, beginning with the start of the process at the bottom left, with the information sources from which the test scenarios will be developed. These will be structured and organised in test databases. The test performance and evaluation of the test results will be used to carry out a risk analysis. If the risk is below the comparison measure then the item can be released for series introduction.

Fig. 6

PEGASUS metrics perspective

3 Processes Required for Establishing Safety

In order to be able to evaluate requirements and quality measures as well as test results in a reproducible and comparable manner which is as consistent as possible, they must be fundamentally similar but sufficiently flexible so that they can be used by every company.

Accordingly, a process analysis and modification of the existing process in the automobile industry and a review of innovative concepts is required in order to establish unified testing for highly automated vehicles. To this end, PEGASUS will determine the required modification of existing and established metrics and processes (including functional safety) with a focus on the early phase of the product development model (V-model).

Building on this, and in close collaboration between the subprojects, the systematic scenario guidelines will be transferred into process steps taking into account system classifications and vehicle usage stages. Innovative concepts (e.g. breaking down the driving task) may additionally be transferred into processes which enable the derivation of further specific test cases for the driving task. For carrying out the actual tests, the requirements for simulation, test site and field tests and the test documentation in the form of guidelines and templates will be defined. The result will be a new and unified state-of-the-art in the field of development processes suitable for (highly) automated driving. Figure 7 shows the approach.

Fig. 7

Implementation process

4 Actual Testing

In order to achieve a safe (highly) automated driving function, it must be guaranteed that this function can handle all anticipated driving situations and is thus “safe enough”. Whether the accompanying proof is achieved by means of field tests, test site testing or in the laboratory/simulator is left open. This kind of proof would require several hundred million (based on accidents resulting in injuries) up to almost 10 billion kilometres of driving (based on fatal accidents) for a highly automated driving function on motorways, see [6, 7]. If we carried out this driving exclusively on the road, this would be highly uneconomic and would not be compatible with the time constraints within the automobile development cycle. Accordingly, when it comes to actual testing, PEGASUS focuses on achieving the greatest test coverage possible in the laboratory/in simulation (including software-in-the-loop (SIL), hardware-in-the-loop (HIL) or vehicle-in-the-loop (VIR) tests). It will nevertheless not be possible complete all tests purely in simulation; several models, for example in the field of sensor technology, are still showing too many weaknesses for this to be possible. Test results from simulation will need to be verified and validated on test grounds and in field tests.

Functional scenarios are the basis of all tests. At next logical scenarios with a specified parameter space will be developed from functional scenarios. A scenario instantiated with a concrete set of parameters out of this space deliver is called concrete scenario, ref. Fig. 8.

Fig. 8

Generation of scenarios: levels of abstraction

Which tests will be carried out in which environment will be defined in test preparation. Here, it will be ensured that the three test elements (simulation, test site, field tests) are closely meshed and complement one another. Thus test site tests verify simulation runs and field tests validate their results in turn. The basis for all test runs are the previously developed scenarios and quality levels and the processes and guidelines drawn up for the test. This information as well as the results of the tests will be centrally available in the test specification database. The quantity and quality of the test basis can be continuously increased by adding in new data and findings. The resulting toolchain will be put together in PEGASUS as an example in order to be able to demonstrate, test and evaluate practical fitness for purpose and usefulness. Figure 9 shows the schematic layout of the test procedure.

Fig. 9

Testing

In order that testing conforms to standards, in some areas we will need to change the way we think—a paradigm shift is required [8,9,10]—from “safety by test”, the testing of a black box system, to “safety by design”, in which there is complete system knowledge of the complex vehicle system. From sensors to actuators, the condition of all components with regard to system integrity is observable here. Function limits are determined using simulator runs and the established scenarios. The entire function space is established with testing along these limits. This defined test space can be verified on the test site primarily along the expected function limits. The objective of the “safety by design” approach is to ensure sufficient completeness within the function limits by systematically creating scenarios and methods for test coverage.

4.1 Simulation

The ideal criteria for efficient testing using simulation and laboratory environments are that tests should be reproducible, cost-effective and as complete as possible. Thus, in the ideal case, the expected results and a wide range of situations and environmental conditions are looked at in simulation, integrating all vehicle components.

In order to achieve comparable and consistent testing PEGASUS relies on a unified description format for test cases and test results. The same applies for the models, interfaces, formats and tools of the individual elements of the toolchain, resulting in a modular toolbox for virtual testing.

4.2 Test Site Test

New and existing test site facilities/test equipment which can be freely combined with one another to achieve the most flexible testing possible—like a toolbox—are selected and used depending on the requirements and situations to be tested. PEGASUS offers a generic approach to this which enables practical demonstration of the required quality measure. Simulation results are verified based on the test preparation.

4.3 Field Verification

In the last validation step, field verification validates simulation and test site results. In the field, critical situations in particular are considered in which the requirements for the successful introduction of a (highly) automated system which are defined by traffic and behaviour must be verified in a real-world environment. This will also result in new traffic situations which can be fed into the test specification database in the “replay2simulation” process and are then available for further investigations (e.g. in simulation).

5 Proof of Concept/Transfer of Results

Continuous transfer of PEGASUS results into project partners’ product development is one of the essential objectives of this joint project. This requires that the results achieved in the form of methods, processes and tools must be sufficiently robust and efficient.

Therefore, throughout the entire project, a proof of concept (see Fig. 10) is employed as well as a continuous bullet-point style check of individual essential elements using a maturity management system. At the same time, the consistency of the requirements analysis is ensured right through to final testing. The traceability of this chain is evaluated using a traceability concept. This offers the opportunity of quickly and efficiently adapting test cases at individual points in a targeted way as requirements change, without the need to carry out a complete re-evaluation.

Fig. 10

Reflection of results & embedding

6 Conclusion and Outlook

17 partners from science and industry are working together in the PEGASUS project to define new criteria and standards for the release of highly automated driving functions.

Current test procedures, as used today in driver assistance systems, cannot simply be used without further work because they are too time- and cost-intensive for highly automated driver functions and, most importantly, are specific to each manufacturer. With PEGASUS, we will in future be able to be quickly and efficiently transfer the results of research and development projects as well as already existing vehicle prototypes to market-ready products. Until June 2019 the project partners are therefore developing generally accepted methods and tools for validating highly automated vehicle functions and demonstrating these using the Highway Chauffeur example system.

As part of the interim project presentation in November 2017, an international symposium was held on 9.11.2017 with the title “How Safe Is Safe Enough?”. The results achieved so far were presented in the context of presentations by other international initiatives. More details can be found on the project webpage http://www.pegasusprojekt.de/en.