Recent incidents with autonomous vehicles in road traffic underscore the critical importance of extensively testing the safety of these vehicles before their widespread deployment. Despite significant advancements, challenges persist in the verification and validation, posing obstacles to their seamless integration into everyday transportation systems. Therefore, Siemens Digital Industries Software developed a robust chain of simulation tools for scenario-based testing.

Scenario-based testing, a virtual validation, and the adherence to the Safety of the Intended Functionality (SOTIF) standard play pivotal roles in assessing the safety of Advanced Driver Assistance Systems (ADAS) and systems for Autonomous Vehicles (AVs). These methods allow for thorough evaluation across a wide range of scenarios, mitigating the need for extensive public road testing. While traditional validation methods may fall short in capturing the complexity of real-world traffic scenarios, scenario-based testing and virtual validation provide insights into system performance under various conditions. By incorporating SOTIF alongside scenario-based testing and virtual validation, automotive manufacturers can enhance the reliability and safety of ADAS and AV systems, facilitating their successful deployment and integration into transportation networks.

The increased complexity necessitates a radical change of test methods and new concepts for comprehensive vehicle verification and validation in both the physical and the virtual world, which is captured in new regulations.

In February 2021, the United Nations Economic Commission for Europe (UNECE) presented the New Assessment/Test Method for Automated Driving (NATM) [1, 2]. This is a framework, which introduces a multi-pillar approach for safety validation of ADAS, Figure 1.

Figure 1
figure 1

Multi-pillar approach for safety validation of automated driving systems according to UNECE NATM [1] (© Siemens Digital Industries Software)

Full size image

In August 2022, the EU Commission adopted the 2022/1426 Regulation laying down rules for the application of the 2019/2144 Regulation of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type approval of the ADAS for fully automated vehicles [3, 4].

The multi-pillar safety validation of automated vehicles specifies five certification pillars which support the safety argumentation. Virtual testing, track testing and real-world testing are all scenario-based. As users move from virtual testing to real-world testing, the number of scenarios decreases, and the degree of reality of the scenarios increases.

After deployment, during the monitoring and reporting phase, the relevant scenarios are recorded and after scenario extraction and scenario selection they are fed-back into virtual testing, test track testing and real-world testing. The recording and extraction of new scenarios enable the discovery of unknown-unsafe scenarios and continuous improvement of the ADAS.

Finally, it is important to remark that with the EU legislation and UNECE NATM framework [1] (see the “in-service” pillar in Figure 1) a continuous integration and continuous deployment workflow is introduced, which is well-known in the software industry. What is needed is a software solution and an integrated tool chain that align with international standards and legislation, enabling scalable solutions for comprehensive safety validation workflows. The tool chain must facilitate automated and efficient scenario generation, leveraging recorded data to extract known safe scenarios, generate known unsafe scenarios, and identify unknown-unsafe scenarios. By addressing both the software infrastructure and scenario generation aspects in its products, a company such as Siemens Digital Industries Software contributes significantly to streamlining the validation process and mitigating potential safety risks in ADAS and AV systems.

Scenario-based Testing and Virtual Validation Solution

Siemens provides a tool chain that is designed to systematically compile an extensive database of scenarios, and to assess the AV stack effectively and reliably [5] through a combination of Model-in-the-Loop (MiL), Software- in-the-Loop (SiL) and Hardware-in- the-Loop (HiL) test platforms. Additionally, the tool chain serves to reiterate the verification and validation process on the basis of test results or new requirements.

The tool chain is composed of the three main phases requirements, extraction, and validation, which are described in more detail in the following. The requirements are gathered based on the Operational Design Domain (ODD), the Dynamic Driving Tasks (DDTs), and applicable regulations and standards. The requirement management tools from Siemens enable traceable and flexible integration of new requirements in the verification and validation process. Data collection in the ODD captures actor behaviors or patterns unique to the ODD. Siemens provides engineering services for sensor setup, data collection, and data processing.

The Siemens tool chain, including among other things Simcenter Autonomy Data Analysis (SADA) and the patented Critical Scenario Creation (CSC), enable the extraction of all categories of scenarios as per SOTIF. The SADA tool analyzes the data, extracts known scenario types, and applies safety thresholds to define safe and unsafe behavior. The CSC tool defines a search space over the recorded data. An optimization-based approach then systematically identifies known-unsafe and unknown-unsafe scenarios based on criticality and novelty.

Large-scale virtual validation of the AVs is performed in Siemens physics-based simulator, Simcenter Prescan, based on the extracted scenarios, together with relevant protocol scenarios from standards and regulations. The reliability of the results is further enhanced with Siemens offerings of HiL test beds.

Requirements Management and Data Collection

Requirements setting and defining a recording plan are the first steps in the safety assessment of ADAS and AV systems. The development of automated driving system shall start with definition/description of the ODD and the DDTs. Next, the applicable regulations and international standards are gathered, and the requirements definition of the automated driving system is initiated. The requirements are grouped in categories such as: safety requirements, system requirements, hardware requirements, software requirements, user requirements, etc. During the development process, existing requirements could change often as well as new requirements are defined, thus using a requirements management tool is a must. Siemens supports the requirements management with software tools like Polarion. The traceability, which refers to the ability to track and trace requirements to artifacts, test runs, and anything else in the product lifecycle is essential and mandatory in most cases, Figure 2.

Figure 2
figure 2

Process flow for the SOTIF scenarios in the Siemens tool chain for verification and validation of ADAS: The arrows in green show scenarios from safe-known, the orange and red arrows show the known- unsafe and unknown-unsafe scenarios, respectively. The purple arrows show the input points of ODD and DDTs to different tools. (SCDBs: Scenario Databases) (© Siemens Digital Industries Software)

Full size image

Data collection of trajectories can be performed for the ODD and the individual scenes through instrumented vehicles, static drone footage, or infrastructure-mounted sensors. Siemens can assist with setting up a comprehensive sensor setup designed for the recording of real-world scenarios. The collected data can contain errors due to noise, missed detections, occlusions, etc. Thus, some data processing is needed to extract full trajectories and remove signal noise.

Scenario Extraction and Creation

The Siemens tool chain facilitates the extraction of all categories of scenarios in accordance with the SOTIF requirements. The Simcenter Autonomy Data Analysis tool gets the recorded raw data and by applying analysis on the data provides the following features:

  • extracting and replaying scenarios with the perception of the environment

  • categorizing the scenarios and evaluating them based on key performance indicators

  • diving into the details of a categorized scenario and selecting data of interest manually

  • exporting raw data for replay or for simulation via OpenDrive and OpenScenario (both standards from ASAM), including validating of the simulation models.

The Critical Scenario Creation tool contributes to the scenario generation process in two folds. First, it provides the possibility of generating known-unsafe scenarios, see orange arrow in Figure 2. For this purpose, the criticality of known-unsafe scenarios extracted in the previous step using the Simcenter Autonomy Data Analysis tool is optimized. Second, it provides a method to generate unknown-unsafe scenarios, see red arrow in Figure 2.

Figure 3 shows the main three steps of the CSC tool. More details about the tool can be found in [6]. The feature extraction provides a finite set of features which can describe the behavior of actors (for example, automobiles) in the scene. In order to extract features for the actor's behavior, the road layout is described as a graph: The probability distributions are determined for each parameter and nodes combination. Pa,i is the probability of actor i and is calculated as mentioned in Eq. 1:

Figure 3
figure 3

The main three steps extraction, configuration and identification of features in the CSC tool in order to generate critical scenarios (© Siemens Digital Industries Software)

Full size image

Eq. 1 \({P}_{a,\phantom{\rule{0em}{0ex}}i}=\phantom{\rule{0em}{0ex}}{P}_{p,\phantom{\rule{0em}{0ex}}i\phantom{\rule{0em}{0ex}}}\times {\prod }_{j=1}^{m}{P}_{par,j}\)

Here, m is the number of parameters extracted for the actor i with the probability of Ppar,j as it was aggregated from the collected data. Pp,i is the probability of a path cluster for an actor's type. To streamline optimization, the software users configure the behavior of the actors and the parameters to define the search space. However, considering all possibilities leads to a large and unmanageable space. The tool automatically identifies non-interacting actor paths and parameters, then partition the remaining space based on discrete parameters. This creates manageable optimization studies.

The identification phase evaluates risk severity using proprietary indicators to assess criticality and novelty. It employs an optimization algorithm to identify scenarios with the highest potential impact. The objective function is formed according to Eq. 2:

Eq. 2 \(f\left(\eta \right)=G\left({P}_{s}\left(\eta \right)\right)\times \left(2-ϵ\left(\eta \right)+TT{C}_{min}\left(\eta \right)\right)\)

This incorporates three terms: a proprietary unexpectedness metric (ϵ(η)) to identify unknown scenarios (a constant value of 2 is added to the objective function to ensure that its value remains non-negative.), Time to Collision (TTC) to measure the scenario safety, and a function (G(Ps)) of the scenario probability. Here, Ps is calculated according to Eq. 3:

Eq. 3 \({P}_{s}={\lambda }_{TL}\times {\prod }_{i=1}^{n}{P}_{a,i}\)

Here, λTL is the traffic light factor. The optimization problem can be formalized as follows in Eq. 4:

Eq. 4 \(\begin{array}{c}\underset{\eta }{\mathrm{min}}f\left(\eta \right)\\ s.\phantom{\rule{0em}{0ex}}\phantom{\rule{0em}{0ex}}t.\phantom{\rule{0em}{0ex}}\phantom{\rule{0em}{0ex}}\phantom{\rule{0em}{0ex}}\phantom{\rule{0em}{0ex}}\eta \in \chi \\ {\zeta }_{col}=0\end{array}\)

Here, χ defines the bounds on the parameter values. The value ζcol is a Boolean variable and is 1 in the case of collisions with non-ego vehicles.

Various scenario databases are utilized during assessment, including those from standards, experts defining scenarios based on ODD, critical scenarios, and accident databases. For ADAS, the Euro NCAP, UN R131 (AEBS), UN R152 (AEBS), and UN R157 (ALKS) define test scenarios. For AVs, ISO standards such as ISO 22737 and ISO/DIS 23374-1 describe the test scenarios. The Siemens tool chain enables virtualizing and testing of these scenarios before testing in road traffic. While standards provide a baseline for safety and performance, additional requirements defined by experts are often necessary during development and post-deployment.

Assessment and Digital Twins

All collected scenarios from the database are tested to reveal the digital twin behaviors under varied and complex conditions. Transitioning from the virtual system to the physical system, the automated driving system, embedded within a control unit, faces the ultimate tests. This transitioning process is executed in (at least) two steps, known as SiL and HiL testing.

For the design of AV control software, the “sense, plan, act” paradigm structures the development into perception, planning, and control algorithms. Each team develops a simplified model of the other two components, which are integrated in each other to create the full software stack. Testing occurs in a virtual environment with vehicle dynamics, sensor models, and infrastructure simulation. This integration presents challenges, particularly regarding sensor model quality, as simulated sensor data must match real-world functionality to create realistic scenarios for the AV.

The final step before the actual deployment in the vehicle entails HiL testing of the advanced vehicle control system, running in real-time on the intended hardware platform, Figure 4. This allows the assessment of the control system's responses to the selected scenarios.

Figure 4
figure 4

HiL architecture with limited sensor suite of two cameras and one lidar system (NIC: Network Interface Controller, GMSL2: Gigabit Multimedia Serial Link 2, PCIe: Peripheral Component Interconnect Express, UDP: User Datagram Protocol) (© Siemens Digital Industries Software)

Full size image

To this end, the same virtual environment can be used as for SiL testing, with the added non-trivial requirement that it must run in real-time. The real-time hardware platform also requires sensor inputs. These have to satisfy specific and often sensor-dependent communication protocols, since it is the exact same platform that will be deployed in the real vehicle. Hence, the environmental simulation must output the simulated sensor signals in these formats. The simulation hardware can be scaled based on sensor requirements of the AV system under test.

As an example, a HiL setup is considered, assuming a very limited sensor federate consisting of two cameras and a lidar system. The environment contains the vehicle dynamics and the sensor models. Since these high-fidelity models are computationally expensive, they might be implemented in a federate architecture. The camera models output their data in a dedicated packet-based protocol operating at a fixed GMSL2 link rate whereas the lidar system employs the standard UDP protocol. As a result, the ADAS control unit receives the data in exactly the same way as it would in reality.

After the real-time performance is assessed, the data recording plan can be updated based on the previous assessment. This ensures that the vehicle is prepared for the unpredictable nature of real-world driving.

Summary

The tool chain presented by Siemens offers a comprehensive solution to accelerate the deployment of autonomous vehicles while ensuring their safety and reliability. By uncovering unsafe scenarios as per SOTIF standards, the tool chain facilitates an accelerated deployment of these vehicles through robust testing methodologies, an enhanced system robustness by increasing scenario coverage, an increased confidence in system performance for both internal and regulatory key performance indicators, as well as a reduction in development costs by streamlining validation processes. In conclusion, the tool chain provides a seamless and scalable approach covering three key areas of SOTIF scenarios, thus paving the way for the widespread integration of AVs into transportation systems.

References

  1. [1]

    UNECE (ed.): New Assessment/Test Method for Automated Driving (NATM) Guidelines for Validating Automated Driving System (ADS). Online: https://unece.org/sites/default/files/2022-05/WP.29-187-08e.pdf, access: May 23, 2024

  2. [2]

    UNECE (ed.): Guidelines and Recommendations concerning Safety Requirements for Automated Driving Systems. Online: https://unece.org/sites/default/files/2022-06/WP.29-187-10e.pdf, access: May 23, 2024

  3. [3]

    European Union (ed.): EU Regulation (EU) 2019/2144 of the European Parliament and of the Council on type-approval requirements for motor vehicles and their trailers, and systems, components and separate technical units intended for such vehicles, … Online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32019R2144&from=EN, access: May 23, 2024

  4. [4]

    European Union (ed.): Commission Implementing Regulation (EU) 2022/1426 laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles. Online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32022R1426, access: May 23, 2024

  5. [5]

    Kalra, N.; Paddock, S. M.: Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?. In: Transportation Research Part A: Policy and Practice 94 (2016), S. 182-93

  6. [6]

    Singh, T.; Hassel, E. van; Sheorey, A.; Alirezaei, M.: A Systematic Approach for Creation of SOTIF's Unknown Unsafe Scenarios: An Optimization based Method. SAE Technical Paper, Online: https://doi.org/10.4271/2024-01-1966, access: May 23, 2024