Keywords

1 Introduction

Advanced Driver Assistance Systems (ADAS) are developing rapidly. They rely on the environment perception system to assist and support the driver to avoid accidents. This development urges a need for test methods and test tools for the assessment of different aspects of these systems. Beside verification of the functions, it is also necessary to validate the positive effects of these systems on traffic safety. To identify the requirements for these more global evaluations, the whole system of the vehicle, the driver and the environment, has to be taken into account.

Existing testing methods and testing tools for ADAS have a varying spectrum of purposes. Many of these methods are designed to verify the sensor detection abilities or the performance of the hazard identification and warning algorithm of an ADAS function. Their purpose is to prove with measurements that a specific attribute or feature of the product fulfills the defined requirements. Therefore, these testing methods and tools need to be highly specific for the given attribute.

Regarding the purpose of an ADAS system in real traffic situations, the intended benefit is often global in terms that it is inclusive, because the systems boundaries used for the assessment include the driver in the vehicle within the traffic situation. Example intended benefits are “reduce the accident rate” or “reduce the severity of accidents”. Moreover, it needs to be proven that the benefits outweigh the potential downsides, like hazards due to failures or false reactions. In case of potential failures, these tests are required to obtain the functional safety approval of the system according to ISO 26262 (2009, Part 3, p. 6) . False reactions are not addressed in this standard, although the resulting situation for the driver may be similar.

These “global” characteristics of a system cannot necessarily be derived from the functional requirements, as they are highly dependent on situations and usage of the vehicle. Therefore, test methods for the characteristics need to be highly relevant for the field, must take into account the whole system of the driver with the vehicle in the environment and must allow a transferable assessment which is independent of the specific system.

2 Requirements for the Assessment of Global Characteristics

To assess the global characteristics, suitable test situations have to be identified. These situations must clearly reveal the characteristic and must be evaluated and ranked as to their relevance in real traffic. Therefore, the characteristic and its influencing factors need to be analyzed and connected to the situational factors. By ranking the influencing factors and identifying their relevance and impact, the minimum number of test situations will be determinable so that the test effort can be kept to a minimum. The relevance of a situation is highly dependent on the usage of the vehicle. For this reason, the definition of the situation spectrum depicting the use profile is crucial.

Based on the situations, objective values must be defined, describing the attribute in a common and transferable way. For the benefit of collision mitigation systems, for example, Hoffmann (2008, p. 34) uses the reduction of speed before the crash, standing proxy for the reduction of crash energy.

In addition, the identification of an absolute reference for these objective values increases transferability and comparability to similar systems and enables preceding risk assessment methods to be carried out, such as the “Hazard and Risk Assessment” according to ISO 26262 (2009, Part 3, p. 6).

3 Controllability Assessment in ISO 26262

An example for a global characteristic of an ADAS system is the controllability of its functions in case of unintended reactions. If the system is working as intended, it will reduce accidents, but relying on an environmental sensor system can also lead to false reactions by the function.

In such cases, the driver or other involved persons must be able to reach a safe state of the vehicle and avoid an accident. This is referred to as “controllability” of the event. For functional safety, ISO 26262 defines requirements for safety processes for electric and electronic systems in the automotive industry. Controllability is therein defined in classes describing the percentages of persons who are able to control a hazardous situation (see Table 7.1).

Table 7.1 Classes of controllability (ISO 26262 2009, Part 3, p. 9)
Full size table

In combination with the classification of the Exposure and Severity of a potential hazard, ISO 26262 determines an Automotive Safety Integrity Level (ASIL) which defines the minimum safety requirements for hard- and software components of the function. ASIL D, for example, describes the maximum allowable hardware failure to be less than \(10^{-8}\) per operating hour.

The recommended methods and processes for the controllability assessment within the meaning of ISO 26262 are summarized in the “Code of Practice” (PReVENT 2009, p. 13ff). The Code gives as an example that for proving controllability at level C2 by a car clinic with naïve test subjects, a minimum of 20 valid data sets with positive controllability is needed, resulting in a minimum of 85 % of controllability, which is considered to be sufficient for C2. With the same approach and at the same level of confidence (\(\gamma = 95\) %), to prove a controllability of 90 %, 29 test persons are needed without a single case of uncontrollability occurring. Formula 7.1 describes the criterion for the number of test subjects needed, as a function of the level of confidence and the proportion of controllability required for approval.

$$\begin{aligned} {{n}}=\frac{\mathrm{log}_{10}({1}-{\gamma })}{\mathrm{log}_{10}({\mu }_{{\varvec{x}}})} \end{aligned}$$
(7.1)
$$\begin{aligned}&n: {\text{ Number } \text{ of } \text{ test } \text{ subjects }}\\&\gamma : {\text{ Level } \text{ of } \text{ Confidence }}\\&\mu _x: {\text{ Required } \text{ proportion } \text{ of } \text{ controllability }} \end{aligned}$$

In the “Code of Practice” (PReVENT 2009, p. 15) for the approval of C1, the effort for a statistical proof by tests with subjects is stated as too high. Again using the explained approach, at this level 299 test persons will be needed without accepting one negative test result. If negative results occur, the number of tests needed increases. As the minimum number of test persons is defined by the 5 % limit (see Formula 7.1), the rate of occurrence of this specific event (in this case, zero uncontrollable tests) is 5 % as well. This “success probability” of controllability testing, describing the likelihood that a controllability level can be proven with a specific number of test subjects depending on the expected controllability within the collective of drivers, can be calculated (see Table 7.2).

Table 7.2 Number of tests needed versus success rate dependent on controllability proportions (Weitzel and Winner 2012, p. 19)
Full size table

For low controllability proportions the success rate is low, even at a higher numbers of tests. Vice versa, for the example described in the “Code of Practice” (PReVENT 2009, p. 15), even if the controllability in the collective of drivers is on level C1, the testing will only have a 75 % chance of success. In most cases, if no transferability can be proven such an effort is unjustifiable, especially because it has to be repeated for each system specification. Therefore, for the separation of classes C1 and C2 expert judgment is a commonly used method (Fach et al. 2010, p. 429).

4 Controllability Assessment for Unintended Reaction Scenarios

Apart from difficulties for the objective assessment of controllability on a higher level than C2, the focus of the ISO 26262 shows another challenge in its application to ADAS functions. The standard is intended for the functional safety of electric and electronic systems for vehicles up to 3.5 t. But hazardous situations caused by ADAS functions are not only a result of hard- or software failures. As they rely on environmental sensor systems, unintended reaction scenarios can be a result of incomplete information about the environment, e.g. due to limited capabilities or the number of available sensors, or also a misinterpretation of the situation by either the driver or the system. In these cases the system works within its specification but nevertheless the reaction is unintended. In addition, these “failures” are difficult to detect and additional sensors do not necessarily solve this problem, as they relocate the problem to the question of which sensor should be trusted—a strategy that increases costs rapidly. The expectable rates of misdetection and misinterpretation of a system are closely connected to the situation and to the utilization profile and cannot be determined in the same way as for hardware components.

This issue is not clearly addressed by ISO 26262. However, as the effects for the driver or other involved persons are considered to be equal, the methodology and testing described in the standard and according to the Code of Practice (PReVENT 2009, p. 15) should be feasible for these questions. Nevertheless, the transferability of the ASIL to this problem, in the meaning of absolute failure rates, is questioned in some cases (Ebel et al. 2010, p. 396). The appropriate limiting risk measure depends on the system boundaries chosen. If the system for itself is taken as a function within the car, technically motivated levels like in ASIL should be useful. If the system boundaries consider the vehicle within its environment, the limiting risk is more to be seen in comparison to the driver capabilities.

5 Analysis of Unintended Reaction of ADAS

In order to be able to identify the minimum number of test scenarios needed to prove relevance for the field, the unintended reaction is examined and categorized. In parallel, the driving situation is analyzed in order to identify characterizing situational factors. These situational factors are then discussed and evaluated in relation to the categories of the unintended reaction. In combination with the anticipated probability of occurrence of the situational factor, the overall relevance of the factor is determined. Thereby a ranking of the situational factors on the unintended reaction scenario should be possible. Based on this, the minimum set of needed test situations can be identified. In the following these test situations are referred to as “Necessary Test Cases”. The approach is to start at a “Best Case” of controllability for the system and add situational elements which potentially diminish the controllability. In combination with the probability of occurrence, their relevance could be identified similar to the ASIL Matrix in ISO 26262 (2009, Part 3, p. 10) (see Fig. 7.1), even though, as discussed earlier, the absolute levels are not transferable.

Fig. 7.1
figure 1

ASIL determination matrix (ISO 26262 2009, Part 3, p. 10)

Full size image

For the definition of the causes of unintended reactions of an ADAS function, two approaches are possible. The first approach assumes that the reaction is unintended by the driver and contradicts the planned maneuver or anticipated behavior of the vehicle or the system. The second approach is based on an objective situation assessment, choosing the best available option within the situation and comparing the actual reaction of the system to that “best option”. However this “best option” is not clearly definable in many situations, especially if it is necessary to anticipate the situation development including the behavior of other traffic participants. Even an “after-the-fact” evaluation, where the situation development is completely known, can be difficult, as the other traffic participants may have reacted differently if the reaction of the analyzed vehicle had been different. The two approaches are combined to an unintended reaction characterization tree as shown in Fig. 7.2.

Fig. 7.2
figure 2

Characterization tree for unintended reaction of ADAS

Full size image

As the aim of the test situation is to reveal the controllability of the unintended reaction, the situation should be as distinct as possible for the test person. Therefore, the most appropriate cases out of the characterization tree need to be identified. It has to be considered that after the incident, the driver will try to build an internal model of the system functions that match the experienced behavior (König 2012, p. 36). A contradiction between the driver’s point of view and the “after-the-fact” evaluation of the situation, therefore, will cause bias, as the driver’s reaction within the situation could be delayed due to additional decisions needed and an inconsistent internal model. To avoid this, in the definition of the Necessary Test Cases, both approaches should lead to the same subjective and objective conclusion. Assuming that only false reactions are critical, just the cases number 4 and 8 are suitable for controllability assessment.

The technical causes of unintended reaction are highly dependent on the sensor system, its signal processing, criticality estimation algorithm and rules for decisions about actions. The analytical identification of causes dependent on situational factors will lead too far into the field of sensor technology and post processing and will not be addressed here. An exception is, if the driver is planning a maneuver in the near future, e.g. an overtaking or lane change. In this case, as long as ADAS are not able to detect the driver’s intentions, it is supposed that an unintended reaction is more likely.

To get a real and relevant situation where the driver is urged to intervene, a potential hazard must be perceivable (Muttart 2005, p. 3). Following the argumentation of ISO 26262, this hazard is a pending crash. Considered are two types of crashes, colliding with objects, including leaving the road, or colliding with other vehicles/other traffic participants. In the second case, the likelihood of hazardous objects depends on the traffic density. The objects are moving and controlled by a human being and may be able to avoid the accident by their own actions. So their ability of controlling the situation needs to be taken into account as well.

In summary, to compose a controllability situation, the cause must be distinct for the concerned person and the hazard must be present and urgent to trigger a reaction. This urgency is due to the limited time for reaction. Also, enough time for reaction must be available to enable the person to maintain control. This narrows the time span where controllability could be observed. This controllability time span is limited by two values, the last distance where a collision is just avoidable (at this point the time-to-react (TTR) is zero) (Hillenbrand 2008, p. 112ff), and the maximum distance where the situation can be perceived as critical by the driver. In addition, the reaction’s dependency on internal/mental information processing and driver capabilities and character induces variation. Information processing in situations of unfamiliar vehicle behavior cannot be explicitly divided into phases with distinct values or described as a combination of these phases, because the processes can be serial and/or parallel and are not independent of each other (Olson and Farber 2003, p. 321). A suitable and valid driver model for the described problem would be the solution, but is not known. The existing driver models are able to address the typical driver behavior, but for controllability figures of 90 or 99 % the problem of the high numbers (see Table 7.2) occurs again and obviates any validation. As the individual processes cannot be observed in isolation, the minimum observation time span and other mental processes possibly involved have to be determined. However, the test results of naïve persons will always show scatter, sometimes with very broad variances. To achieve transferability, the source of the scatter and offset in the experiments must be analyzed. A differentiation must be made between scatter due to variation of driver behavior in steady situational conditions and scatter and offset due to variation of situation. This allows a prediction about the impact of situational variations and thereby simplifies the selection of relevant test cases. To enable such analysis, the assessment criterion for controllability has to be defined.

6 Assessment Criterion for Controllability

In the Code of Practice (PReVENT 2009, p. A50) the criterion for controllability is a nominal scale value. It differs according to whether or not the crash is avoided by the reaction of the considered person (Fach et al. 2010, p. 431). This binary assessment criterion is needed in the course of the risk level determination. In general, risk is determined as the probability of occurrence of a given hazard and its severity (ISO 31000 2009, p. 8 and 13). The assessment is based on the probability that the virtual hazard is transferred to a real accident. For testing with naïve persons the option of a real accident is not available. At the same time, the situation must be perceived as threatening by the person to provoke relevant reactions (Muttart 2005, p. 3 and 11). In some cases, deformable targets are used which are very close to the real accident but still with some trade-offs in real appearance. As discussed in the last section, to assess controllability the reaction must be observed to the point where no time to react is left. Subsequent reactions will only reduce the severity of the crash, which is assumed to be determinable by calculation. In summary, a real contact is not necessarily needed for the controllability assessment, as long as the last possible distance of reaction, called the point-of-no-return is included in the assessment period. If the driver reacts within the period before the point-of-no-return, the next question is whether the reaction was appropriate and intense enough to avoid the collision. Defining the point-of-no-return as the last possible moment for starting a lateral or longitudinal intervention means that the intensity of the counteraction needed for intervention increases from the start of the situation to that point where it reaches its maximum, limited by the maximum longitudinal or lateral force available. Time lags due to response characteristics of the system have to be added to this.

7 Situational Analysis

Theoretically, many factors influence a driving situation. Moreover, the possible detailing of these factors is nearly unlimited. A common approach is to classify these parameters in three parts (König 2012, p. 34):

  • Environment: e.g. other traffic participants, weather, lighting, road condition...

  • Driver: e.g. driving capabilities, internal model, attention, fatigue, character...

  • Vehicle: e.g. vehicle behavior, speed, acceleration...

Many of these factors have interdependencies, for example all vehicles and drivers are influenced by the weather and therefore their driving behavior could change. For the controllability assessment, choosing many situational parameters and detailing the situation will cause a reduction of the probability of occurrence and thereby reduce the impact of the specific situation on the risk assessment. As a starting point, the unintended reaction is divided into three elements: causes, hazards, and reaction delaying factors. The influences of situational parameters in these three elements have to be discussed. To allow this, a start set of parameters for a generic situation definition is used and discussed in terms of impact on controllability. They are divided into two parts, parameters characterizing the state of the driver and environmental parameters describing the situation of the vehicle(s) and driver.

7.1 Driver Parameters

The driver influence factors according to Kopf (2005, p. 119) can be divided into three classes, depending on the rate of change. The slowest changing factors change over months or years, e.g. driving experience, learning effects or character of driver. For the present purpose they are not considered as explicit factors, but are taken into account by choosing an appropriate driver collective. The next class includes factors changing within a day or hours, e.g. drowsiness, drugs/alcohol. ISO 26262 (2009, Part 3, p. 9) defines that the driver should be in good constitution, therefore, influences due to these factors are also not considered. The last class comprises factors that change within minutes or seconds and therefore are relevant for the described tests because these factors can change within the test situation and thereby directly change the driver’s behavior. According to Kopf (2005, p. 119), these factors are:

  • Driver intention

  • Situation awareness: concluded from cognition, perception and anticipation

  • Looking away: as a conscious decision

  • Visual distraction: due to a prominent stimulus outside the focus

  • Vigilance/Attention

  • Strain/Stress

From the above list, the driver intention is outstanding as it comprises the maneuvers planned in the near future and thereby, for example, influences the viewing direction (like to mirrors).

Following the best case approach mentioned above and the analysis of the unintended reaction, the consequences of variation in these parameters are estimated or assessed based on existing studies and data. This results in a ranking of situational factors that are expected to change the controllability.

To increase transferability and reduce test effort, driver models are very popular to simulate the driver behavior in different situations. In specific cases of driving dynamics, the driver is characterized as a transfer element (Mitschke and Wallentowitz 2003, p. 642 ff) in control theory. For unintended reaction scenarios, including warning elements, the information perception and processing of the driver is expected to consist of parallel and serial elements that are superimposed and interact with each other. These elements cannot be observed in isolation and feasible driver models are not known. However, if relevant driver parameters for unintended reaction scenarios can be identified and measured with appropriate accuracy, conclusions about the requirements for a driver model will be possible.

7.2 Environmental parameters

Different publications deal with the generic definition of potential driving situations (Domsch and Negele 2008, p. 7; Fastenmeier 1995, p. 48ff), or catalogues of situations. Reichart (2001, p. 52) focuses on situational parameters describing the road and environmental conditions which can be considered to be analogue to each other. After adaptation and simplification to match the requirements of the controllability assessment, the factors used are summarized in Table 7.3.

Table 7.3 Environmental factors
Full size table

For the risk assessment of ADAS, the level of detailing of these influence parameters is crucial. By detailing the influence parameters, the overall rate of occurrence for the then more specific situation is lower. Assuming that the safety level needed for the system is derived from the highest risk of all situations, detailing will decrease the required safety level. A strategy to cope with this challenge is needed. It must take into account the detailing level of the different factors influencing the assessment of the controllability and allow a quantification of the coverage of situations at the different detailing levels. Although these requirements seem to be demanding, they must be fulfilled to enable a relevant and objective controllability assessment.

8 Conclusion and Outlook

For the assessment of the global characteristics of ADAS a more global/inclusive approach is needed which takes into account the whole system of traffic situation, vehicles and the driver and other involved persons. For the assessment of controllability, another challenge is added, as there is no predefined “use-case” for the controllability of unintended reactions of ADAS.

In addition, it is not economically feasible to address all possible unintended reaction scenarios. Therefore, it is necessary to identify a minimum number of relevant and reliable test scenarios for approval. The approach described here has been developed to this end. The unintended reaction has been analyzed and matched with factors for a generic situation definition. Based on this, the definition of categories of situational factors was proposed. This will allow the definition of necessary test cases and a rating of their relevance. The next steps are to carry out a detailed matching of the situational parameters with the elements of the unintended reaction and examine the feasibility of concrete relevance factors. The achievable accuracy for the factors has then to be analyzed. In addition, criteria for controllability need to be developed to enable a more detailed and transferable assessment. If this approach is applicable and successful, it can reduce the effort for approval testing. If it is not suitable, the reasons have to be discussed and the implications for the development of ADAS have to be concluded.