Keywords

1 Introduction

Statistical data of traffic accidents shown in Fig. 1 reveals that pedestrian fatalities are a particular priority in Europe and Japan. For instance, Figs. 1 and 2 show that pedestrian fatalities account for about 35 % of all road fatalities in Japan [1], about 12 % in US [2] and about 17 % in Europe [3]. Most pedestrian accidents relate to situations when pedestrians are crossing a road, as indicated in Fig. 2 [4]. Advanced driver assistance systems which implement technical solutions for pedestrian protection could help to lower the number of these fatalities. Current approaches make use of stereo cameras, millimeter wave radar and stereo camera sensor fusion or stereo infrared sensors for example (see [5–10]).

Fig. 1
figure 1

Pedestrian fatalities around the world

Full size image
Fig. 2
figure 2

Detail of pedestrian accidents in Japan from the traffic-accident database in 2007 (The right chart shows situations in which accidents with pedestrians occur.) [4]

Full size image

In this research we scope on the use of monocular camera and millimeter wave radar. The monocular camera, practically used for several existing functions of driver assistance systems, has potential to be also used for pedestrian classification [11, 12], while the millimeter wave radar can be used to directly measure the distance from the host vehicle to pedestrians for a precise collision risk assessment.

Two approaches of pedestrian detection will be considered. The first approach is based on a sensor fusion technique which combines the information of moving objects from a millimeter wave radar and the crosswalk location information from a monocular camera. The second approach employs an appearance-based pedestrian classification algorithm which leads to a high computational cost in image processing.

To reduce the computational cost of the pedestrian classification algorithm, the authors have presented the effectiveness of the first approach in [13], presenting an alternative approach of the pedestrian detection algorithm based on the assumption that moving objects on a crosswalk can be interpreted as pedestrians. Consequently, the pedestrian classification module is not required in the image processing task, and it is necessary to develop an algorithm to detect the existence of crosswalk in front of the vehicle. This approach does not claim to cover 100 % of all pedestrian accidents, but has the advantage of high robustness, low false alarm rate and cost efficient implementation.

2 Experimental Vehicle

An experimental vehicle equipped with a monocular CMOS camera and a PC for data acquisition and processing is used to investigate the feasibility of the proposed approach. Sensors used in production vehicles are utilized: a monocular CMOS camera which can be used for the purposes of lane marker detection, traffic sign recognition, or vehicle detection. The sensor and measurement system configuration on the experimental vehicle is shown in Fig. 3.

Fig. 3
figure 3

Experimental vehicle

Full size image

3 System Design Principle

As the concept of the system, moving objects on and nearby a crosswalk are interpreted as pedestrians. Consequently, the image processing algorithm based on the images captured from the on-board camera is designed to detect the existence of a crosswalk in front of the vehicle. The feature extraction technique used in the algorithm is based on the value of cross ratio of crosswalk edges and the periodicity of the crosswalk paints. Then, the moving object position and velocity obtained by the optical flow computation are utilized to infer that a pedestrian is on the crosswalk in front of the vehicle. Figure 4 shows an example of image data by the measurement and testing system on the experimental vehicle. A warning can be provided to the driver when there is a potential of a collision. Figure 5 shows the schematic diagram of the pedestrian collision warning system.

Fig. 4
figure 4

Crosswalk and pedestrian image taken by the camera

Full size image
Fig. 5
figure 5

Schematic diagram of the pedestrian collision warning system

Full size image

4 Crosswalk Detection

To detect a crosswalk, a feature called “cross ratio” of four collinear points is utilized. Since the cross ratio is invariant under the perspective projection, the cross ratio of the crosswalk in 3D space is the same as that of its image (Fig. 6). Here, the cross ratio [abcd] of four points a, b, c, d are calculated as follows [8]:

Fig. 6
figure 6

Description of cross ratio in each coordinate

Full size image
$$ [abcd] = \frac{{\overline{ac} }}{{\overline{bc} }} \cdot \frac{{\overline{bd} }}{{\overline{ad} }} $$
(1)

where, the symbol written as \( \overline{ab} \) refers to the distance between the point a and the point b.

The flow diagram of our crossing detection algorithm is shown in Fig. 7. First, to extract boundaries of the white lines on a road, an image captured by the camera is processed by two algorithms: binarization and edge detection. To extract a number of edges from the input image, Canny’s algorithm is employed [9]. Next, an AND operation of the two processing results is calculated in order to reduce the errors influenced by shades on roads. Then, the system classifies the pattern of vertical edge change into two categories: (1) the boundary where the intensity changes from dark to bright, and (2) the one from bright to dark.

Fig. 7
figure 7

Process flow for crosswalk detection

Full size image

Finally, we calculate the cross ratios of two pairs of boundaries. Here, it is assumed that the vehicle is driving perpendicular to crosswalk, therefore the algorithm calculates the cross ratio from four points (a, b, c, d in Fig. 6) which have the identical vertical axis level (v) on the image plane. In this step, we reduce the candidates of crosswalk by using the periodicity of the crosswalk by checking the number of boundaries at the identical vertical axis level (v) on the image plane and the width between two edges. Then, the cross ratio [abcd] of four points a, b, c, d are calculated according to the formula in Eq. (1).

According to road traffic regulations in Japan, the value of cross ratio is within the range from 0.203 to 0.309. The crosswalks in Germany are in the range between 0.198 and 0.316. The crosswalk detection is made more robust by the following heuristic. The crosswalk is detected when there are more than three pairs of edge change patterns at the same vertical axis level (v) in the image plane and more than two pairs of edges which are within the range of the cross ratio of crosswalk described above. Moreover, the continuity of the crosswalk detection result at each sampling time is checked with the detection results from six previously-acquired image frames in order to reduce misdetections.

In addition, a tracking algorithm by using the optical flow of four corners of the crosswalk is employed to improve the continuity in the crosswalk detection algorithm. An example of optical flow computation result is shown in Fig. 8.

Fig. 8
figure 8

Optical flow of four corners of crosswalk

Full size image

Urban road image sequences in Japan which include a crosswalk in front of the vehicle are used as a test data set to evaluate the performance of the crosswalk detection algorithm. The detection performance was evaluated by the distance to each crosswalk. The performance indices True Positive and False Negative are used here. True Positive refer to the cases that the system can correctly detect crosswalks, whereas False Negative refer the cases that the system fails to detect crosswalks where it should detect. The evaluation result is shown in Fig. 9. The system shows high true positive rate over 90 % when the distance to crosswalk is long up to around 20 m.

Fig. 9
figure 9

Crosswalk detection performance by the distance

Full size image

5 Optical Flow for Pedestrian Detection

Figure 10a shows the flowchart of the pedestrian detection algorithm based on optical flow. First, the region of interest (ROI) which covers the detected crosswalk are extracted based on the estimated distance to the crosswalk. Then, the optical flow is computed inside the region of interest (ROI) to get the optical flow vectors. Next, the optical flow vectors which deviate from the epipolar lines and the distance between the epipolar lines and the edge of the optical flow vector are larger than a certain value are classified as “Outliers” as shown in Fig. 10b. The outliers which are close together are grouped as moving objects. In the proposed method, the moving objects exist on the crosswalk, so the objects can be inferred as pedestrians. Figure 11 shows an example of optical flow computation result for pedestrian detection. The red arrows show the outlier deviated from the epipolar lines.

Fig. 10
figure 10

Flowchart of pedestrian detection by optical flow. a Pedestrian detection flowchart. b Focus of Expansion (FOE), Epipolar lines and outliers

Full size image
Fig. 11
figure 11

Pedestrian detection result by optical flow

Full size image

Figure 12 shows the pedestrian detection result by the integration of the crosswalk detection and the pedestrian detection system. The crosswalk can be detected at a distance of 19.0 m and the system indicates “CROSSWALK IS DETECTED” message. When the system detects the frontal pedestrian, the message “CROSSWALK OCCUPIED” is indicated.

Fig. 12
figure 12

Detection result of a pedestrian on a crosswalk

Full size image

6 Conclusions

This paper described an on-board pedestrian detection and collision warning system by using a monocular camera. The design principle proposed here to deal with the pedestrian detection problem is based on the assumption that objects moving along a crosswalk can be inferred as pedestrians under defined circumstances. This principle simplifies the image processing unit computation as it does not require a computationally expensive appearance based pedestrian classification method. A key concept of the approach is the detection of a crosswalk by image processing methods based on the cross ratio feature. Moving objects on the crosswalk can be detected with the application of the optical flow feature. The sensor setup on the experimental vehicle for pedestrian detection was presented and experimental results which show that the proposed sensor fusion approach works in practice were given.

  • In future, to improve the robustness of the proposed pedestrian detection method, the following investigations will be conducted.

  • The investigation of the robustness of the crosswalk detection algorithm against weather conditions.

  • The effectiveness of the proposed pedestrian protection system will be analyzed based on statistical data.

  • The pedestrian collision risk assessment will be quantitatively conducted for the design of a pedestrian collision warning system.