1 Introduction

One of the major causes of fatal accident on the road is due to the improper operations performed by drivers, which are caused by misjudgement, drowsiness and inattention to the surrounding vehicles. Within Malaysia, which is a relatively small country, 548,598 cases of accident were reported in year 2018, while many are not reported. To ensure better safety and pleasant condition on the road, many innovations are put forward as means to reduce accidents, including lane departure warning system, collision avoidance system, autonomous cruise control system, to name a few. However, these (mostly sensor-based) technologies come at steep prices, and they are not available in vehicles that are already on the road. Therefore, there is an urgent need to devise some low-cost yet effective solutions to assist drivers in various driving conditions.

For Advanced Driver Assistance Systems (ADAS) technology, lane detection and tracking are some of the major challenges to be addressed. Challenging road conditions such as shadows, heavy traffics and fading lane markings may cause the lane detection algorithm to misrecognize other objects as lane marking. Addressing this issue involves many computer vision techniques and concepts [1, 7, 11, 13, 18, 19].

A natural application of lane detection and marking algorithms is lane departure warning system, which emerges as a promising tool to combat the aforementioned conditions. Lane departure warning method in [2, 3, 6, 8,9,10, 21] achieved favourable results in ideal highway conditions. In addition, in papers [3, 5, 14], the author justified that these algorithms are capable of performing under slight shadows and uneven illumination. However, most of these methods failed when tested against challenging road conditions, such as heavy traffic, faded lane markings, other road sign markings, etc.

Therefore, in this work, we propose a simple, fast, robust and effective lane detection method to address the challenging road condition under various illuminations. The main focus of this paper is to provide an efficient and cost-effective solution to a lane departure warning problem, such that it can be widely deployed. This paper combines several computer vision techniques, such as Adaptive Canny Edge Detection, Lane Detection, Gabor Filter, Hough Transformation and Multithreading for real-time implementation. Our method is formulated based on Euclidean distance and departure angle to determine the position of the vehicle relative to the lane boundaries. Specifically, since lane marking patterns exhibit directionality, it can be extracted easily from road surface regardless of shadows interfering the lighting conditions. Our method analyses frequency content in the specific direction localized within the region of interest. As a proof of concept, the whole system is integrated and tested on a vehicle for real-time detection.

Our paper makes the following contributions: (a) we introduce a dataset of Malaysia roads with different settings, including highway and urban roads under sunny and grim weather conditions, to test the performance of our proposed method; (b) we propose a lane candidate extraction method to perform lane detection using Gabor filtering; (c) we integrated lane departure algorithm with our proposed lane detection method to establish the Lane Departure Warning System (LDWS), and we tested the real-time performance of the system by installing it onto our test vehicle.

The rest of the paper is organized as follows. We present related works on lane detection method and lane departure warning system in Section 2, followed by Section 3 which details the proposed method of lane candidate extraction and lane departure algorithm. Experimental results are presented and discussed in Section 4. Finally, Section 5 concludes this paper.

2 Related work

This section elaborates related works in the area of lane detection and lane departure warning.

Lane detection

Author [3] proposed inverse perspective method to obtain the bird eye view of the road images and applied adaptive thresholding method to extract road lanes. The lane extracted is filtered with RANSAC to remove outlier and fitted with cubic spline method. Several IPM methods, including [3, 6], have similar structure as mentioned above. However, adaptive thresholding mapping is vulnerable to highly varying illumination condition and scattered shadows. As a result, inaccurate threshold may lead to inaccurate road lane markings. On the other hand, author [12] employed Canny edge method to identify the edge of road boundaries, where edges of the road boundary which is higher than the hysteresis threshold value are selected and connected with a straight line, which is computed by a voting mechanism of Hough transformation. The limitation of this method is its vulnerability to inconsistent lane marking, irregular shadows, and varying weather conditions.

RGB Histogram Filterization and boundary classifier are proposed in [10]. The RGB Histogram converts the image frame from image coordinate space to RGB histogram space, where the histogram data is utilized in classifying the boundary of the road by using two minimum points beside the peak. Furthermore, the detected boundary is fitted using cubic spline equation to model the lane boundary. This method assumes that the road lane marker is constant and has high visibility as well as contrast as compared to the road surface. Hence, it becomes the main limitation of this method. In addition, generating histogram data may result in high computational cost. Meanwhile, author [5] suggested Line Center Candidate method to capture the road lane marking, which has higher distance value when compared to the surrounding. They use line differential accumulation to search and crop the correct lane candidate. Incorrect recognition occurs as road pavements and shadows are misrecognized as road lane markings. Author [14] compared lane edge to check whether they belong to the same lane. Edge grouping is performed to group LEFT or RIGHT lane edges to their corresponding mask. Lane edges which overlap with the mask are considered as lane bound. Furthermore, Regional adaptive thresholding method is proposed to compute threshold for localizing regions of image based on light intensity. The proposed method excels in handling curvy roads and straight roads, as well as road with varying illumination and light shadows.

Based on these related works, only Inverse Perspective Method (IPM) provides clear guidelines to reproduce their work, hence allowing a fair comparison on our test dataset. The proposed method is compared with the conventional inverse perspective methods including [3, 6], which generally consist of the following steps:

  1. 1

    Finding homograph and perspective transformation: Compute 3 × 3 matrix to transform the image to its bird’s eye-view.

  2. 2

    Gaussian blurring and adaptive thresholding: smoothen and transform RGB image to binary image using threshold value determined by the environmental condition.

  3. 3

    RANSAC line fitting: Compute a 3rd order best fit line to model the road lane curvature detected in the image.

The Conventional Inverse Perspective Method is simulated by using the general framework described above. The simulated method converts the original road lane image as shown in Fig. 1(a) to a bird eye view as shown in Fig. 1(b). The results of applying adaptive thresholding and RANSAC line fitting are shown in Fig. 1(c) and Fig. 1(d), respectively.

Fig. 1
figure 1

Result of simulated Inverse Perspective Method; (a) Original image (b) Homograph transformation (c) Adaptive thresholding (d) RANSAC line fitting

Full size image

Lane departure warning

Author [9] proposed a method to correct uneven intensity interference with the pixel with fuzzification. Lane Departure warning is issued when 3 consecutive frames of departure is detected. This method is able to achieve 14–20 fps on dual-core 1.8GHz smart phone platform. The method is compact and effective to ease the execution on mobile devices. However, challenging road condition such as shadows and high traffic are not considered in their method. On the other hand, author [2] combined piecewise linear stretching function and Euclidean distance for their proposed lane departure warning method. Their proposed method is tested on rainy, sunny and night conditions and is found to be highly dependent on visibility of road lane marking. In [17], the author proposed a lane departure offset estimation method, which issues a warning when distance between the edge of a vehicle and lane is smaller than a threshold. The experiment is conducted on highway and urban road during daytime. On the other hand, author [4] proposed a lane departure warning system which utilizes spatial and temporal mechanism for lane departure application. Warning is triggered when host vehicle approaches the lane boundaries too fast. This technique is sensitive to the surrounding vehicles when the distance between front and host vehicle is too near. However, this method suffers from false positive as the host vehicle pass through curved roads.

3 Methodology

Figure 2 illustrates the overall flow of processes of our proposed framework. Our proposed framework consists of two main components, which are lane detection and lane departure warning system. Each component is elaborated further in the subsections below.

Fig. 2
figure 2

Overall flow of processes in the proposed framework

Full size image

3.1 Lane detection

The flow of processes of the proposed lane detection method is shown in Fig. 3. The following sub-sections explain each process in detail.

Fig. 3
figure 3

Flow of processes of our lane detection method

Full size image

3.1.1 Texture analysis

Image segmentation is performed to classify the road lane into two different patterns, namely, left lane pattern and right lane pattern as shown in Fig. 4. Due to its analysis capability, Gabor filter is applied to distinguish the road lane marking on the road surface from the presence of road bumps, vehicles, and other road signs. Since large Gabor kernel consists of numerous target pixel which is computationally and time expensive, the kernel size of 20 × 50 is used in our case, as shown in Fig. 5. Gabor filtering method is hypothesized to be sufficiently robust to handle challenging road conditions.

Fig. 4
figure 4

Left lane pattern (left) and Right lane pattern (right)

Full size image
Fig. 5
figure 5

Gabor kernel of size 20 × 50 with rotational angle of 45o (left) and rotational angle of 135o (right)

Full size image

Specifically, the camera sensors may not be able to capture fading road lane markings and shadowy conditions. Gabor filter provides texture enhancement and increases the edge response of the road lane marking, as shown in Fig. 6(a) and Fig. 6(b). Furthermore, noises such as vehicles, road signs and car lighting are suppressed due to dissimilarity of patterns from the Gabor kernel, as shown in Fig. 6(c) and Fig. 6(d).

Fig. 6
figure 6

Results of applying Gabor Filter. (a) Enhance fading road lane edges; (b) Enhance lane markings under dark shadow; (c) Suppress vehicle and road bump object; (d) Suppress non-related road sign on road surface

Full size image

3.1.2 Adaptive edge detection

Canny edge detector is proven to be a reliable technique to extract edges, which include road lane marking. It classifies edges and non-edges based on 2 parameters, namely, the upper threshold value (U) and the lower threshold value (L). Pixels with gradient values above U are selected as edges, while pixels with gradient values lower than L are considered as non-edges. Pixel with gradient value in the range of [L, U] is considered as part of an edge if and only if it is connected to another edge pixel, otherwise it is considered as a non-edge.

To address the potential changes in illumination, we introduce an adaptive edge detection technique, which can adjust the threshold values adaptively depending on the illumination condition. In order to achieve this, the result of Gabor filter is further divided into 3 sub-regions, as illustrated in Fig. 7.

Fig. 7
figure 7

Sub-regions for adaptive edge detection

Full size image

The mean value of each region is computed to model the illumination condition of that region of interest. This mean value is a criterion for defining the upper threshold value U of the Canny edge method, which is computed by using Eq. (1) below:

$$ {th}_{upper}(k)={Upper}_{lim}-\left(1-\frac{M(k)}{\alpha}\right)\ast {Upper}_{lim},\kern1em k=1,2,3 $$
(1)

where thupper denotes the upper threshold value of Canny edge detector, M denotes the mean value of each sub-regions, α denotes an arbitrary constant (i.e., configuration dependence) and Upperlim is the maximum upper threshold value, which is empirically set at 220. Low mean value indicates the possibility of low illumination in the sub-region; hence the upper threshold value of Canny edge detector is reduced to allow less prominent edges to be selected. Canny edge detector computes the road lane edges in sub-regions individually and the same process is performed for all frames in the video.

3.1.3 Lane candidate selection

After the edge pixels are computed, two best lane candidates are selected to represent the left and right lanes of the image. Probabilistic Hough Transform is applied to estimate the lane candidates by converting the lane recognition process to a simple peak point detection process. Edge pixels of the image are transformed to Hough Space coordinate, where polling for the best lane candidate occurs. Any coordinate having a greater poll than the defined threshold is chosen as a valid lane candidate. The left and right lanes are determined by applying gradient analysis in Eq. (2) to the valid lane candidates (u1, v1) and (u2, v2):

$$ \left\{\begin{array}{c}\frac{-{v}_2+{v}_1}{u_2-{u}_1}>0, Left\ Lane\ candidate;\\ {}\frac{-{v}_2+{v}_1}{u_2-{u}_1}<0, Right\ Lane\ candidate.\end{array}\right. $$
(2)

In order to simplify the lane candidates to a single lane candidate for the left lane (the same applies for right lane), we compute the weighted average of the lane candidates according to the length of the lanes using Eq. (3).

$$ {\displaystyle \begin{array}{c} Wn=\sqrt{{\left({V}_{2n}-{V}_{1n}\right)}^2+{\left({U}_{2n}-{U}_{1n}\right)}^2}\\ {}U{(k)}_{final}=\frac{\sum \limits_1^n Wn{U}_{kn}}{\sum \limits_1^n Wn},\kern0.5em k=1,2\\ {}V{(k)}_{final}=\frac{\sum \limits_1^n Wn{V}_{kn}}{\sum \limits_1^n Wn},\kern0.5em k=1,2\end{array}} $$
(3)

where n denotes the number of lane candidates, Wn denotes the weight of the lane candidate, (U1n, V1n) and (U2n, V2n) denote the edge value of the lane candidate and (Ufinal, Vfinal) denotes the edge value of the lane after weighted average is computed. The result of this line weighting selection is illustrated in Fig. 8.

Fig. 8
figure 8

Result of Hough transform prior to line weighting selection (left) and after line weighting selection (right)

Full size image

3.2 Lane departure warning system

In this section, the lane departure warning system is put forward as a layer of process on top of the lane detection method proposed in Section 3.1. Specifically, the lane candidates generated in the previous section are essential information in obtaining position of the vehicle with respect to both left and right lanes. The overall flow of processes of our lane departure warning system is shown in Fig. 9.

Fig. 9
figure 9

Flow of processes of lane departure warning system

Full size image

Based on the result of lane detection, we computed the position of the vehicle which is represented by Euclidean distance (λ) and lane departure angle (θ) using Eq. (4):

$$ \left\{\begin{array}{c}{\lambda}_x=\sqrt{{\left\Vert O\right\Vert}^2+{\left\Vert {P}_x\right\Vert}^2-2\left\Vert O{P}_x\right\Vert }\ \\ {}{\theta}_x={\tan}^{-1}\frac{-\mathrm{V}{2}_x+\mathrm{V}{1}_x}{\mathrm{U}{2}_x-\mathrm{U}{1}_x}\end{array}\right. $$
(4)

where O refers to image coordinate of the vehicle origin, P represents image coordinate of the lane candidate and x denotes the left or right lane candidate. Note that the position of a point in a Euclidean space can be treated as a Euclidean vector, hence, O and P are Euclidean vectors, starting from the origin, where their tips indicate two points (i.e., \( \overrightarrow{O}\ and\ \overrightarrow{P} \)). The vector coordinates are illustrated in Fig. 10. The position of the vehicle is estimated via the computation of Euclidean distance between the vehicle origin’s O and the edge of the left and right lane candidate, i.e., (Pl, Pr).

Fig. 10
figure 10

Vector coordinates on the road image

Full size image

On the other hand, by implementing basic trigonometry formulae, lane departure angles of the lane candidate are computed from the image coordinates of the detected edge. These parameters are empirically found to be linear to the change of position of the vehicle, which can be utilized to estimate potential lane departure. In case of lane departure, the departure side of the lane exhibits a larger angle θ as compared to the non-departure side. By integrating both parameters, a robust and more accurate waning system can be presented to the driver.

We developed a notification control system, which consists of three-step notification to inform the driver about his/her current position relative to the road lanes. First, the system initializes a ‘Safe’ notification to the driver. Second, when driver negligence occurs, the vehicle may swerve across the lane abruptly and trigger the ‘left departure’ or ‘right departure’ notification of the proposed system. Third, if corrective action is made such that the vehicle stays at the correct lane position, the ‘Safe’ notification is issued to the driver. The alarm rate of the system is minimized to prevent the user from getting aggravated by the relentless warning from the system. Table 1 shows the condition in which a notification is issued to the driver. Here ξ denotes the threshold value for Euclidean distance and ϕ denotes the threshold value for lane departure angle. ξ and ϕ affects the time it takes for the driver to react and make corrective action before lane switching occurs. Varying these thresholds excessively results in inadequate response time for the driver to perform corrective action or causing the system to be overly sensitive.

Table 1 Conditions for different notification status
Full size table

In addition, to ensure robust and accurate results, the notification control system checks the availability of the left and right lanes on the road. In some occasions, one of the road lanes is not detectable due to missing road lane marking or occlusion. To address this potential scenario, the proposed algorithm is designed to make judgment based on the only lane candidate detected.

4 Experiments and results

4.1 Prior configurations

In this section, we detail the hardware, software and database setups in our experiment.

4.1.1 Hardware configurations

We utilized Raspberry pi 3B as the main processing unit as it allows us to design a cost-effective solution while achieving real-time performance using multithreading compared to other types of microprocessors. We have also integrated raspberry pi camera v2 as the image acquisition media and Raspberry pi 7-in. display as the display media, which are installed at the rear-view mirror of the vehicle, as shown in Fig. 11. The test vehicle used in our experiment is a 2008 version Honda Accord 2.0 L.

Fig. 11
figure 11

Hardware configurations

Full size image

4.1.2 Software configurations

For the software configurations, we used OpenCV 3.1.0 and Python 2.7 to perform image processing algorithms and mathematical computations. Furthermore, the main code is multithreaded into 3 threads to execute independently from each other. The threads are as follows: first thread deals with the operation of raw image extraction (input). Second thread executes all the image processing algorithms to perform lane detection and lane departure warning system. The last thread is responsible for displaying the output of lane detection and notification status to the driver via Raspberry pi 7-in. display (output). The main purpose of multithreading is to reduce the delay latency of the main code [12].

To evaluate the proposed lane detection and lane departure warning system, the image dataset of various local road conditions is required. Sunny and grim weather conditions are considered in our dataset as two major illumination conditions. Furthermore, highway and urban roads are the two major types which drivers encounter during daily driving routine. Both types of road have dissimilarities in terms of road curvature, lane marking feature, and traffic capacity. These driving scenes are included in our database as recorded in Table 2.

Table 2 Summary of dataset
Full size table

Some representative examples of our dataset are shown in Fig. 12.

Fig. 12
figure 12

Representative examples of our dataset under different road and weather conditions

Full size image

4.1.3 Dataset setup

4.2 Ground truth and result verification

In order to obtain an accurate quantitative assessment of the proposed method, the result verification process in [9] is adopted. A total number of 948 test images (see Table 2 for detailed information) of various challenges are taken from our dataset and hand labelled as the ground truth using LabelMe online software. A representative example is shown in Fig. 13(b) and the lane candidates identified by our proposed method are shown in Fig. 13(a). The lane candidates detected by the proposed method are compared to the ground truth lines (see Fig. 13(c)) to determine the accuracy of proposed method.

Fig. 13
figure 13

A representative result for urban road: (a) Lane candidate generated from our proposed method, (b) Hand labelled ground truth lines, (c) Comparison between generated lane candidate and ground truth

Full size image

To determine if both ground truth and the lane candidates identified by the proposed method are the same, two points from both lanes are selected and the minimum distance between both points are computed as follows:

$$ \left\{\begin{array}{c}\ \min \left( pt{1}_o, pt{1}_{gt}\right)<{t}_1\ \\ {}\ \min \left( pt{2}_o, pt{2}_{gt}\right)<{t}_2\end{array}\right. $$
(5)

where pt1o and pt2o are two points from the lane candidate, while pt1gt and pt2gt are two points from the ground truth line. Here, t1 and t2 are the threshold values. We apply strict thresholds to ensure that lane candidate which is highly dissimilar from the ground truth is removed. Furthermore, the distance between the detected lane candidate and ground truth line must be less than the width of lane marking to qualify the lane candidate as accurate detection [17]. The results are classified into 3 categories: completely accurate, partially accurate, and not accurate. Completely.

accurate indicates that the detected lane candidates are almost similar to the hand labelled ground truth. Meanwhile, partially accurate indicates that either left or right lane candidate is correct (but not both). Meanwhile, not accurate indicates that neither left nor right candidate is correct. We give each category a score of 1.0, 0.5 and 0 for completely accurate, partially accurate and not accurate, respectively, with the respective semantics detailed below:

$$ \left\{\begin{array}{c}\ 1.0\kern1em ,\kern1em if\ both\ left\ and\ right\ lane\ candidate s\ are\ accurate;\\ {}0.5\kern1em ,\kern0.75em if\ either\ of\ the\ lane\ candidate\ is\ accurate;\kern4.75em \\ {}\ 0\kern1.75em ,\kern1em if\ both\ lane\ candidates\ are\ inaccurate.\kern6.75em \end{array}\right. $$
(6)

Both the proposed method and conventional Inverse Perspective Method are tested on the test dataset. To obtain high quality results, the percentage of correct rate only considers the frames which are completely accurate.

4.3 Lane detection result

We compare our proposed lane detection method with the conventional IPM and GVP method [21] on the test dataset as shown in Fig. 14. It can be observed that our proposed method outperforms both the conventional IPM and GVP method on almost all conditions. Urban – Sunny category seems to be the most challenging road conditions among all the test datasets, which is reflected by the number of errors made by all the methods. However, our proposed method significantly outperforms the conventional IPM and GVP method, as our proposed method achieved correct rate of 92.67% as compared to 80.67% and 87.67% for conventional IPM and GVP method, respectively. This poor performance of the conventional IPM is caused by its inability to detect lane marking in dark shadows and uneven illumination conditions. The adaptive thresholding phase of conventional IPM selects inaccurate threshold value to differentiate the lane marking from the road surface. Meanwhile, GVP method could not establish an accurate vanishing point under the challenging conditions, which causes inaccuracies in detecting proper lane candidates. On the other hand, the proposed method uses edge detection method to generate lane boundary correctly. Furthermore, conventional IPM is unable to handle the presence of rumble strips on the road surface, which appears frequently in this test dataset. Our proposed method improves the overall performance by utilizing Gabor filter to enhance lane marking feature under shadows and eliminates non-related road markings. Compared to GVP method, our method is more robust in handling fading lane markings and high traffic situation as our method is designed to make judgment based on the only lane candidate detected.

Fig. 14
figure 14

Lane detection results of our proposed method, IPM and GVP method [21] on the test dataset

Full size image

Overall, the proposed method outperforms the considered benchmark methods by achieving a correct rate of 93.67%. On the other hand, conventional IPM and GVP method achieve a correct rate of 86.60% and 91.21%, respectively (as shown in Table 3). In real-life scenario, most roads are not ideal which is represented by our dataset. Specifically, in a developing country such as Malaysia, the highway and urban road are often plagued with fading lane marking and uneven illumination. It is important to consider these non-ideal conditions when designing a lane departure warning system. The proposed method is designed and developed to handle this kind of situations; hence it performs generally better in the experiments.

Table 3 Performance summary of our proposed method, IPM and GVP method [21] on the test dataset
Full size table

4.4 Lane departure warning result

To test the performance of our lane detection method on the lane departure warning system, we extracted a new set of frames from the test dataset to see how it performs under random scenarios. We only use two datasets: Highway – Grim, and Urban – Sunny dataset because Highway – Grim represents the best-case scenario where both proposed method and Conventional IPM achieves the highest lane detection accuracies. Similarly, Urban – Sunny represents the worst-case scenario where both the proposed method and conventional IPM yield the lowest lane detection accuracy. In this new set of frames, we extracted 406 and 537 frames for Highway – Grim and Urban – Sunny, respectively.

The lane departure warning system issued visual warning to the driver during unintended lane departure scenario. Fig. 15 shows that the system is able to successfully detect the host vehicle that deviates from the original lane boundary in the Highway - Grim and the Urban - Sunny datasets.

Fig. 15
figure 15

Lane Departure warning status issued for both dataset (a) Highway - Grim dataset (b) Urban – Sunny dataset

Full size image

Specifically, in the Highway - Grim dataset, the proposed method shows an overall accuracy of 95.32% as recorded in Table 4, while having false positive and negative warnings under 2.5%, which are relatively low. The vehicle crossed the lane boundary for 3 times in this dataset and the proposed method issued warning successfully for all 3 occurrences. There are scenarios where the host vehicle deviated to the lane boundary without crossing it during curving road departure. This causes the system to issue 11 false alerts due to the sudden approach to change of lane boundary. This problem is solved by increasing the threshold value to allow higher tolerance to the conditions.

Table 4 Results of lane departure warning system on the Highway - Grim and the Urban - Sunny datasets
Full size table

On the other hand, for the Urban - Sunny dataset, an accuracy of 95.16% achieved by the proposed method as recorded in Table 4. The proposed method accurately generates 91.42% and 96.58% of the warning status when the vehicle deviated to the left and right lanes when it was not supposed to. Three lane departure attempts from the host vehicle are successfully warned by the lane departure warning system. High amount of shadows in the dataset slightly affect the stability of the lane candidate detected, which results in 10 and 16 frames of false positive and false negative, respectively. Here, the lane departure warning system triggered 16 false alarms when the vehicle maintained within the lanes. This is caused by misrecognizing curbs and pavement as road lane markings, thus triggering false alarm. However, this is not a huge problem as curb and pavement are originally the boundary of the roads.

4.5 Computational complexity and real-time detection result

It is noteworthy that computational complexity of an algorithm directly affects the real-time performance of the system. A lower computational complexity would result in a better real-time performance, which is a crucial aspect in an ADAS technology. Therefore, we investigate into the computational complexity of lane departure warning solutions from IPM, GVP method, and our proposed method in terms of the big O notation. Conventional IPM, GVP method and our proposed method have a computational complexity of O(n2). Since the conventional IPM and GVP method have a similar complexity as our proposed method, the main comparison should be focusing on the accuracy. Specifically, the proposed method outperforms both methods by a huge margin as reported in Table 3.

It is undoubtedly true that neural network-based solutions have equivalent or better accuracies compared to our proposed method and other conventional methods. However, it is also important to examine the real-time capability of such methods. These methods are known to be computationally heavy as they have a computational complexity of O(n4) during the inferencing phase. This indicates the need for a stronger and significantly more expensive hardware /system for processing and inferencing. These models are usually trained and evaluated using a high-end GPU (i.e., NVIDIA GTX 1080Ti, NVIDIA Tesla K80, etc.) for faster training time [15, 20]. However, implementing these models in the edge at the later stages using an embedded device such as NVIDIA Jetson Nano or Jetson TX2 would yield to an increase in the execution time, as the edge device has a lower computing capability (e.g., 472 GFLOPs – 1.33 TFLOPs) compared to the high-end GPU (e.g., >8.7 TFLOPs). This is further supported by performance comparison of lane detection methods by Wang et al. [16], where the author compares the performance of CNN and CNN-LSTM on NVIDIA Jetson Nano. In which, the execution speed of CNN and CNN-LSTM methods are 74.1 ms and 89.3 ms, respectively. This execution speed is slower in comparison to ours, which only requires 55 ms. Our proposed method achieves the same functionality for lane detection with faster execution speed, simpler hardware configuration and lower cost. Although the considered test datasets are different, the above comparison provides a general idea on the execution speed by using a more powerful hardware in comparison to raspberry pi.

Our lane detection and lane departure method proposed previously are implemented on Raspberry pi, which is an edge-based system. After implementation of multithreading, our method achieves an average frame speed of 20 frames per second, which produces smooth transition of images. The processing time for one frame ranges from 45 ms to 55 ms. Here, Raspberry pi utilizes all four cores and 60 ~ 85% of total processing power is required to complete all tasks. This suggests that the proposed lane detection and lane departure algorithms are compact and effective solutions to be implemented in consumer electronic processor. Figure 16 illustrates the performance of our method when executing the lane detection and lane departure algorithms in real-time.

Fig. 16
figure 16

Some examples of real-time implementation and deployment

Full size image

5 Conclusion

In this work, an edge-based lane detection method for lane departure warning system is proposed. The proposed lane detection method exhibits robust performance in detecting and tracking lane boundaries, especially in challenging situations. The integration with Gabor filtering method enhances the lane marking feature on the road surface while filtering out irregular shadows and non-related patterns. Experiments show that our proposed lane detection method achieves an average accuracy of 93.67% and outperforms the simulated conventional IPM and GVP method. For Lane Departure Warning System, we managed to achieve an average accuracy of 95.24% with relatively low false positives and false negatives (below 3%). We have also tested the real-time performance of our system and it is able to achieve an average of 20 fps at 75% CPU utilization with the aid of multithreading. These experimental results indicate that our proposed method serves as a cost-effective solution to the lane departure warning problem.

In future work, we want to further improve the accuracy of the lane detection method. We also want to test our system in different environmental setting, such as rain, night, as well as raining night.