Introduction

A point cloud, which is the coordinate dataset of many points in space, represents an atomized description of the real world. Airborne light detection and ranging (LiDAR) is one of the most important technical means to obtain three-dimensional (3D) point clouds of ground surfaces. The UAV-LiDAR systems, which are based on various light and small unmanned aerial vehicles (sUAVs), have emerged with the rapid development of technologies such as UAVs and micro-laser scanners high-precision positioning and orientation systems (Jaakkola et al. 2013; Lin et al. 2011). To date, UAV-LiDAR technologies have been rapidly and widely applied in various fields, including topographic mapping, soil erosion, glacial changes, remote sensing of snow cover, forestry investigations, and biodiversity (Cavalli et al. 2013; Chen et al. 2021; Harpold et al. 2014, 2015; Kolarik et al. 2019; McClelland et al. 2019; Neugirg et al. 2016; Shellberg et al. 2016; Tarolli 2014).

In recent years, another new method to obtain point clouds is sUAV-based structure-from-motion (SFM) photogrammetry (Park and Lee 2019), integrating computer vision algorithms into the photogrammetry process. Not strictly depending on an image’s positioning and orientation data, the sUAV-SFM can automatically extract and match the features of overlapping images acquired from different perspectives and shooting distances. The feature data are then combined with multiview stereo (MVS) geometry to generate dense point clouds. Factors that affect the accuracy of an SFM point cloud include the camera, lens, topographic features, methods for the control point and image data acquisition, lighting conditions, and data processing software (Carrivick et al. 2016). A way to assess the accuracy of SFM data is to perform direct comparisons with the data obtained through other measurement methods, such as total stations, differential GPS, ground and airborne LiDAR (Stumpf et al. 2015). Another strategy is to compare the differences in geometric parameters between SFM and other point clouds. For instance, the application in surveys on forestry resources showed that the accuracies of canopy parameters extraction using LiDAR and SFM photogrammetry were very similar (Goodbody et al. 2017; Thiel and Schmullius 2016; Wallace et al. 2016; White et al. 2015).

As a primary data product, a point cloud is generally not directly used but requires further processing to derive digital elevation models (DEMs), digital line graphs (DLGs), or mesh models. A point cloud containing color components can also be used to produce digital orthophoto maps (DOMs). Overall, the quality of a point cloud determines the accuracy of its subsequent data products. Therefore, the accuracies of point clouds play critical roles in both LiDAR and SFM photogrammetry.

Many methods have been devised to assess point clouds’ accuracy; in general, a statistical analysis of the dissimilarities between point clouds can be performed by deriving their respective grid data before calculating the difference in their elevations, Z (Salach et al. 2018; Tonkin et al. 2014). However, the detailed topographic information contained in the point clouds disappears after rasterization. Instead of calculating the difference of the DEM (DoD) based on the derived DEMs, directly comparing the point clouds is a more reasonable approach (Akca et al. 2010; Erol et al. 2020; Salach et al. 2018; Tonkin et al. 2014).

The literature mentioned above indicates that LiDAR and SFM have been widely used in topographic measurements, and the comparison between the two kinds of data collection methods has often been carried out. However, during the digital surface model (DSM) data generation based on SFM, a default point cloud filtering algorithm was used to handle the point cloud, which may be inconsistent with the ground point filtering algorithm used in the DSM data production through LiDAR. This difference between different filtering methods may lead to inconsistent DSM accuracy. Especially at surface discontinuities, surface modeling errors and the reference frames of the two DSMs inconsistent may lead to large differences. Though these shortcomings can be overcome by employing the approach where the shortest 3D (Euclidean) distance between each reference point and the produced DSM is used (Akca et al. 2019; Stylianidis et al. 2020), research on the process of the obtained SFM and LiDAR point clouds based on the same point cloud filtering method and comparison between the processed SFM and LiDAR point clouds and the corresponding generated DSMs could help investigate the advantages, limitations, and accuracies of sUAV-SFM and UAV-LiDAR point clouds, and obtain a clearer understanding of their dissimilarities.

This study aims is to compare the SFM and the LiDAR point clouds processed based on the same point cloud filtering algorithm. In order to understand the difference between the SFM and the LiDAR point clouds processed by the same CSF filtering algorithm, we analyzed the accuracies of the point clouds based on the direct and indirect assessment methods in some typical application scenarios. The study is intended to explore the advantages, limitations, and dissimilarities between the point clouds generated by sUAV-SFM and UAV-LiDAR based on the same filtering algorithm, and verify the feasibility of terrain measurement using light and small unmanned vehicles, and try to provide a more economical, convenient, and efficient high-fidelity terrain data acquisition method.

Materials and methodology

General framework

A research route was designed to analyze the differences between the point clouds acquired by sUAV-SFM and UAV-LiDAR, as shown in Fig. 1.

Fig. 1
figure 1

General methodology of this study

Full size image

First, Global Positioning System-Real-Time Kinematic (GPS-RTK) was used to measure the image control points and checkpoints to obtain the corresponding ground reference data. Next, consumer-grade sUAVs and UAV-LiDARs were separately used for data acquisition, after which the acquired data were immediately processed to obtain the corresponding point cloud data. Then, the DTMs derived from sUAV-SFM and UAV-LiDAR point clouds are obtained, respectively. Finally, profile analysis and the calculated earthwork volumes were used to evaluate the differences between the two types of point clouds. We note that in this study, multiple UAVs were used to obtain data on the same day. Furthermore, the weather conditions were good, rendering the results of the comparative analysis highly reliable.

Data acquisition and processing

UAV platforms

Different types of equipment were used in this study to obtain point cloud data to compare and analyze the differences between the data acquired via sUAV-SFM and UAV-LiDAR. The DJI Phantom 4 Pro UAV platform (Fig. 2(a)) was used to acquire sUAV-SFM point cloud data. For UAV-LiDAR point cloud data acquisition, the DJI Matrice 600Pro six-rotor UAV was used as the airborne platform (Fig. 2(b)), which is equipped with the Geosun gAirHawk GS-100 UAV-LiDAR System (http://english.geosun-gnss.com.cn/index.php).

Fig. 2
figure 2

UAV platforms of SFM and LiDAR: (a) The DJI Phantom 4 Pro, and (b) the DJI Matrice 600 pro with a built-in GNSS/IMU equipped with a Geosun gAirHawk GS-100 UAV-LiDAR System

Full size image

The UAV-LiDAR system is mainly composed of a 16-channel laser scanner and a high-frequency positioning and orientation system. The UAV-LiDAR system can achieve high-precision scanning of targets within a 100-m range, with an absolute accuracy better than 10 cm for point cloud measurements. A simple comparison of the two sets of data acquisition equipment for generating point clouds was performed from several aspects, including the weight, price, operability, and convenience (Table 1). Notably, substantial differences exist between the two sets of equipment.

Table 1 Data acquisition equipment comparisons between sUAV-SFM and UAV-LiDAR
Full size table

GPS-RTK measurement of image control points and checkpoints

First, the horizontal datum of the survey area was set to China Geodetic Coordinate System 2000 (CGCS2000) Cartesian coordinate system, with Gauss–Krüger projection, 3° zoning, a central meridian at 114°, and an ESPG ID of 4547. The vertical reference used in this study was the 1985 national height datum (Yellow Sea 1985) for orthometric height with a mean accuracy better than 0.1 m (Li et al. 2017). As there were no notable ground features on the site, the surfaces of roads and stones were spray-painted with crosshairs with side lengths not shorter than 30 cm. The exact locations of their center points were measured directly after UAV image acquisition using a UniStrong G970II All-in-one RTK receiver (http://en.unistrong.com/ProductShow.asp?ArticleID=299) which provides centimeter-accurate positioning and a height accuracy with 8–15 mm. A total of seven crosshairs were used as ground control points. Subsequently, another 45 checkpoints were measured using rover mode. The aforementioned points were distributed at approximately equal intervals throughout the survey area. One of them is shown in Fig. 3.

Fig. 3
figure 3

Picture of a GCP

Full size image

Point cloud acquisition and generation

The primary process of acquiring sUAV-SFM point cloud data includes selecting and measuring image control points and checkpoints, UAV image shooting, image screening, and data import into the automatic calculation software for point cloud acquisition. Previous studies have extensively discussed the operation principles of SFM and MVS, as well as the actual procedures for data acquisition and processing (Lowe 2004; Snavely et al. 2008; Triggs et al. 2000; Ullman 1979). The acquisition of sUAV-SFM data was carried out in mid-August 2019 under good weather conditions. First, parallel flights lines were programmed to have an image sidelap of 80% and fore-and-aft overlap of 60% while considering the Phantom 4 Pro camera sensor size (13.2 × 8.8 mm) and focal length (8.8 mm). The camera was set to shutter-priority mode and used a 1/200 s shutter speed with ISO fixed at 100. The sUAV captured a total of 260 images of the study area, each with a resolution of 5472 pixels × 3648 pixels. The image collection process required 30 min. The process of the UAV-acquired images was done using Pix4Dmapper desktop software (Pix4D SA, Switzerland, https://www.pix4d.com/). The Pix4D software is a highly automated software which includes a powerful camera auto-calibration algorithm that takes the full information of each pixel of the acquired images to estimate the optimal camera and lens calibration for each flight. This feature is essential to guarantee the perfect accuracy under any climatic conditions without any manual and tedious user intervention. First, in the process of initiation photo camera focal length, principal point location and radial, tangential distortions were calculated (Visockiene et al. 2014). Then, a series of processes were carried out to handle the image alignment and produces a sparse point cloud. Next, the 7 GCPs were imported and matched (georeferenced) using the rayCloud Editor menu available within Pix4D. After the georeferenced, the dense point cloud would be generated. During the first two steps, different image resolutions may be used (1, 1/2, 1/4, 1/8). To get a better quality of point cloud, we used the full resolution of all images in the image alignment and densification process in this study. Furthermore, the desired point density of the final point cloud was set as “optimal density,” minimum number of matches was set to 3. Besides, parameters such as “classifying point cloud” and “merging tiles into one file” were selected (“Yes”). Finally, the final point cloud data was automatically saved to *.las format for each input image, one of the most common formats for exchanging point clouds.

The UAV-LiDAR point cloud for the study area was acquired by the gAirHawk light LiDAR point cloud data acquisition system. As with the aforementioned image data collection method, the UAV flight paths were first established based on the scope of the survey area. Next, the UAV-LiDAR system automatically scanned the terrain according to the designed route.

The post-procession of GNSS/IMU data was conducted using Shutter software provided by Geosun, which adopts the world’s leading single epoch ambiguity algorithm and high-order Kalman filter to maximize the integration of the GNSS carrier phase and IMU information. Processing of UAV-LiDAR was completed using gAirhawk toolkits, which were provided with the Geosun gAirHawk GS-100 UAV-LiDAR System. gAirhawk LiDAR Data Process Software is a point cloud computing software self-developed by Geosun Navigation. It supports real-time configuration and monitoring of field data acquisition systems, decoding real-time and post-process laser scanning data, calculating and displaying point cloud data, and supporting software for Geosun LiDAR scanning system. The UAV-LiDAR point cloud was also successfully processed with no errors detected.

Point cloud data pre-processing

Before data acquisition and processing, it is necessary to determine whether the two data sets are different from each other. In order to avoid the absence of translational and rotational difference of the two point clouds, a unified survey control network was established in the study area at the very beginning, and the two measurement methods are applied based on the same geographic reference frame. This was corroborated with the checking of the control points in the study area.

During point cloud data acquisition, both noise and point cloud data from beyond the study area are inevitable, which necessitates pre-processing of the point clouds to ensure that the acquired point cloud data of the study area are at higher precision. In this study, the two types of point cloud data were imported into the CloudCompare software (CloudCompare 2019) to eliminate the outliers and error points before cropping into the same range. In order to verify the consistency of the two types of point clouds for various underlying surface conditions, three sub-regions were extracted from the study area based on the vegetation coverage and characteristics of the topographic undulations: flat sub-region, sub-region with bare soil, and excavated sub-region (Fig. 6).

The flat sub-region has a gentle slope, and its surface comprises low wild grasses, shrubs, isolated trees, and aqueduct facilities. Tall vegetation cover is absent in the excavated sub-region, this region comprises sparse wild grasses. The terrain is steep, with a slope of nearly 90°. The sub-region with bare soil is an earthwork stacking yard formed during the excavation process, which has almost no vegetation cover. After the boundary polygons of each sub-region were drawn using ArcGIS software, the two sets of point cloud data were cropped and stored in the LAS format for further analysis.

Ground point classification

The original point clouds must be classified to distinguish between ground and non-ground points to obtain accurate DTM data. In order to avoid the influence of different filtering parameters on the point cloud filtering results and ensure the consistency of the point cloud filtering process, we chose the same filtering algorithm. There are numerous algorithms available for point cloud classification, among which cloth simulation filter (CSF) is a ground point filtering algorithm widely used in recent studies (Zhang et al. 2016). This algorithm is similar to the cloth modeling process in computer graphics. First, outliers were automatically or manually handled. Second, the reversed point cloud is covered by a piece of rasterized and rigid virtual cloth. Under the effects exerted by gravity and adjacent nodes, the grid node particles are displaced and attached to the topographic surface. Then, the cloud points and the grid particles are projected to a horizontal plane. Next, the nearest cloud point for each particle is found and the difference in the elevation between the point cloud and virtual cloth is calculated point-by-point when the maximum height variation of all particles is small enough or when it exceeds the maximum iteration number which is specified by the user, if the different is equal to or less than the threshold, the point is classified as bare earth, otherwise it is classified as objects (Zhang et al. 2016). Finally, the ground and non-ground points are distinguished by the method. The CSF algorithm is simple, easy to use, and integrated into multiple software programs, including Octave, CloudCompare, MATLAB, and 3DF Zephyr. In this study, the ground point classification results were obtained through the CSF plugin in CloudCompare, an open-source software.

During the cloth simulation process, the particle would move a certain displacement under the effect of gravity and internal force. According to Newton’s second law:

$$\mathrm{m}\frac{\partial \mathrm{X}\left(\mathrm{t}\right)}{\partial {t}^{2}}= {F}_{ext}\left(X,t\right)+ {F}_{int}\left(X,t\right)$$
(1)

where, \(m\) is the mass of the particle which has a default unit 1, \(X(t)\) is the position of the particle at time \(t\), \({F}_{ext}\) and \({F}_{int}\) are the gravity and the spring force of the particle at the position \(X(t)\) respectively, \({F}_{int}\) also meets Hooke’s law.

The particle moves in the vertical direction under the influence of gravity. The displacement of the particle in the vertical direction in a certain period can be determined by Eq. (2) as followed.

$$\mathrm{X}\left(\mathrm{t}+\Delta \mathrm{t}\right)=2X\left(t\right)-X\left(t-dt\right)+g\Delta {t}^{2}$$
(2)

where \(\Delta \mathrm{t}\) is the duration of the particle movement, \(g\) is the acceleration of gravity.

In the model, the internal force is an elastic force of spring. When considering the particle’s movement under the internal force, the traversal calculation of all springs is necessary. If the particles at both ends of a spring are fixed, the particle does not move. When the particles at both ends of a spring are movable, the particle should be moved to the average elevation of the two endpoints of the spring. Besides, the particle should be moved by half of the elevation difference between the two end particles of the spring. The following equation can calculate the displacement of the particle.

$$\overrightarrow{\mathrm{d}}=\frac{1}{2}b\left( \overrightarrow{{p}_{i}}- \overrightarrow{{p}_{0}} \right)\bullet \overrightarrow{n}$$
(3)

where \(\overrightarrow{\mathrm{d}}\) is the displacement vector of the particle. \(b\) is 1 when the particle is movable; otherwise, \(b\) is 0. \(\overrightarrow{{p}_{i}}\) is the position of the moving particle, \(\overrightarrow{{p}_{0}}\) is the position of the adjacent particle. \(\overrightarrow{n}\) is the standard vector in the vertical direction, taking \(\overrightarrow{n}={\left(\mathrm{0,0},1\right)}^{T}\).

Accuracy assessment methods of point clouds

The accuracies of the point clouds were evaluated from two aspects to better understand the differences between the two types of point clouds in question. One method was to directly evaluate their respective accuracies using the M3C2 algorithm, while the other method was to evaluate the accuracy of the raster DTMs generated from the point clouds, thereby indirectly evaluating the accuracy of the latter. The second accuracy assessment method can be based on the analysis of either the DTM profiles or calculated earthwork volumes.

Point cloud accuracy assessment based on GNSS survey

After obtaining the point cloud data, it is necessary to conduct an initial check on the accuracy of the point cloud data so that subsequent experiments can be carried out correctly. The method to check the accuracy of the point cloud data is to compare and analyze the point cloud data with the GNSS-RTK measured checkpoints and use the analysis results as the basis of the point cloud accuracy evaluation. In the contrast analysis, the accuracy in the z-direction is often used to represent the accuracy of the point cloud. The accuracy of 3D point clouds was assessed by the mean absolute error (MAE) in the vertical direction (Dang et al. 2020; Martinez et al. 2021). It was calculated by summing the absolute value of actual Z coordinate values of the CPs surveyed (\({Z}_{k}^{cp}\)), subtracting the z values estimated from scanning points closest to CP coordinate positions in point cloud (\({Z}_{k}^{pt}\)), and then dividing by the total number of surveyed CPs (n). Ideally, the error metric should be zero, indicating that the point cloud achieved a reasonably good accuracy.

$$MAE=\frac{1}{n}{\sum }_{k=1}^{n}\left|{Z}_{k}^{cp}-{Z}_{k}^{pt}\right|$$
(4)

where \(MAE\) represents the average absolute error in the z-direction, \({Z}_{k}^{cp}\) is the elevation of the checkpoint \(k\) measured by GNSS, \({Z}_{k}^{pt}\) is the elevation of the point cloud to be evaluated, which is closest to the checkpoint \(k\), \(n\) is the total number of the checkpoints.

Point cloud accuracy assessment based on M3C2

The accuracy of the point clouds directly affects the digital topography, such that it is critical to first evaluate the accuracy of the acquired point clouds. Although point cloud data can express complex topographic structural units in 3-D space, the operation to determine the differences between point clouds is highly complicated. At present, SFM data processing mainly involves using differential GPS to measure a large number of checkpoints to evaluate data quality. However, there are certain limitations: the layout and the density of checkpoints are entirely based on the experience of field staff; besides, the topographical conditions of the site impose certain constraints (such as steep cliffs and dangerous locations). M3C2 is a multiscale comparison algorithm that directly acts on point clouds to assess their accuracy (Lague et al. 2013), providing reliable results (Stumpf et al. 2015).

The main computational procedure of the M3C2 algorithm includes three steps. In the first step, the core point \(i\) was selected from the referenced point cloud and the unit normal \(\overrightarrow{N}\) was calculated based on the defined normal scale \(D\). In the second step, the two subsets of the reference and compared point clouds were defined by the intersection of the reference and compared point clouds with a cylinder calculated by the unit normal N, over the core point \(i\), the diameter of projection scale \(d\), and the length of the projection \(L\). After that, the distribution of distances which are used to define the mean (or median) position of each sub-cloud \({i}_{1}\) and \({i}_{2}\) as shown in Fig. 4, would be given along the normal direction N when each sub-cloud was projected on the cylinder axis. Finally, the distance \({L}_{M3C2}(i)\) between the projection centers is calculated, which was used as the algebraic deformation from the reference point cloud to the compared point cloud at the core point \(i\). Besides, a confidence interval is introduced to estimate the distance measurement accuracy and to assess whether a statistically significant change is detected or not at the prescribed confidence level, which is set at 95% (Lague et al. 2013). The confidence level can be calculated by the following equation.

$$LO{D}_{95\%}\left(d\right)=\pm 1.96\left(\sqrt{\frac{{\sigma }_{1}{\left(d\right)}^{2}}{{n}_{1}}+\frac{{\sigma }_{2}{\left(d\right)}^{2}}{{n}_{2}}}+reg\right)$$
(5)

where \({\sigma }_{1}\left(d\right)\) and \({\sigma }_{2}\left(d\right)\) are the roughness figured up from the sub-cloud of the reference and the compared point clouds in the projection cylinder, respectively. \({n}_{1}\) and \({n}_{2}\) are the number of points in each corresponding sub-cloud. \(reg\) is the stitching error between the point clouds.

Fig. 4
figure 4

Illustration of calculating M3C2 distance on referenced point cloud and compared point cloud

Full size image

Accuracy assessment based on the DTM profile analysis

This method evaluates the accuracy of point clouds by comparing the accuracies of the generated DTMs. In order to obtain the DTMs, the tool embedded in the ArcGIS LAS dataset to raster was used to convert the point cloud into DTMs. The parameters including interpolation type, output data type, sampling type, and z factor may be specified in the rasterization process. In this study, the binning method of interpolation technique was selected to determine cell values of the output raster, the output data type was set to be Float, and the sampling type was set to be Cellsize to define the resolution of the output raster. The cell size was finally chosen 0.1 m. Z factor was set to default because the coordinate system of the project was established initially making it unnecessary to change the elevation.

After the DEMs have been generated, several profiles are extracted from the same location of the two DTMs. The profiles should be distributed to cover the entire study area to the greatest extent possible, forming a crisscross pattern to reduce random adoption errors. Finally, separate profile diagrams are drawn, and the elevation difference (∆Z) of the profiles is calculated. In this study, the mean error (ME) and root mean square error (RMSE) were used to assess the accuracies of the two methods (Javernick et al. 2014; Smith et al. 2014; Tamminga et al. 2015; Williams et al. 2013). The ME and RMSE were calculated as follows:

$$E=\frac{1}{n}\sum_{i=1}^{n}{\Delta z}_{i}$$
(6)
$$RMSE=\sqrt{\frac{{\sum }_{i=1}^{n}{{\Delta z}_{i}}^{2}}{n}}$$
(7)

where n is the number of extracted elevation points, and ∆Z is the elevation difference (representing the difference in elevation values at corresponding points in the DTMs generated by the two-point clouds), calculated as follows:

$$\Delta z= {Z}_{sUAV-SFM}-{Z}_{UAV-LiDAR}$$
(8)

Accuracy assessment based on the analysis of calculated earthwork volumes

This method analyzes the accuracy of the point clouds by calculating the earthwork results based on the different sets of point cloud data. As the study area is located along a section of the water transmission lines for the water resources allocation project, it has experienced extensive ground excavation processes. Moreover, the excavated earth was piled on-site, forming nearby piles of stacked earth. Calculating and comparing the difference between these two amounts should reflect the disparity in the accuracy between the two point clouds.

Traditional methods for earthwork calculations include the section, contour, and DTM methods (Chen et al. 2016). For the section method, the construction site is divided into a number of section planes, which are then used to calculate the volume of the surrounding earth. Although the calculation is simple, this method leads to large errors for complex topographies. The contour method uses the contour lines of the construction site to calculate the enclosed area, which is then multiplied by the contour interval to obtain the volume. This method is generally less accurate and not often used. The DTM method calculates the difference in the volume between the original and designed surfaces of the terrain.

In this study, the DTM method was used to calculate and analyze the earthwork volumes based on the two types of data, calculated as follows (Li and Jing 2010):

$$V=\sum \left({Z}_{Before}-{Z}_{After}\right)*{S}_{Cell}$$
(9)

where \(V\) is the earthwork volume, \({Z}_{before}\) and \({Z}_{After}\) are the elevation values of a single grid before and after the earthwork, respectively, and \({S}_{cell}\) represents the area of a single grid. The earthwork volume was obtained by summing the volumes of all the grids. We note that the DTM must be reversed prior to performing the volume calculation of the excavated area, indicating that the excavation pit has to be converted to a raised mound.

Case study and results

Study area

The study area (31.62° N, 114.07° E) is located approximately 118 km northwest of Wuhan, Hubei Province, China. The study is part of the Guangshui–Dawu section of the water resources allocation project in northern Hubei. The study area consists of various surface cover types, including low wild grasses, shrubs, isolated trees, water bodies, wasteland, and bare soil. Slope terrain changes are significant, including flat areas and steep slopes after excavation. Power lines or residential areas within the study area were absent, facilitating the safe operation of UAVs and rendering them ideal for conducting this type of research. The study area is denoted by the red frame in Fig. 5(b).

Fig. 5
figure 5

Study area: (a) Location within the Hubei Province; (b) satellite image from Google Earth of the study area denoted by the red frame; and (c) a site photograph

Full size image

Results of the point cloud processing

Following the aforementioned methods, the UAV images, image control points, and other data were imported into the Pix4Dmapper software to obtain the orthophotos (Fig. 6(a)), colored point clouds (Fig. 6(b)), and DSMs. Besides, the quality report of SFM from Pix4D was obtained. From the report, the mean RMSE was calculated to be 0.08. The ME value in X, Y, and Z directions was −0.01, 0.00, and 0.04, respectively. The ME and RMSE values of GCPs in the X direction were calculated to be −0.01 and 0.07, and in Y were 0.00 and 0.07, and in Z were 0.04 and 0.10, which indicates the image processing was successful.

Fig. 6
figure 6

Generated images and point clouds: (a) Orthophoto generated by sUAV-SFM; (b) sUAV-SFM colored point cloud; and (c) UAV-LiDAR point cloud. The study area was further divided into three sub-regions based on the topographic and land cover characteristics: A — flat sub-region, B — sub-region with bare soil, and C — excavated sub-region

Full size image

Fig. 6(c) shows the rendering result of the UAV-LiDAR point cloud. To compare the UAV-LiDAR and sUAV-SFM point clouds under the same conditions, the two types of point clouds were first cropped into the same range, after which their coverage areas were both set to 0.057 km2. A statistical analysis of the two types of point clouds was performed, whose results are listed in Table 2. The number of sUAV-SFM point clouds was found to be 3-fold higher than that of UAV-LiDAR point clouds. The data for the former also contained more components: each point concurrently contained coordinate and color data. In contrast, the point clouds generated by UAV-LiDAR only contained 3-D location data.

Table 2 Metadata for the UAV-LiDAR and sUAV-SFM point clouds
Full size table

Results of the ground points classification

The UAV-LiDAR and sUAV-SFM point clouds for the flat sub-region, sub-region with bare soil, and excavated sub-region were subjected to CSF filtering to extract the ground points. The parameters of the CSF filter were optimized by considering the undulating topographic features, such that the maximum number of ground points could be extracted (Table 3). Next, the point clouds of the three aforementioned sub-regions were filtered in turn. The points were manually verified, and those that were misclassified were corrected, yielding the final ground points.

Table 3 Parameter settings for CSF filtering of the point clouds
Full size table

Fig. 7 shows the effects of ground point classification. Fig. 7(a1) and (a3) are the two types of original point cloud data for the flat sub-region, while Fig. 7(a2) and (a4) are the results of the same sub-region after CSF filtering. For both types of point clouds, the artificial structure (aqueduct) at Position 1 and the vegetation cover at Position 2 were correctly classified as non-ground points, indicating that the filtering results were good. In the UAV-LiDAR point cloud, the waterbody at Position 3 on the flat sub-region became a hole after filtering.

Fig. 7
figure 7

Comparison of various point clouds before and after CSF filtering: A, B, and C correspond to the three sub-regions of Fig. 6, i.e., the flat sub-region, sub-region with bare soil, and excavated sub-region. a1/a2, b1/b2, and c1/c2 are comparisons of the sUAV-SFM point cloud before and after CSF filtering; a3/a4, b3/b4, and c3/c4 are comparisons of the UAV-LiDAR point cloud before and after CSF filtering

Full size image

Fig. 7(b1) and (b3) are the two types of original point cloud data for the sub-region with bare soil, while Fig. 7(b2) and (b4) show the results of the same sub-region after filtering. The effects before and after filtering were similar for this sub-region. The point clouds rendered in Fig. 7(b3) and (b4) have stripes traversing them. The cause of this phenomenon was the laser beam, which scanned the ground in a rotating manner when the laser scanner was acquiring data, resulting in inconsistencies between the longitudinal and lateral sampling rates, with the latter rate being significantly higher than the former.

The two types of original point cloud data for the excavated sub-region are shown in Fig. 7(c1) and (c3), while the filtered point clouds are shown in Fig. 7(c2) and (c4). A comparison between the two sets of point clouds revealed that the vegetation cover at Position 4 had been filtered, indicating that the filtering result for the excavated sub-region was satisfactory. Overall, the sUAV-SMF point clouds had a higher density, the ground features were more detailed, and the rendering of the ground points was smoother. The CSF-based classifications of the two types of point clouds were accurate, with satisfactory ground points obtained from both methods.

Accuracy assessment results of sUAV-SFM and UAV-LiDAR point clouds

Accuracy evaluation of point clouds based on GNSS survey

In order to assess the accuracy of the point clouds generated from UAV-LiDAR and sUAV-SFM, the 45 checkpoints obtained by the GNSS survey were used for the contrast analysis. The distribution of the checkpoints and vectors which shows direction and magnitude of the discrepancies were analyzed as shown in Fig. 8.Through Eq. (4), the \(MAE\) values representing the accuracies of the two-point clouds were calculated to be 0.07 m and 0.08 m. Moreover, the RMSE of the two cloud points were 0.08 m and 0.10 m. From the values of \(MAE\) and RMSE, it can be seen that values of LiDAR were both slightly smaller than that of SFM.

Fig. 8
figure 8

The distribution of the checkpoints and vectors showing direction and magnitude of the discrepancies

Full size image

Accuracy comparison of point clouds based on M3C2

Considering the substantial difference in the density of the two types of point clouds and the accuracy results assessed based on the GNSS survey, the UAV-LiDAR data was used as the reference point cloud for M3C2 analysis. Fig. 9 shows the results of the comparative analysis of the three sets of point clouds. Fig. 9(a2), (b2), and (c2) present the distributional characteristics of the M3C2 distances in the form of histograms. The M3C2 distance of the flat sub-region was 0.13 m, with a standard deviation of 0.17. In comparison, the M3C2 distance of the sub-region with bare soil and excavated sub-region was 0.03 and 0.02 m, respectively. Both were lower than that of the flat sub-region. Although the M3C2 distances of these two sub-regions were similar, the standard deviation of the M3C2 distance for the excavated sub-region was higher than that of both the flat sub-region and the sub-region with bare soil. In general, the M3C2 distance of the three topographic types in Fig. 9 did not exceed 0.13 m, indicating that the two types of point clouds had excellent goodness-of-fit and comparability.

Fig. 9
figure 9

Comparison of the sUAV-SFM and UAV-LiDAR point clouds based on the M3C2 method: a1, b1, and c1 are the M3C2 distances of the two types of point clouds for the flat sub-region, sub-region with bare soil, and excavated sub-region, respectively. a2, b2, and c2 are the corresponding M3C2 distance histograms, with the mean and standard deviation of the respective M3C2 distance stated at the top

Full size image

Accuracy comparison of point clouds based on DTM profile analysis

To evaluate the differences between the DTMs generated by the two types of point clouds, their profiles were extracted and drawn at the same position for an intuitive comparison, as shown in Figs. 10 and 11. These two figures compare the profiles of the excavation and bare soil sub-regions in the DTMs generated by the two point clouds. In the images, the solid red and blue lines are the profiles extracted from the DTMs generated by the sUAV-SFM and UAV-LiDAR point clouds, respectively. In the distribution of sections in Fig. 10 (excavated sub-region), six sections were extracted in the west-east and two sections in the north-south. For Fig. 11 (sub-region with bare soil), four sections were extracted in the north-south and northeast-southwest directions. From Fig. 10, it can be seen that Section h was identical but with slight deviations within a few centimeters. In addition, the curves in the other sections had high levels of conformity, indicating that the two DTMs had goodness-of-fit and similar accuracies. After a more detailed analysis, the sections generated by the UAV-LiDAR point cloud had “burrs” and were not as smooth as the sUAV-SFM sections, which was likely because there were more sUAV-SFM point clouds than UAV-LiDAR point clouds.

Fig. 10
figure 10

Comparison of the profiles for the excavated sub-region: (A) Distribution of Sections a–h and (B) Sections a–h, where the red and solid blue lines represent the sections extracted from the DTMs generated from the sUAV-SFM and UAV-LiDAR point clouds, respectively, ME is the difference in the mean elevation, and RMSE is the standard error of the difference in the mean elevation

Full size image
Fig. 11
figure 11

Comparison of the profiles for the sub-region with bare soil: (A) Distribution of Sections i–p and (B) Sections i–p, where the red and solid blue lines represent the sections extracted from the DTMs generated from the sUAV-SFM and UAV-LiDAR point clouds, respectively, ME is the difference in the mean elevation, and RMSE is the standard error of the difference in the mean elevation

Full size image

After extracting the sections of the sub-regions with bare soil and excavation, Eqs. (6) and (7) were used to analyze the difference and standard error of each section, where Figs. 10(B) and 11(B) show the results. Fig. 10(B), for the excavated sub-region, shows that the differences in the ME and RMSE values of each section in the east-west direction were small. The maximum and minimum mean values were 0.13 and −0.06 m, respectively, while that for the RMSE values were 0.064 and 0.012 m, respectively. The ME and RMSE values remained at the centimeter level. In the north-south direction, the mean value of the difference in Section h was 0.13 m. Although it was larger than the mean value of the section in the east-west direction, the RMSE of the difference was still smaller than that of the east-west section. Fig. 11(B) shows that the ME and RMSE values of the difference between the horizontal and vertical DTMs for the sub-region with bare soil were minimal. The maximum and minimum values of the mean for the differences were 0.04 and −0.06 m, respectively, and that of the RMSE values for the differences were 0.017 and 0.009 m, respectively. The results remained at the centimeter level, consistent with, but slightly better than, that of the excavated sub-region.

To analyze the comparison results of the profiles more intuitively and comprehensively, we calculated the differences in the average value of the analyzed results for the sub-regions with excavation and bare soil. These results indicate that the differences in the ME and RMSE of the excavated sub-region were −0.03 and 0.035 m, respectively; the differences in the ME and RMSE of the sub-region with bare soil were 0.00 and 0.005 m, respectively. An analysis of the two DTM profiles shows that the difference between the two profiles was small and the conformity was high, proving that the dense point clouds acquired using sUAV-SFM can have the same accuracies as that using UAV-LiDAR.

Accuracy comparison of point clouds based on surface volume calculations

To evaluate the difference between the two types of point clouds more accurately, the theory of calculated earthwork volumes, as described in the “Accuracy assessment based on the DTM profile analysis” section, was separately applied to the two-point clouds for the sub-regions with excavation and bare soil based on the same reference elevation. Table 4 lists the results of the calculated earthwork volumes. The earthwork volumes in the sub-regions with excavation and bare soil, calculated based on the two-point clouds, were basically similar. The absolute differences were 18 and 20 m3, respectively, and in percentage, the absolute differences between the two point clouds were 0.53% and 0.07%, respectively, both of which were within 1%, indicating that the results of the calculated earthwork volumes using the two types of point clouds were the same. This result again proved that dense point clouds acquired using sUAV-SFM and UAV-LiDAR could have similar accuracies.

Table 4 Results comparison of the calculated earthwork volumes
Full size table

Discussion

In this study, sUAV-SFM and UAV-LiDAR were used to obtain point cloud data of the study area, the point cloud filtering method was used to extract the ground points, and then different methods were used to compare the accuracies of the two types of point clouds. Before processing, the 45 checkpoints obtained through the GNSS survey were used for the initial accuracy assessment of the point clouds generated by UAV-LiDAR and sUAV-SFM. From the assessment results, it can be seen that the accuracy of the LiDAR was a little better than that of SFM, indicating that the point clouds of LiDAR, which served as a benchmark, were correct and reasonable.

During processing, vegetation cover and water bodies partially impacted the classification results of the point clouds under different topographic conditions. Previous studies have found that vegetation cover and land use/land cover changes (LUCC) affect the accuracy of point clouds (Liu et al. 2017; Luo et al. 2019). LiDAR can penetrate the canopy of vegetation and acquire ground points covered by low grasses and shrubs to obtain the topography underneath. However, the laser has limited energy and cannot penetrate dense vegetation. Besides, the shadow areas within the sides of wooden blocks, the edges of wooden blocks, and the flat and textureless evaluation chart area had poor image match (Alshawabkeh et al. 2021; Verma and Bourke 2019). Position 2 in Fig. 7(a1) contains taller isolated trees and shrubs. After filtering, a hole formed at this position in the point cloud because sUAV-SFM and UAV-LiDAR only acquired the canopies classified as non-ground points during CSF filtering, resulting in data holes.

Position 3 in Fig. 7(a1) is the filtered result of the water surface on the point cloud. Both point clouds did not correctly reconstruct the surface of the waterbody because both sUAV-SFM and UAV-LiDAR were unable to acquire surface data effectively. The incorrect water surface point clouds were generated by sUAV-SFM due to the weak texture characteristics of clean water surfaces, while UAV-LiDAR created data holes (the absorption of laser pulses led to a lack of point clouds). Therefore, both data acquisition methods face limitations when the ground cover consists of dense vegetation or water bodies.

Fig. 7(b1)–(b4) shows the filtered results of the sub-region with bare soil, where there was no major distinction in the classification between the two types of point clouds because there was no vegetation interference in the sub-region with bare soil, such that practically all of the point clouds were classified as ground points. Position 4 in Fig. 7(c1) consists of low grasses. Based on the original and filtered data of the two point clouds, the UAV-LiDAR point cloud had minimal changes at this position before and after filtering. After analysis, UAV-LiDAR was able to penetrate the low grasses and directly acquire the ground points, such that the corresponding UAV-LiDAR accuracy was higher.

Position 4 in Fig. 7(c1) is located at the edge of the excavated steep slope. The CSF filter grid was set to 0.10 m to preserve the point cloud of the steep slope (see Table 3) before its accuracy was evaluated using M3C2 (see Fig. 9(c2)). Although the M3C2 mean value was ideal, the standard deviation was significant. The main reason was that the small grid spacing caused the CSF particles to erroneously classify the vegetation cover at the edge of the steep slope as ground points. There was no guarantee that the classification was entirely correct despite the manual editing of the classification.

Position 5 in Fig. 7 is a disorganized stacking of soil and rocks formed during excavation. The surface roughness is large, and the texture feature is weak, causing sUAV-SFM to create matching errors, leading to a significant M3C2 standard deviation. Fig. 7(c3) and (c4) contain holes in the point clouds. Testing and analysis of the CSF algorithm revealed that when the grid was set to excessive size, a large number of ground points on the steep slopes that formed in the excavated sub-region were incorrectly classified as non-ground points. Therefore, holes were generated on the steep slopes, causing data loss, indicating that the results of the CSF algorithm were more sensitive to the parameter settings.

Slope affects the quality of the point cloud data (Guerra-Hernandez et al. 2018). By analyzing the original point cloud data, we found that UAV-LiDAR could properly acquire point clouds in the flat sub-region, but the point cloud density decreased rapidly in topographic environments with greater slopes. In comparison, sUAV-SFM used oblique photography to acquire images of the steep slope and could thus obtain detailed data. Position 4 in Fig. 7 is a steep slope in the excavated sub-region with low grasses. Comparing the two types of point clouds at this position shows a significant difference, possibly due to the slope, which caused the point cloud density to decrease here, yielding the notable contrast between the two-point clouds.

Table 2 indicates that the sUAV-SFM point cloud contained RGB color components, which can generate ultra-high resolution and real orthophotos. These are helpful for the interpretation of ground objects. After analyzing the acquired results of the two types of point clouds, we found that sUAV-SFM could accurately acquire the elevation data of bare ground. The point cloud density was high, and the noise was prevalent. Although the CSF method can effectively extract the ground points in areas with vegetation cover, CSF filtering was observed to be sensitive to the parameters for topographic features. Through further analysis, SFM photogrammetry based on optical images was found to possibly create incorrect results when modeling areas with vegetation cover. The texture of areas with vegetation cover is similar, such that features with the same name cannot be extracted for matching; in contrast, the leaves on trees are susceptible to the effects of winds, causing mismatching.

A comparative analysis of the profiles and results of the calculated earthwork volumes in the point cloud rasterized DTMs revealed that both types of point clouds maintained a high degree of conformity on the DTM profiles, and the difference between the earthwork volumes remained within 1%, indicating that both types of point clouds had similar levels of accuracy.

This analysis and discussion show that sUAV-SFM point clouds acquired by consumer UAVs have similar accuracies as UAV-LiDAR point clouds. However, the former incurs lower equipment costs, and UAVs are more portable and easier to operate, such that data acquisition based on sUAV-SFM has extensive applicability.

Conclusion

The UAV airborne platform was used to concurrently acquire two types of point cloud data for the study area. First, the CloudCompare software was used for pre-processing of the two datasets. Subsequently, the same CSF algorithm was used to filter the two kinds of point clouds for the different sub-regions within the study area before extracting ground points. Finally, direct and indirect evaluation methods were used to assess the accuracies of the sUAV-SFM and UAV-LiDAR point clouds. Before comparing the two point clouds, the 45 checkpoints were used to assess the initial accuracies of the two kinds of cloud points, and the results suggested the accuracy of the point cloud generated through LiDAR was little better than that of SFM. Given this, the point cloud obtained from LiDAR was chosen as the benchmark in the following comparison analysis. The direct evaluation method was based on the M3C2 algorithm, which assessed the accuracies of the point clouds under the conditions of distinct topographic features; the indirect evaluation method used the rasterized DTMs of the point clouds to analyze the generated profiles and calculate the earthwork volumes. The M3C2 comparison results revealed that the mean M3C2 distances under various topographic conditions were almost zero, indicating conformity in the accuracy of the two types of point clouds. The results of the DTM profile analysis showed that the DTMs generated by the two types of point clouds were basically consistent and only had minor errors. The results of the calculated earthwork volumes indicated that the surface volumes of the sub-regions with bare soil and excavation in the two types of point clouds were relatively close, indicating that both types of point clouds had similar accuracies for this calculation method. Overall, under complex topographic conditions, sUAV-SFM can obtain dense point clouds with the same accuracies as UAV-LiDAR. In addition, we note that data acquisition based on consumer-grade UAVs has the advantages of low cost, convenient operation, and easy portability. Thus, sUAV-SFM has broad application prospects in construction sites monitoring, earthwork measurement, glacial degradation, river restoration assessments and so on.