Keywords

5.1 Introduction

Leveraging the unprecedented growth of data collected through mobile devices, drones, laser scanners, rovers and sensors on construction sites today (Ham et al. 2016; Han and Golparvar-Fard 2017), CPS technologies enable continuous progress updates from the downstream to establish the bidirectional communication cycle for effective project controls. With the recent development of computer vision and robotics in construction and the adaption of n-dimensional Building Information Modeling (BIM), the collected data are processed and integrated with BIM, project schedules, and used to compare between Reality and Plan for completing the feedback loop of CPS in construction progress monitoring (Lin and Golparvar-Fard 2018). This CPS for construction progress monitoring could improve the process of project control and enhance communication, coordination, and planning during the project execution. A typical process of project control usually includes (1) weekly plan coordination; (2) progress and issue tracking through job walks; (3) progress and issue documentation; and (4) plan review and adjustment. To accomplish the aforementioned process, the CPS needs to have a complete workflow from data collection, progress monitoring, activity analysis to reporting and decision making. Research has been investigating automated data collection through drones, rovers, and sensors; progress monitoring and activity analysis through the latest computer vision techniques such as material classification, object detection and tracking; reporting through color-coded models with predictive data analytics and digital daily construction reports. CPS that integrates the components above with project control theories and construction management workflow is a potential solution that addresses the drawbacks of the current status of project control through progress monitoring.

Efficient progress monitoring provides project stakeholders- owners, contractors, subcontractors- the updated information for project control decision making (Golparvar-Fard et al. 2011; Yang et al. 2015). With the construction put in place value surpassing $1.2 trillion (U.S. Census Bureau 2019), reports still show workers spend about 30% time of the week on solving avoidable issues such as looking for information and 50% of rework are caused by miscommunications. To improve the efficiency of construction, research has been focusing on the development of project control theories and progress monitoring technologies.

Project controls theories such as Last Planner System (Ballard 2000) has achieved better planning and communication that stabilize workflow by preventing direct work from upstream variation and uncertainty. Even though the benefits and achievements are widely documented, it remains more of an art than science to accomplish its full potential throughout the construction lifecycle and across different projects. Recent empirical studies indicate that the implementation of the control mechanism requires full commitment from all project team members and a dedicated champion in a relatively long learning process. With the absence of the champion, the project control workflow could easily revert to traditional practices (Leigard and Pesonen 2010; Sacks et al. 2010, 2013; Gurevich and Sacks 2014; Dave et al. 2015). While these challenges are mostly attributed to the organizational and people process involved in implementation, there is a growing interest to leverage production control theories with progress monitoring technologies such as CPS to better understand, analyze and communicate the performance problems while preserving a two-way communication information flow.

Research has developed CPS that leverages the visual data collected on the construction site and applied state-of-the-art computer vision techniques to produce 3D reality models of ongoing operations and automatically organize and manage them over project timelines. The integration of these models with Building Information Modeling (BIM) and project schedules enables deviation analysis between Reality and Planned to better communicate the production status through color-coded models and reports generated by predictive data analytics (Lin et al. 2015; Lin and Golparvar-Fard 2016, 2018; Han and Golparvar-Fard 2017). However, to successfully implement this CPS system for progress monitoring requires four key components:

Data Collection

Desired frequency and completeness is necessary to sustain a smooth information input for progress and activity monitoring. The current practice of data collection still highly relies on manual procedures where quality and quantity usually do not meet the requirements for efficient project control. Progress monitoring data collected on construction sites can be categorized into visuals, sensors and text data with a range of formats from photos, videos, texts to laser scans. The collection process can be through commodity smartphones, drones, rovers, and different vehicles. To ensure the quality of the data collection process can produce informative data for progress monitoring, recent research mainly focused on areas such as completeness of reality models, feasibly integrated platforms for different environments, efficient data collection strategy driven by change detection.

Progress Monitoring

Providing timely progress updates of the Reality and comparing it against the Plan model can keep a smooth information flow of production. Previous research presents a typical 3D reconstruction pipeline to create 3D geometric Reality models from hand-held cameras and registered to the 4D (3D + time) Plan model. Geometry-based and appearance-based progress monitoring techniques are performed to automatically analyze the space occupancy and material from 3D models and images for progress status. However, there are still plenty of open research problems and challenges to fully automate the process. For example, improving the efficiency and reliability of image-based reconstruction, material recognition, geometry analyzation, and camera viewpoints optimization.

Activity Monitoring

Near real-time analysis of worker activities and equipment are used to help the management of onsite construction. Current practice is expensive, labor-intensive and incomprehensive due to the inefficiency of the manual procedure, inability of covering multiple locations and restrictions of computational power. Research has focused on developing camera networks that could track and connect different locations, and leveraging the recent development of machine learning and deep learning to improve the efficiency and accuracy of worker and equipment detection.

Reporting

Visualizing the analysis result in an intuitive interface and an accessible platform is important in construction to complete the feedback loop of CPS. The interface needs to convey the analysis in construction grammar and express all critical states and changes associated with the Reality model. Research has used traffic light metaphor on BIM models to present the status of progress through web-based platforms. The analysis was also presented in the form of charts and figures in dashboards or in typical spreadsheets. These reports are used in the current construction workflow to improve and enhance the decision-making process, such as (1) 3-week look-ahead scheduling and coordination among key stakeholders of a project (state-of-the-art practice), and (2) daily operation planning by job site superintendents and foreman to set performance targets for their workforce and study potential improvements (Fig. 5.1).

Fig. 5.1
figure 1

The CPS contains data collection, progress and activity monitoring and reporting. The UAVs, rovers and network of fixed cameras are performing change-driven autonomous data collection, the cloud system processes the collected data and analyze the production rate and progress and provide an action plan for the project managers’ decision-making process

Full size image

With the current research advances in each domain significantly but separately, this chapter provides a holistic view of how to integrate the bits and pieces to complete the CPS for construction progress monitoring. In the following section, an overview of the state of the current construction industry is presented from a project control perspective with specific practical problems. Next, the opportunity of CPS using visual data as a source for capture, analytics, and representation of Reality and Planned data on construction projects are discussed. Next, we present the data collection process and the latest research on improving and automating this process through robotics and computer vision. We also discuss how the cutting-edge computer vision and machine learning techniques such as deep learning, object detection, and Structure from Motion together with BIM and schedule are integrated and applied in the context of CPS to enable performance monitoring at both project task and operations level. The comparison of reality vs. plan will be discussed in detail for progress monitoring. For each use case, we will provide a concise literature review and assessment of current state-of-the-art solutions in the market, and will discuss the key underlying methods and recent solutions in detail. We will demonstrate their performance in the real-world by using a building project case study. The challenges of applying CPS in typical project workflows and open areas for research and development are also discussed in detail.

5.1.1 The State of Productivity in Construction

Today, the construction industry is still plagued with inefficiencies, including cost overruns and delays in execution of projects. The average productivity compound annual growth of US construction was negative while other industries such as manufacturing, oil and gas and other sectors were performing three times better than construction. According to KPMG’s Global construction survey (Armstrong and Gilge 2017), only 25% of projects finished within 10% of their original schedule and only 31 percent of all projects completed within 10% of their budget in the past 3 years of common commercial and industrial building projects. While best practices such as the Last Planner System and lean construction principles do improve schedule and cost performance on construction sites, still 22% and 13% of projects where best practices are implemented exhibit schedule delays and cost overruns respectively (Beven and Jones, 2016).

The waste and variations also come from the utilization and productivity of the workers, equipment, and materials. Recent research indicates craftspeople spent an average of 52% of their work hours at the workface while only 34% they stayed for more 10 min which are considered as value-creating, and 50% of their time they were not at where the schedule shows. The workers have little control over their own productivity lost throughout the day, because of the material locations and plan changes.

The same report by KPMG (2016) shows that among various project controls metrics, companies consider adherence to the project schedule is the number one issue that they face in the execution of their projects. There is a myriad of factors that have contributed to the lack of growth in construction productivity and the complexity of executing projects on time and on budget. A careful examination of the most recent studies and reports including McKinsey & Company (Changali et al. 2015; Barbosa et al. 2017), KPMG Global Construction Survey (Armstrong and Gilge, 2017), ENR Dodge Data and Analytics (Beven and Jones, 2016), and internal anecdotal observations from more than 100 construction projects over the past 10 years that the Real-time and Automated Monitoring and Controls (RAAMAC) lab at the University of Illinois at Urbana-Champaign has been involved in, has revealed a list of issues as key contributors and the root-causes of lack of productivity in the construction industry: (a) Inadequate communications- inconsistencies in reporting of project plans and actual work in place makes it difficult for subcontractors, contractors, and owners to maintain a common understanding of how projects are progressing at any given time; (b) Flawed performance management – due to the lack of systematic and frequent communication and accountability in execution, the unresolved issues quickly stack up; (c) Poor short-term planning- construction firms are good at understanding and planning progress to be achieved in 2–3 month but rarely have an insight for next week or two; (d) missed connections to actual progress - the individuals involved in project planning or revising short-term and long-term plans are usually not working on construction sites; (e) Insufficient risk management- reliability and risk in short-term project plans are not systematically assessed; and (f) Poor decision-making- day-to-day planning and decision making is frequently inhibited due to poor communication surrounding daily work progress. In the next section, we discussed the opportunities of leveraging the growth of data in construction for progress monitoring.

5.1.2 The Unprecedented Growth of Data in Construction

Although the construction industry has been seen as one of the worst in terms of technology adoption rate, the collection of visual data throughout the construction process has grown exponentially in recent years. Onsite personnel collects daily photos to document the progress through smartphones and tablets and uses various applications to track issues and changes. The easy access to cameras and other technologies such as camera-equipped ground/aerial vehicles increases the number of images and videos gathered on construction site tremendously. Recent research shows that there are about 325,000 images are taken by professional photographers, 95,400 images by webcams, and 2000 images by construction project team members at a typical commercial building project (∼750,000 sf). This trend of using visuals and sensor data to document construction progress provides a unique opportunity for CPS as inputs to keep the production status updated.

5.1.3 The Potential of CPS for Construction Progress Monitoring

Through the previous sections, reports and research indicate that waste and inefficiency occur in construction due to the failure of maintaining smooth communication. With the unprecedented growth of visual data as an input in the CPS, it could improve communication and establish a bidirectional feedback loop for construction progress monitoring. The visual data provide the Reality in the CPS to continuously update the current progress of construction. On the other hand, with the broad implementation of n-dimensional BIM (i.e. 3D models enriched with information such as time, cost, safety, and productivity), enhanced 3D visualization with semantic building information provides the Plan in CPS to compare against the Reality for progress verification. Different use cases have shown the value-added by utilizing BIM from early design phase to facility management, such as Lu et al. (2014) report 6.92% cost saving by using BIM, Staub-French and Khanzode (2007) report 25–30% productivity improvement by using BIM for coordination and constructability reviews to identify design conflicts. The integration of Reality (visual data collected onsite) with Plan (nD BIM) can efficiently communicate the necessary information for successful project control.

5.2 Review of Current State-of-the-Art CPS Technologies for Construction Progress Monitoring

CPS for construction progress monitoring can be divided into four key components: data collection, progress monitoring, activity monitoring and reporting. Over the past few years, researchers have been developing and validating new robotics, computer vision and predictive data analytics techniques for these components in the construction domain, and many of these have already been used in the industry. The following sections introduce the state-of-the-art CPS technologies in research and industry in terms of progress monitoring, we will discuss the fundamental theory behind the applications and the implementation of the real-world use cases.

5.2.1 Data Collection

The current practice of data collection is still relying on manual procedures of photo taking and video camera set up. The process is time consuming, costly and often does not guarantee the completeness of the capture. To address these challenges, recent studies have focused on automating the process through unmanned aerial and ground vehicles to acquire visual data (Ibrahim et al. 2017; Asadi et al. 2018; Ibrahim and Golparvar-Fard 2019). These systems often are equipped with multiple types of sensors and cameras and integrated with a computational platform that could perform autonomous navigation in a construction environment. The vehicle needs to collect data for progress monitoring and automatically navigate and map the environment at the same time. In this section, we will discuss the current technologies used in unmanned aerial and ground vehicles, and the optimization of the data collection process.

5.2.1.1 Autonomous Data Collection

Autonomous data collection is developed to provide the desired frequency of data collection and to ensure the completeness of the resulted 3D Reality models usually through unmanned aerial vehicles (UAVs) and ground vehicles/rovers/mobile robotic system. Because of the nature of the two platforms, UAVs are often used for exterior environment data collection and rovers are used for the interior environment data collection on construction sites. Although there is also a significant amount of research utilize UAVs for interior navigation and mapping, there is currently little adoption in construction due to safety concerns. The following sections discuss the applications of these two different types of robotic system in construction site.

Unmanned Aerial Vehicles

Performing autonomous flight through UAVs in an exterior construction environment is relatively mature because of the well-developed GPS-based navigation technologies. With good reception of GPS, UAVs can perform autonomous flight through predefined flight plan according to the requirements and guidelines. Currently, there are multiple applications provide flight planning feature for users to plan the flight on the map before going to the actual construction site. To ensure the completeness of the resulted 3D Reality models, these tools usually require input for a minimum percentage of overlap between images. However, there are still several challenges regarding the completeness and efficiency of the flight plan: (1) flight plans are based on existing orthographic map that does not consider the complexity of the building structure; (2) risks associated with permanent and temporary structures are not considered; (3) flight plans only support 2D plans with]] specific patterns which do not capture the z-dimension of the structure and often results in Reality model without the views from sides; and (4) construction usually progress in specific areas significantly while other areas remain almost unchanged, current flight plan is not driven by changes of the construction.

To overcome these challenges, researchers utilize Reality data generated beforehand as a priori to create flight plans accordingly with a fixed safety distance to ensure the completeness of the results. 3D flight plans are also created using a bounding box around the target structure with a preset offset to maintain a safe distance to the target. BIM-driven visual quality metrics are developed to create flight plans that guarantee the completeness of Reality and quality of images for progress monitoring. Change-driven flight plans are also developed by using the 4D BIM as a priori to predefine the frequency and coverage for the construction site. Flight simulators are developed to ensure the safety and visibility of the UAVs. UAVs based Reality model evaluation is developed by synthesizing feature tracks using flight plans and BIM or Reality models to better control the variables during the actual flight.

Unmanned Ground Vehicles, Rovers, Mobile Robotic System

Unlike UAVs, rovers are often deployed in an interior construction environment that is limited to GPS-based navigation . With limited computational resources on the platform, the real-time vision-based processing system can also only deal with relatively simple planning tasks. To overcome the challenges and support construction progress monitoring using rovers, autonomous navigation through onboard processing unit that can integrate multiple types of sensory and visual data is necessary. Simultaneous Localization and Mapping (SLAM) is used in robotics to create and update the map of an unknown environment while simultaneously identifying the location of itself in the map. To enable autonomous navigation, rovers obtain information from multiple types of sensory and cameras and process it using SLAM onboard to quickly localize images taken in the building and mapping the environment. Hector SLAM (Kohlbrecher et al. 2011, 2014), ORB SLAM (Mur-Artal et al. 2015) uses different sensory data such as 2D LiDAR sensor and monocular camera to build a navigation map. Although SLAM is well suited for construction interior mapping, it does suffer from with drift errors leading to misalignment of local maps and affect the navigation. Research uses Internal Measurement Unit (IMU) data and Extended Kalman Filtering (EKF) to improve the accuracy of localization and reduce the mapping errors (Einicke and White 1999). SLAM-related research in construction has focused on generating and registering point clouds in an efficient and inexpensive way (Jog et al. 2011; Brilakis et al. 2011a; Amer and Golparvar-Fard 2018), and autonomous navigation for construction progress monitoring(Jin et al. 2018; Kim et al. 2018c; Asadi et al. 2018). Several studies integrated BIM-driven path planning, Ultra-Wideband indoor positioning and other sensors as a mobile robotic navigation system for indoor construction applications. Other than autonomous data collection, recent research has also investigated methods to better distribute cameras and to analyze the best camera viewing angle for optimizing the data collection process. (Fig. 5.2)

Fig. 5.2
figure 2

An example of an autonomous rover settings and operating in construction site, the rover can navigate, map and analyze data in real-time automatically

Full size image

5.2.1.2 Optimization of Existing Data Collection Process

Camera placement is critical to effectively monitor operation-level activity monitoring. Current strategies of placing fix cameras onsite are largely depending on the engineers’ experience and surrounding environment restrictions (Kim et al. 2019). The problem of camera placement is similar to the well-studied art gallery problem, research has integrated construction-related variables and developed mathematical methods to optimize the numbers, locations, types, and orientations of the camera placement in construction sites. Research on cost and coverage optimization and BIM-driven indoor camera placement also shows the potential to monitor interior spaces. To streamline the data collection process for progress and activity monitoring, we have discussed the autonomous data collection and optimization of the existing process. In the next section, we will introduce state-of-the-art research in construction progress monitoring.

5.2.2 Construction Progress Monitoring Techniques

Current practices of construction progress monitoring still highly rely on site engineers conducting job walks to document the status and issues. While this process is labor-intensive, costly and subjective, research has developed methods that utilize reality capture techniques to obtain as-built status and compare to the 4D BIM for automated progress monitoring. Reality capture has also gained popularity in practice for construction progress monitoring by providing visual verification and measurement capability in recent years. In this section, we will review the latest research and commercial application on using reality capture for progress monitoring.

5.2.2.1 Reality Capture Techniques

Reality capture transforms real-world subjects such as buildings, site conditions, bridges into to digital model representation. This process results in 3D models that are formed by millions of points or meshes that are usually called point clouds or mesh models. Image-based 3D reconstruction and laser scanning are the two techniques that are widely used in practice. Image-based 3D reconstruction can take in all images taken from different sources on the construction site and generate the Reality model. It is currently used as one of the main documentation and project control tool. Laser scanning provides high accuracy results that could be used for quality control and assessment, but the process is relatively time-consuming and labor-intensive. These techniques provide site engineers with quick and accurate access to the current site conditions, where it has the potential to replace the traditional time-consuming site survey and daily job walks. Using Reality capture for progress monitoring.

Schedule task-level progress monitoring uses computer vision techniques to obtain the task status by analyzing the geometry and appearance of the corresponded task location in the Reality model. The task locations that are derived from Work Breakdown Structure (WBS), task names, task IDs, 2D drawings and 4D BIM are shown as an area in the images or volumes in the 3D point cloud models. To examine the state of progress of the task, the geometry and appearance of the task location in the Reality are then compared to the Plan to determine the status. Geometry is used to analyze the physical occupancy of the element, and appearance is used to examine the state of the task at the same location. Today, there are two dominant practices for leveraging images for tracking work in progress:

  1. 1.

    Generating large panoramic images of the site and superimposing these large-scale high-resolution images over existing maps (see Fig. 5.3) – While these images provide excellent visuals to ongoing operation, they lack 3D information to assist with area-based and volumetric-based measurements necessary for progress monitoring. Also, none of the current commercially available platforms provide a mechanism to communicate who is working on which tasks at what location and they mainly deliver high-quality maps of construction sites.

Fig. 5.3
figure 3

DroneDeploy and Skycatch drone based visual data management platform – high-level top-down images are used to produce large-scale high resolution orthophotos and overlay them over existing maps. These images are also used to generate point cloud models

Full size image
  1. 2.

    Producing 3D point cloud models–The state-of-the-art in image-based 3D modeling methods from computer vision domain has significantly advanced significantly over the past decade. These developments have led to several commercially available platforms that can automatically produce 3D point cloud models from collections of overlapping images.

In practice, today’s several AEC/FM firms have started to utilize Reality models generated from images taken by UAVs to document progress and issues. However, to fully reach the potential of using Reality models, it needs to be integrated with 4D BIM models to streamline the process of project controls.

5.2.2.2 4D Reality Capture Integration With BIM

Point clouds generated from the typical Structure from Motion pipeline have arbitrary coordinate systems. Although the current 3D reconstruction process usually uses the GPS information from the image metadata, the output point clouds could still be up-to-scale due to missing or inaccurate information. Thus, their pixel units do not directly translate to real-world Cartesian coordinates. To register the point clouds in the world coordinate system, at least three points or correspondences with Ground Control Points or BIM are required. The three correspondences are used to solve for the similarity transformation between the two coordinate systems (Golparvar-Fard et al. 2009, 2012). These corresponded could be based on (1) setting visual surveying benchmarks with known real-world coordinate systems such that the user (manually or through an automated detection procedure) can establish their correspondence with site coordinates, or (2) manually finding correspondence between up-to-scale point clouds and BIM. Examples of GCP are shown in Fig. 5.4, where markers can be automatically or manually detected and their coordinates from the point cloud data can be matched to their equivalent from 4D BIM.

Fig. 5.4
figure 4

Example of GCPs (marked in red) place on the construction site for registering the Reality models to BIM or real-world coordinate system

Full size image

Several researchers have also focused on automating the process of alignment between BIM and point clouds without markers or GCPs. This especially becomes a difficult problem in built environments where structures and elements usually share similar geometry shape with symmetric characteristic. Previous works achieved limited automated registration with pre-defined constraints (Nahangi et al. 2015), semi-automated approaches (Bosché 2012), limited symmetric geometry identification or partial or pre-processed data (Son et al. 2015) and prior information assisted system (Bueno et al. 2018). While this research area remains open, general purposes such as progress monitoring can be satisfied with manual registration discussed above. With having the BIM model registered to point cloud model, progress information extraction and comparison of reality vs. plan are discussed in the next section.

Methods to Compare Reality Versus Plan

Research on visual construction monitoring has focused on an automated comparison of 4D BIM with time-lapse videos, or 3D image-based and laser scanning point clouds. These investigations are mainly focused on how the physical presence of building elements or their appearance can be detected. Much additional work in model-driven visual sensing is needed to bring these methods into an application. Also at best, these methods only tie performance deviations with retrospective Earned Value metrics and do not communicate who is working on what task in what location on a daily/hourly basis. Hence, a major time lag exists between facing an issue on-site once work is underway and when managers and other trades on the site are informed to mobilize teams into unoccupied locations, streamline workflows, and minimize waste. The inability to have two-way communication on task scope, methods and resources also delays work approvals, quality inspections, contractor hand-overs, and leads to waste.

The state-of-the-art methods of automated comparison are still in its infancy. Largely because these methods leverage the geometry of the 3D reconstructed scenes to reason about the presence of elements on the construction sites. As such, they are unable to differentiate operations details such as finished concrete surfaces vs. forming stage and cannot accurately report on the state of work-in-progress. On the other hand, methods that detect and classify construction material from 2D images have primarily been challenged in their performance due to their inability to reason about geometrical characteristics of their detected components.

Geometry-Based Progress Monitoring

Image-based 3D point clouds are generated through an SfM-MVS pipeline and integrated with BIM model in Industry Foundation Classes (IFC) format to reason about the occupancy and visibility of the elements. A supervised machine learning method that utilizes Support Vector Machine (SVM) is developed to determine the state of progress (Golparvar-Fard et al., 2012). On the other hand, laser scanning point clouds of Mechanical Electrical Plumbing (MEP) system are compared against the BIM model to monitor progress for interior construction progress which has a lower tolerance of accuracy (Bosche et al. 2014). To effectively differentiate operation details of concrete activities such as formwork, rebar and concrete placement, research has developed methods to detect construction objects (Turkan et al. 2012). However, these methods are limited in their ability to detect operational details and the occlusion and visibility of elements in the point clouds. (Fig. 5.5)

Fig. 5.5
figure 5

Progress is shown in as-built and 4D BIM models with color-coded status superimposed together (left) (Golparvar-Fard et al. 2012), laser scanned as-built (middle) and 4D BIM model (right) (Turkan et al. 2012)

Full size image

Appearance-Based Progress Monitoring

To efficiently detect operation level progress of tasks located in the same place, appearance-based methods focus on using computer vision techniques to classify the material of the task location and integrate with geometry information to infer the progress status. For example, formwork and concrete placement activity occur at the same location but with a different appearance, as a result, occupancy-based method is unable to detect the difference while appearance-based method can detect the material. Research developed methods to backproject planned BIM element location to the corresponding image using the camera information from the 3D reconstruction process, and classify the construction material from the image patches (Han and Golparvar-Fard 2015). They further leverage the geometry feature of the image patches to enhance the accuracy of material recognition (Han et al. 2018). However, these methods are unable to utilize the geometrical characteristics of their detected components (Fig. 5.6).

Fig. 5.6
figure 6

Using patches retrieved from BIM to 2D back-projection to classify material and performing depth test to exclude occluding area (left); progress status is extracted by comparing as-built and as-planned after occupancy detection and material classification (right)

Full size image

Even though significant improvements are achieved in the past decade, to automatically detect the progress in full-scale projects within the CPS requires (1) accounting for the lack of details in 4D BIM, (2) addressing as-built visibility issues, (3) creating large-scale libraries of construction materials that could be used for appearance-based monitoring purposes; and (4) methods that can jointly leverage geometry, appearance, and interdependency information in BIM for monitoring purposes.

5.2.3 Activity Analysis

To detect activities and operation-level details, site engineers analyze the video footage from the fixed cameras manually and create a crew balance chart to understand productivity and safety. The process of manual examination is time-consuming, labor-intensive and expensive. Besides, the high per-hour cost of the heavy equipment and risk of struck-by accident when workers are on the site also draws attention from construction researchers. Recent research used the latest computer vision techniques such as object detection, tracking and pose estimation, to analyze the activity of construction resources and monitor resource allocation and progress. The following section introduces the state-of-the-art computer vision techniques that are applied in the context of construction for activity analysis.

5.2.3.1 Computer Vision Techniques for Activity Monitoring

Activity analysis includes several computer vision tasks to successfully analyze a complete sequence of activities. The method needs to first identify the construction resource, track the pose of the object and further estimate the movement of the object. Each step is considered a challenging task for a computer to automatically perform (Szeliski 2011) because the occlusion, appearance, and poses of the object can vary in different environment settings. Traditional methods such as bag of words, brute force matching against large databases have been proven not reliable because of its high dependencies on the surrounding elements. Machine learning methods such as boosting, neural networks, SVM and recent deep learning approach have attracted more attention to address the challenges of activity analysis. To apply these detection methods to fixed cameras’ video footage, it also involves object tracking from a sequence of video frames and pose prediction and association between frames.

Computer vision-based operation-level monitoring focuses on tracking the construction resources (workers, equipment and materials) and analyzing the interaction between each other via visual data collected on the construction site. Object detection and activity recognition techniques are applied on construction equipment and workers to track the trajectory and motion for measuring the input resources in each activity (Fig. 5.7). For equipment productivity analysis, single and multiple equipment activity recognition methods are developed to examine the earthmoving and dump truck efficiency (Golparvar-Fard et al. 2013; Kim et al. 2018b), then dirt loading cycling time is evaluated through the identified activity to improve productivity (Rezazadeh Azar et al. 2013). Point cloud volumetric measurement with video analysis for finer time scales productivity estimation are fused to analyze the productivity onsite to the schedule task level (Bügler et al. 2017). These productivity data are also inputted into simulation models to better estimate task completion and project duration (Kim et al. 2018a). Besides equipment productivity analysis, pose estimation and worker detection method are also developed for work sampling automation and productivity assessment. To be able to train a machine learning method to detect worker’s activity, researcher has developed a crowdsourcing web-based annotation tool to gather ground truth data efficiently (Liu and Golparvar-Fard 2015). Ironworker, carpenter activity are classified into 16 different types of activities for individual work through surveillance videos (Luo et al. 2018b). Activities are also recognized through the spatial and temporal relevance between workers and objects where 17 types of construction activities are recognized (Luo et al. 2018c, a). The majority of these works only track the location of the workers. However, without interpreting activities and purely based on location information, deriving meaningful workface data is challenging (Khosrowpour et al. 2014; Yang et al. 2015). For example, for drywall activities, distinguishing between idling, picking up gypsum boards, and cutting purely based on location is difficult, as the location of a worker would not necessarily change during these tasks.

Fig. 5.7
figure 7

Using the image sequence from a network of fix cameras, research developed methods that automatically detect the equipment, the activity and the locations. The output could be generated in the form of crew balance chart and used to improve the productivity

Full size image

However, computer vision methods are also not advanced enough to conduct detailed assessments from videos or RGB-D data because methods for fully automated detection and tracking (Brilakis et al. 2011b; Escorcia et al. 2012; Memarzadeh et al. 2013), and deriving activities from long sequences automatically (especially when workers interact with tools) are not mature (Gong et al. 2011; Golparvar-Fard et al. 2012; Khosrowpour et al. 2014). The current taxonomy of construction activities also does not enable “visual activity recognition” at a task level to be meaningful for workface assessment (Liu and Golparvar-Fard 2015). While full automation is appealing, training machine learning methods require very large amount of empirical data which is not yet available to the construction informatics community (Liu and Golparvar-Fard 2015).

The following section discusses the development of organizing the information analyzed from the data collection, progress and activity monitoring into construction language that can be used for project managers’ project control decision making.

5.2.4 Construction Progress Monitoring Reporting

To organize the analyzed data into an actionable deliverable, reporting completes the last mile of the progress monitoring CPS. Reporting support decision making for proactive project control, visual verification for production tracking and progress documentation for various purposes such as billing and issue management. Research has developed a web-based system that color-code the Reality and Plan model according to the status; dashboards and reports that organize construction data into informative predictive metrics, charts and weekly work plan format; and daily construction reports that formalize the analyzed data into a company-specific format that could serve as billing and documentation purposes. In this section, we will discuss the background and applications of these reporting formats.

5.2.4.1 Color-Coded Reality and Plan Model

Construction practitioners have been using color-codes to present the progress of construction through different interfaces. Even in today’s construction, it is common to find printed 2D drawings highlighted with different colors to communicate the current status of various locations (Fig. 5.8). Research has also been investigating using color-coded models to visualize the status, performance and risk (Golparvar-Fard et al. 2009; Han and Golparvar-Fard 2015; Lin and Golparvar-Fard 2018). Color-coded models are often used during coordination as a visual aid to facilitate communication. Several examples show that using a color-coded model is easier to visually communicate the issues and identify potential risk. For example, the façade of complex high-rise buildings usually involves several trades working in parallel on top of each other. This becomes a major coordination task for project managers to coordinate the sequence and safety between subcontractors. Without the use of color-coded models, it is hard to visualize at what time which subcontractor is working at what location. With the color-coded models, each subcontractor is represented as one color, and the 4D model highlights the BIM elements of the responsible contractor with its color as the timeline moves. This representation could facilitate the communication and planning regarding sequencing, resource management and logistics in coordination meetings so that issues are found beforehand, and the plan can be adjusted accordingly in real-time. Status visualization of delay and on schedule is also used during the daily huddles on the construction site. Superintendents review the progress of each subcontractor through the 4D BIM model during the meeting daily huddles and adjust the tasks accordingly. Risk visualization of locations is useful during schedule meetings to identify the potential delays based on location and discuss the action plans circling the location from the models. With research showing much progress on visualization, color-coded models is also provided by construction software such as Navisworks and Synchro for various purposes such as 4D BIM simulation, delay and trade location visualization, Earned Value Analysis. Whereas color-coded visualization is helpful for coordination and planning, construction infographic dashboards and reports can quickly provide project managers and other stakeholders a grasp of the project status.

Fig. 5.8
figure 8

Examples of printed 2D drawings with highlighted progress to communicate the actual status, this process is time-consuming and labor intensive

Full size image

5.2.4.2 Construction Infographic Dashboards and Reports

Infographics provide an overview of the project status intuitively. Project managers usually prefer a higher level of information that could numerically and visually summarize project performance. Different metrics and charts are accordingly developed to indicate the status of projects. Among various metrics for progress monitoring, Percent Plan Complete (PPC) are widely used to track the ratio between the actual completed tasks to the planned committed tasks. PPC is easy to understand and allows project managers to quickly examine the reliability of the short-term plan retroactively over time. Coupling the PPC with the root-cause analysis enables indirect production flow tracking to improve the short-term plan in a weekly cycle. However, PPC does not capture the production flow directly and the numbers could be deceiving as it only reflects the progress on the short-term plan without connecting it to the master schedule (Sacks et al. 2017), for example, the PPC could be 80–90% for the week, but the overall project progress is behind schedule. To address the shortcomings of PPC, research developed Task Anticipated, Task Made Ready (TMR) (Hamzeh et al. 2012), Construction Flow Index (Sacks et al. 2017) and Task Readiness, Readiness Reliability (Lin and Golparvar-Fard 2018) to proactively measure risk and reliability of the plan.

In practice, traditional construction management metrics such as the Schedule Performance Index (SPI) and Cost Performance Index (CPI) from Earned Value Analysis (EVA) is broadly used to monitor progress and cash flow. However, it is still retroactively measuring and predicting progress with mainly deriving the criticality of tasks from cost. This results in undermining the level of effort for the actual progress. In addition, it is often hard to direct the tasks from the schedule to the work packages that are used for cost estimation. Project managers end up estimating the budgeted cost for the EVA.

Currently, the above-mentioned metrics are all used to generate infographics and presented in a form of dashboards and reports. Usually this metrics are tracked in a weekly basis and used to create a trend line to provide the users a glance of the project status overtime. Metrics are also calculated based on task and shown in a weekly work plan format. Several commercial software provides interactive dashboards that provides user better understanding of the project performance. In the CPS for progress monitoring, the dashboards and reports and generated using the data collected on the construction site in real time, and provide direct feedback from the downstream of the production.

5.2.4.3 Daily Construction Reports

General contractors often monitor subcontractors’ performance based on the actual progress that is reported in the daily construction reports (DCRs) . General contractors receive DCRs from each subcontractor daily and summarized it into one internal report to document the overall project status (Fig. 5.9). These reports are the progress summary of the reported day and include information such as weather, subcontractor’s name, trade type, worker’s level of experience, manhour, number of onsite equipment, safety issues and the description and location of work. Currently, these DCRs serve as progress documentation for schedule improvement and records in a legal proceeding. These reports contain information that could support and defend delays and material costs due to change clause, constructive changes, work suspension, sequence changes or disruptions. It keeps a progress document that has been agreed upon by all project stakeholders. Whereas the main purpose currently is to document the progress, the potential of using these data for risk management of schedule, subcontractor billing and cost estimation have not been fully explored. These data are valuable and could be used to improve planning, coordination, and communication.

Fig. 5.9
figure 9

The process of generating DCR to track daily progress on the construction site

Full size image

The DCR generation process is currently supported by commercially available software and general contractor’s in-house software. However, the inefficiency and inaccuracy of the data collection still affect the quality and reliability of the DCRs. CPS could streamline the process from data collection to DCR generation with an autonomous process. With different sources of autonomous data collection, the Reality is captured and generated in the form of visual production model with 4D Reality and BIM, the required DCR progress data are then automatically output in required format with the manhour, equipment, crew balance and related information organized based on work breakdown structure.

5.3 Opportunities, Challenges and Limitations for CPS in Construction Progress Monitoring

With the recent development in applying robotics and computer vision in construction and the exponential growth of visual data collected on the construction site, each component of the CPS for progress monitoring has improved significantly to support the automated process. Through the previous sections, we see substantial achievements have been made over the past decades, yet still, many problems remain as open research challenges. In the following section, the problems of each CPS component are discussed, and possible solutions are introduced.

The autonomous data collection process is still not well developed for construction progress monitoring purposes. The data collection process can be a lengthy but non-trivial work depending on the size of construction. The optimization of the data collection process has not been fully investigated. The current data collection process does not fully utilize the existence of 4D BIM, and focus on navigating and mapping based on the input from Reality. 4D BIM provides the planned changes that should happen in the future and could be used as a priori to optimize the data collection path. On the other note, coordinating multiple vehicles for data collection could be another way to facilitate the collection process of large construction space.

After the data collection, although recent algorithm improvement on image-based and video-based 3D reconstruction has shown promising results, the accuracy and completeness of the point cloud can still be improved on generating consistent good results in different environmental settings. For example, the reconstruction tends to produce poor quality results on reflective surfaces and thinner structures which are commonly seen in construction as curtain walls and steel components. On the other hand, geometry-based progress monitoring only provides binary results of the observed object. With the recent development in deep learning, integrating appearance-based methods that extract colors, texture, shape and semantic information from the 2D image can streamline the automated progress monitoring process. This further brings up the need for a complete construction material database that could be used for progress monitoring. For activity analysis, current research is also limited to the labeled data size. Today’s research on equipment productivity analysis is still limited to a few machines such as dump trucks, crane, loaders, excavators, and workers. The data size needs to be expanded to fully support comprehensive activity analysis with a dynamic and realistic data source. There is also limited research on linking the input (equipment utilization and man-hours) analyzed from activity analysis with the output (progress changes) analyzed geometry and appearance-based method to examine the budgeted productivity rates.

Reporting is rather mature compared to the other three components. However, research has been developing complicated and specialized metrics for specific workflows. Generalized metrics that could apply to different workflows are more practical to implement across projects. Generalized metrics provide a common ground for comparing different projects and obtain more insights. For example, PPC is only applicable for projects that follow lean principles where it is not feasible for the critical path method (CPM) schedule projects.

The following section introduces the latest application of CPS for construction progress monitoring, implementation details and practical feedback from construction practitioners are discussed.

5.4 Case Studies

A complete CPS framework for progress monitoring is implemented and evaluated through a case study using a web-based visual production management system (Lin and Golparvar-Fard 2018). The data collection component of the CPS is performed by drone and rover to gather progress images at different times and locations. The system takes in the images and automatically generate 4D Reality with localized unordered images in the same environment. The 4D Plan model is created in the system by linking the look-ahead schedule to the BIM according to the work breakdown structure. The visual production model that integrates the Reality and Plan is used to compare and analyze the schedule deviation for progress monitoring with productivity input from the site engineers using the mobile application. This visual production model is used during the construction to provide “who does what work in what location” and the state of progress through the color-coded model and Reality model with images. The risk analysis based on the progress and productivity is provided weekly to help project managers better understand the reliability of the plan and tap off potential delays proactively. Daily construction and productivity report are generated automatically to document the progress with verification from Reality. The system leverage the CPS to communicate progress efficiently for project control decisions, and enhance the process of planning, coordination, and planning. (Fig. 5.10)

Fig. 5.10
figure 10

The visual production management system in the CPS has been used on different construction site during the coordination meetings, it has been proved that it can efficiently enhance planning coordination and communication, and the reports provide insights for decision making

Full size image

5.5 Conclusions

We introduced the state-of-the-art research and applications of the four key components -data collection, automated progress monitoring, activity analysis, and reporting- in CPS for construction progress monitoring. The current development of CPS has shown promising results but the automation of each of the components and the integration between each other remains challenging with many open research problems such as optimization for automated data collection, integration of geometry and appearance-based progress monitoring, creating comprehensive datasets in dynamic environments. We provide a case study that illustrates the potential of CPS for construction progress monitoring using visual production models to improve planning, coordination, and communication.