Keywords

1 Introduction

With the introduction of robotic tools into architectural fabrication processes, computational designs can be manifested into the physical world more easily than before. An example of this can be seen in the Gantenbein vineyard [1]. However, robotic systems also have their shortcomings. Since every construction site and building project is different, it is more challenging to calibrate robots to suit new construction sites and building materials compared to setting up a production line for a repetitive task [2]. Additionally, complex sensing systems (e.g., with reinforcement learning [3]) and tighter tolerances are needed to work with materials possessing non-standard geometries, such as natural elements (e.g., bamboo rods), or building blocks with manufacturing inconsistencies. Thus, it is essential to consider the transferability of robotic assembly systems across different projects involving distinct environments, building materials, and robot models [4].

In addition to the technical challenges faced by robotic construction systems, there has also been a growing trend for segregation and specialization between design and construction disciplines. In most cases, construction only begins after the design is finalized. Even with the development of notions such as building information modeling (BIM) [5] and robot-oriented design (ROD) [6], the role of robots in today’s construction techniques is mainly categorized as a passive element of fabrication rather than an active element integral to the design process.

The relationship between humans and robots in construction settings is of concern as well. We ask: How can humans, both as designers and construction workers, best collaborate with robots so that the strengths of both sides can be amplified? How can humans’ design-construction experience be altered and improved by the introduction of robotic tools? The emerging field of collective human-robot construction (CHRC) [7] has begun to explore these questions by pointing researchers towards a wide range of possibilities for human-robot team creativity. It is primarily concerned with investigations and explorations into how design decisions can be distributed across robotic and human agents in order to enhance the collective performance.

In this work, we introduce the concept of improvisation to the human-robot design-construction process. Improv-Structure, a 7′ x 14′ x 7′ bamboo structure consisting of ∼500 4’ long and 3/8″ wide bamboo rods, was designed and constructed by two ABB IRB 4600–255/40 robots and several humans over the span of 5 days. The robotic arms functioned as guidance and structural support, while the humans led the design and construction process. This led to an immersive and improvisational experience for the human builders that was profoundly different from the cut-and-dry experience typically encountered when building a structure from a pre-determined design. Since no planning occurred and no expectations were made with regard to how the final structure would look, collective design decisions were made by humans based on observing the built portion of the structure. Throughout the process, the robot received its input parameters based on LiDAR scans of the existing structure into the 3D computer model. Improv-Structure serves as a proof of concept for improvisational construction with an immersive design process in the CHRC setting.

2 State of the Art

2.1 Improvisation and Robotics

Improvisation refers to actions (e.g., art performance, emergency response) made without advance planning. This method is often adopted in musical and theatrical settings to nurture group creativity [8, 9], in corporation management to enhance team performance [10], and in emergency scenarios to maximize the effectiveness of decision-making under time pressure [11, 12]. In recent years, the notion of improvisation has been introduced into the context of robotics, for example in robotic teams that improvise jazz music with human musicians [8, 13] and human-robot improvisational dance [14, 15]. However, the concept of improvisation is still very new in the design-construction field.

Improvisational skills can be divided into two categories - open and closed skills [16]. According to Jeff Pressing’s article in 1988, open skills “require extensive interaction with external stimuli”; meanwhile, closed skills “[rely] only on self-produced stimuli” [16]. In a heterogeneous human-robot team, we expect humans, robots, and the built environment to act as external stimuli for each other. Inherent qualities such as intuition and stylistic preferences will be key aspects of the agents’ closed skills. We expect improvisation to enhance creativity and construction efficiency as shown in research from adjacent fields (e.g. [8, 10, 12]).

2.2 Segregation Between Design and Construction

Since the industrial revolution, design and construction have become increasingly segregated and specialized. Nevertheless, a harmonious collaboration between these disciplines (i.e. architects and engineers) is essential for the development of quality structures [17]. The introduction of robotic tools has provided a means to bring complex parametric designs into the physical world. However, most robotic fabrication processes in the building industry regard computational design and robotic construction as two distinguished steps. In other words, despite the fact that computational design allows for a large number of quick iterations before a design is finalized, we expect robots to follow pre-planned assembly steps to achieve a pre-determined geometry once the construction phase begins, as seen for example in the Gantenbein vineyard [1].

In order to better integrate design and construction processes, several solutions have been proposed and implemented. From the industrial point of view, BIM [5] intends to more effectively connect designers with relevant construction disciplines through consistent data gathering and representation. Similarly, ROD [6] emphasizes the consideration of robotic parameters when designing robotic construction processes. Examples such as the ICD/ITKE Research Pavilion 2016/2017 [18] and the LightVault [19] have also shown the importance of considering robotic kinematics and workspace when designing robotic fabrication processes. Even with BIM and ROD, however, design and fabrication processes are still segregated in the majority of construction projects.

2.3 Human-Robot Interaction and Immersive/Participatory Design Using Robots

Recent developments in human-robot interaction (HRI) and immersive design with augmented and virtual reality (AR/VR) have brought forth new possibilities for integrating design and fabrication processes. Evidence shown by Paes et al. has proven the cognitive benefits of immersive design with VR for 3D perception and presence [20]. Additionally, he adoption of HRI and HCI (human-computer interaction) has enabled co-design through fast and cheap physical prototypes [21, 22]. Despite these obvious advantages, there are few examples of co-designing and co-constructing architectural-scale structures in real-life immersive design settings that are not enabled by virtual or augmented reality.

A key question faced in participatory design involving high tech is whether the technology itself imposes another layer of segregation and bias. For instance, how can we involve people without a background in robotics (i.e., community members) into a participatory design process that uses robotic arms? In the Improv-Structure project, we aim to open up new channels for design decision making so that they are less centralized and have lower technical barriers.

2.4 Collective Human-Robot Construction (CHRC)

Collective Human-Robot Construction (CHRC) “concerns multi-agent construction involving both human and robotic collectives. It is an emerging interdisciplinary field that combines collective fabrication, human–robot interaction, and heterogeneous teams. Research focused on CHRC spans from autonomy to collaboration, indicating novel ways of designing and fabricating.” [7] Building on top of cooperative robotic assembly, where multiple robots can achieve complex structural compositions by alternating between placing new elements and holding existing structures [23], CHRC brings humans into the loop for enhanced design-construction experiences, creative formal expressions, and building efficiency.

3 Methodology

In the project Improv-Structure, we introduced the concept of improvisation into construction through the design-construction process of a bamboo structure carried out by two ABB IRB 4600–255/40 robots and several humans. As described in Sect. 2.1, we took advantage of robots’ closed skills in strength and precision, as well as humans’ closed skills in sensing and flexibility. By creating feedback channels using a LiDAR scanner (for robots) and real-life observations (for humans), we triggered agents’ open skills to improvise based on external stimuli. Specifically, we built a human-scale structure made of 4 feet long bamboo rods with an average diameter of 3/8″ and connected with zip tie knots by alternating the placement of rods between robots and humans. Our two main goals were to 1) combine the strengths of robots and humans and 2) distribute design decision-making through the proposed improvisational building framework in CHRC.

3.1 Combining the Strengths of Robots and Humans

The distribution of design-construction roles across agents was based on each agent’s strengths. To build the Improv-Structure, the two robotic arms were responsible for placing guiding rods, which provided temporary structural support and enhanced the alignment between the physical construction and the computational design intent. Specifically, in this prototype, the pre-defined Grasshopper [24] algorithm took in 1 to 2 curve geometries each time and generated the guiding rods’ location for the robotic arms accordingly. The tunable factors included the distance between adjacent guiding rods (d in Fig. 1) and the rods’ rotation angles around the curve input, mainly within the YZ plane (see Fig. 1). Orientating the guiding rods around the YZ plane was a simple way to ensure that they were not connected to each other. In other words, a new guiding rod was held in mid-air, detached from the main structure. Here, d was bounded within a range of 0.65∼0.85 times the rod’s length, which the authors observed to be a good gap size to catalyze humans’ creativity for finding bridging solutions while the robotic arms provided efficient temporary point support.

Fig. 1.
figure 1

Guiding rods marked in red color; world X-Y-Z axis defined at the bottom right corner; d = gap size between guiding rods.

Full size image
Fig. 2.
figure 2

LiDAR scanning of existing structure into 3D models to inform robotic movements

Full size image

The human’s role was to 1) connect the guiding rods (marked in red in Fig. 1) held by the robots into the existing structure, 2) make design decisions on the fly based on the immersive physical experience in and around the partially-built structure, and 3) translate that design into 1 to 2 curves that help define the guiding rods based on the LiDAR model. This way, we utilized the robot’s precision and strength without worrying about the complex sensing and tolerance problems caused by organic building elements (i.e., bamboo rods). Similarly, humans were freed from the highly specialized position of either designer or constructor and formed a co-design co-fabrication relationship with the robotic arms.

For communication between the virtual and physical world, the robots received information about the on-going building process through LiDAR scanning (see Fig. 2). Compared to AR/VR, this allowed humans to observe and experience the 3D structure in real life to inform future design decisions (see Fig. 3).

It is worth mentioning that the mechanism presented here is just one version of how design decisions can be distributed between robots and humans. One could shift the level of autonomy between humans and robots towards either direction in future iterations.

Fig. 3.
figure 3

Designing on the fly by observing the built proportion

Full size image
Fig. 4.
figure 4

Robotic arm (right) holding guiding rods in mid-air

Full size image

3.2 Distributing Design Decision-Making

The design-construction process was divided into multiple action units to distribute decision-making across time. Each action unit consisted of the following steps:

  1. 1.

    Robots LiDAR scan the existing structure

  2. 2.

    Humans observe and experience the existing structure and discuss what the next design features could be. Designers need not be trained in robotics to carry out this step

  3. 3.

    Humans input parameters (i.e., 1 to 2 curves) needed for robotic movements base on the LiDAR model

  4. 4.

    Robots place the next guiding rod in mid-air next to the existing structure (see Fig. 4)

  5. 5.

    Humans propose structural solutions on-the-fly to extend the existing structure towards the guiding rod held by the robot

The execution of the design-construction action units is flexible. Based on the humans’ observations on-site, it is possible to quit the action unit mid-way and change plans on the fly. The design-construction action unit is repeated until the humans decide to stop the construction, assuming the structure can stand alone without external support. In other words, it is arbitrary whether the construction is finished or not. One can always restart the building process and continue adding to the structure.

4 Results

The final product of the Improv-Structure is 7′x14′x7′ in dimension and was constructed within a timespan of five days. It consists of around 500 bamboo rods that are 4’ in length and 3/8″ in diameter, of which ∼30 guiding rods were inserted and temporarily supported by the ABB IRB 4600–255/40 robotic arms.

The design construction process was divided into 5 design-construction action units, between which the existing structure was re-scanned by LiDAR sensor and updated in the 3D Rhino/Grasshopper model. New design decisions and adjustments were made between each action unit. Two out of five action units experienced change-of-plans that were influenced by the human’s observation of the constructed portion of the structure. Improv-Structure provides a proof-of-concept example for using improvisation as a framework for CHRC.

5 Discussion, Limitation, and Outlook

5.1 Discussion

Flexibility and Transferability. The improvisational construction in CHRC combines the strength of robot and human agents and considerably reduces the problem of tolerance. Because the robots are only placing and supporting geometry-defining elements in mid-air next to the structure, we replace the need for a complex robotic sensing system with craft from human designers/constructors. This also eliminates concerns related to material inconsistency due to manufacturing defects or organic geometries (i.e., bamboo rods). Thus, the method for improv-structure is highly flexible and transferable.

Fig. 5.
figure 5

Improv-structure

Full size image

One can argue that replacing the mechanically challenging proportion of the task with human craft is not a permanent or automated solution. As a response, we would like to clarify that the essence of our proposal is to take advantage of a heterogeneous team composition and allocate the tasks in a way that triggers collaboration and maximizes the strengths of the agents. Accordingly, the role of human agents in the Improv-Structure project could theoretically be replaced in the future by robots that specialize in sensing the local environment and connecting material elements. In other words, the improvisational construction framework is not only designed to be applicable to human-robot heterogeneous teams, but also to teams with multiple types of robots. However, going back to the topics mentioned in Sect. 2.2 and 2.3, we may also want to include more humans (e.g., designers, engineers, community members without robotics backgrounds, etc.) into a participatory design-construction process. In such scenarios, a completely automated design-fabrication method may not be desirable. While robots are taking over more and more design-construction tasks in the building industry, it is important to remain mindful of how we would like to leverage human intelligence and creativity as well.

Reducing the Segregation Between Design and Construction.

Improv-Structure provides a unique design-construction experience That’s not comparable by immersive AR/VR or non-immersive CAD modeling or rendering. Designers and builders can physically interact with the built proportion, observe the structure from different angles in real life, and imagine the following design steps according to the full-scale structure.

This design-construction model is not only immersive but also participatory. One doesn’t need a robotics background to be able to play a part in crafting the structure. Thus, the shape of the structure emerges throughout time based on the dynamic decision-making among multiple agents. To illustrate this point, multiple humans participated in proposing the potential following design features in Improv-Structure. It is worth noticing that, for people with expertise in both design and construction, the same improvisational model only requires a minimum of one human and two robots to finish similar tasks. Thus, the Improv-Structure design-construction method encourages a more collaborative and interdisciplinary building process.

Fig. 6.
figure 6

Detailed view of Improv-Structure

Full size image

5.2 Limitations

Even though the robotic arms acquired data necessary for motion planning based on existing structures’ spatial parameters from LiDAR scanning, the robots could have had more agency in deciding what the structure would look like.

Additionally, although it is easier to design/build on the fly in full scale, the cost of corrections is higher. For example, it is much more time- and labor-efficient to prototype intensively using simulations and computer models. Once a construction is completed in full size in the physical world, it is more difficult to erase or redo a part to correct mistakes, not to mention that some material processing systems are non-reversible. However, such an improvisational approach can reduce the overall time and cost by shortening the design-construction period for building processes that are well-studied and easily disassemblable.

5.3 Outlook

In the future, the following aspects of Improv-Structure can be further developed:

1) enhanced agency for robots, 2) heterogeneous team compositions, 3) decision-making mechanisms, 4) tunable levels of autonomy, and 5) design-construction experiences.

Firstly, more agency can be given to robotic arms by developing robotic control systems to respond autonomously to the LiDAR scanning model and human inputs. Secondly, human constructors with different craft styles and robots with varied specialization (e.g., securing joints, transporting materials, etc.) can be invited to the building of future versions of Improv-Structure to explore how different compositions of heterogeneous teams can influence the improvisation process and the final product. Thirdly, how exactly design decisions are made can be further explored. For example, one may use machine learning to train “design intuitions” for robotic agents. On another note, a library of spatial features (e.g., seats, spanning shell, planter, tables, etc.) can be used to offer a number of pre-defined design choices and further accelerate the decision-making process. Fourthly, Improv-Structure is only one version of how design-construction tasks can be distributed among multiple agents. In the future, one may look at all agents’ autonomy as a tunable dial and adjust the levels of autonomy to suit the needs of varied construction tasks. For example, one may tune down the robots’ autonomy to achieve a structure closer to a desired end result or tune up the autonomy for a more unexpected or creative design. Yet another potential extension to this project can be to use augmented reality to assist humans in better imagining and visualizing design sketches in a hybrid environment. In this scenario, further efforts can be put into creating a more intuitive user interface and experience tailored for spatial design and human-robot collaboration.

Improv-Structure is a proof of concept to bring improvisation into construction with a heterogeneous team. We can imagine this framework being applied to the building of community sculptures or urban furniture to enhance the sense of belonging and collective identities. In an industrial setting, improvisational construction can potentially improve the efficiency of the design-build cycle by compressing the design and construction phases into one. Methods for creating a more diverse and tunable human-robot team composition for new design- construction experiences are yet to be explored.

6 Image Credits

Figures 5 and 6: Michelle Deng

All other photos and diagrams are by the authors.