Keywords

1 Introduction

Architectural floorplans play a crucial role in designing, understanding, and remodeling indoor spaces. Their drawings are effective in conveying geometric and semantic information of a scene. For instance, we can quickly identify room extents, the locations of doors, or object arrangements. We can also recognize the types of rooms, doors, or objects easily through texts or icon styles. Unfortunately, more than 90% of houses in North America do not have floorplans. The ultimate goal of indoor mapping research is to enable automatic reconstruction of a floorplan simply by walking through a house with a smartphone.

The Consumer-grade depth sensors have revolutionized indoor 3D scanning with successful products. Matterport [1] produces detailed texture mapped models of indoor spaces by acquiring a set of panorama RGBD images with a specialized hardware. Google Project Tango phones [14] convert RGBD image streams into 3D or 2D models. These systems produce detailed geometry, but fall short as floorplans or architectural blue-prints, whose geometry must be concise and respect underlying scene segmentation and semantics.

Reconstruction of the floorplan for an entire house or an apartment with multiple rooms poses fundamental challenges to existing techniques due to its large 3D extent. A standard approach projects 3D information onto a 2D lateral domain [10], losing the information of height. PointNet [26, 28] consumes 3D information directly but suffers from the lack of local neighborhood structures. A multi-view representation [27, 34] avoids explicit 3D space modeling, but has been mostly demonstrated for objects, rather than large scenes and complex camera motions. 3D Convolutional Neural Networks (CNNs) [29, 36] also show promising results but have been so far limited to objects or small-scale scenes.

This paper proposes a novel deep neural network (DNN) architecture FloorNet, which turns a RGBD video covering a large 3D space into pixel-wise predictions on floorplan geometry and semantics, followed by an existing Integer Programming formulation [17] to recover vector-graphics floorplans. FloorNet consists of three DNN branches. The first branch employs PointNet with 3D points, exploiting the 3D information. The second branch uses a CNN with a 2D point density image in a top-down floorplan view, enhancing the local spatial reasoning. The third branch uses a CNN with RGB images, utilizing the full image information. The PointNet branch and the point-density branch exchange features between the 3D points and their corresponding cells in the top-down view. The image branch contributes deep image features into the corresponding cells in the top-down view. This hybrid DNN design exploits the best of all the architectures and effectively processes the full RGBD video covering a large 3D scene with complex camera motions.

We have created a benchmark for floorplan reconstruction by acquiring RGBD video streams for 155 residential houses or apartments with Google Tango phones and annotated their complete floorplan information including architectural structures, icons, and room types. Extensive qualitative and quantitative evaluates demonstrate the effectiveness of our approach over competing methods.

In summary, the main contributions of this paper are two-fold: (1) Novel hybrid DNN architecture for RGBD videos, which processes the 3D coordinates directly, models local spatial structures in the 2D domain, and incorporates the full image information; and (2) A new floorplan reconstruction benchmark with RGBD videos, where many indoor scene databases exist [1, 4, 33] but none tackles a vector-graphics reconstruction problem, which has immediate impact on digital mapping, real estate, or civil engineering applications.

2 Related Work

We discuss the related work in three domains: indoor scene reconstruction, 3D deep learning, and indoor scan datasets.

Indoor Scene Reconstruction: The advancements in consumer-grade depth sensors have brought revolutionary changes to indoor 3D scanning. KinectFusion [23] enables high-fidelity 3D scanning for objects and small-scale scenes. Whelan et al. [40] extends the work to building-scale scans. While being accurate with details, these dense reconstructions fall short as CAD models, which must have (1) concise geometry for efficient data transmission and (2) proper segmentations/semantics for architectural analysis or effective visualization.

Towards CAD-quality reconstructions, researchers have applied model-based approaches by representing a scene with geometric primitives. Utilizing the 2.5D property of indoor building structures, rooms can be separated by fitting lines to points in a top-down view [25, 35, 37]. Primitive types have been extended to planes [5, 6, 22, 31, 43] or cuboids [42]. While they produce promising results for selected scans, their heuristic-based primitive detection faces challenges with noisy and incomplete 3D data. Our approach conducts global analysis of the entire input by DNNs to detect primitive structures much more robustly.

Another line of research studies the top-down scene reconstruction with shape grammars from a single image [47] or a set of panorama RGBD images [10, 21]. Crowdsensing data such as images and WiFi-fingerprints are also exploited in building scene graphs [7, 8, 12, 19]. While semantic segmentation [4, 26, 28] and scene understanding [45] are popular for indoor scenes, there has been no robust learning-based method for vector-graphics floorplan reconstruction. This paper provides such a method and its benchmark with the ground-truth.

One way to recover the mentioned vector-graphics floorplan models is from rasterized floorplan images [17]. We share the same reconstruction target, and we utilize their Integer Programming formulation in our last step to recover the final floorplan. Nevertheless, instead of a single image as input, our input is a RGBD video covering a large 3D space, which requires a fundamentally different approach to process the input data effectively.

3D Deep Learning: The success of CNN on 2D images has inspired research on 3D feature learning via DNNs. Volumetric CNNs [20, 27, 41] are straightforward extensions of CNN to a 3D domain, but there are two main challenges: (1) data sparsity and (2) computational cost of 3D convolutions. FPNN [15] and Vote3D [38] attempt to solve the first challenge, while OctNet [29] and O-CNN [39] address the computational costs via octree representations.

2D CNNs with multi-view renderings have been successful for object recognition [27, 34] and part segmentation [16]. They effectively utilize all the image information but are so far limited to regular (or fixed) camera arrangements. The extension to larger scenes with complex camera motions is not trivial.

PointNet [26] directly uses 3D point coordinates to exploit the sparsity and avoid quantization errors, but it does not provide an explicit local spatial reasoning. PointNet++ [28] hierarchically groups points and adds spatial structures by farthest point sampling. Kd-Networks [13] similarly group points by a KD-tree. These techniques incur additional computational expenses due to the grouping and have been limited at object-scale. For scenes, they need to split the space into smaller regions (e.g., \(1\,\text {m}\times 1\,\text {m}\) blocks) and process each region independently [26, 28], hurting global reasoning (e.g., identifying long walls for corridors).

Indoor Scan Dataset: Affordable depth sensing hardware enables researchers to build many indoor scan datasets. The ETH3D dataset contains only 16 indoor scans [30], and its purpose is for multi-view stereo rather than 3D point-cloud processing. The ScanNet dataset [4] and the SceneNN dataset [9] capture a variety of indoor scenes. However, most of their scans contain only one or two rooms, not suitable for the floorplan reconstruction problem.

Matterport3D [3] builds high quality panorama RGBD image sets for 90 luxurious houses. 2D-3D-S dataset [2] provides 6 large-scale indoor scans of office spaces by using the same Matterport camera. However, they focus on 2D and 3D semantic annotations, and do not address a vector-graphics reconstruction problem. Meanwhile, they require an expensive specialized hardware (i.e., Matterport camera) for high-fidelity 3D scanning, while we aim to tackle the challenge by consumer-grade smartphones with low data quality.

Lastly, a large-scale synthetic dataset, SUNCG [32], offers a variety of indoor scenes with CAD-quality geometry and annotations. However, they are synthetic and cannot model the complexity of real scenes or replace the real photographs. We provide the benchmark with full floorplan annotations and the corresponding RGBD videos from smartphones for 155 residential units.

3 FloorNet

The proposed FloorNet converts a RGBD video with camera poses into pixel-wise floorplan geometry and semantics information, which is an intermediate floorplan representation introduced by Liu et al. [17]. We first explain the intermediate representation for being self-contained, then provide the details.

3.1 Preliminaries

The intermediate representation consists of the geometry and the semantics information. The geometry part contains room-corners, object icon-corners, and door/window end-points, where the locations of each corner/point type are estimated by a \(256\times 256\) heatmap in the 2D floorplan image domain, followed by a standard non-maximum suppression. For example, a room corner is either I-, L-, T-, or X-shaped depending on the number of incident walls, making the total number of feature maps to be 13 considering their rotational variants. The semantics part is modeled as (1) 12 feature maps as a probability distribution function (PDF) over 12 room types, and (2) 8 feature maps as a PDF over 8 icon types. We follow their approach and use their Integer Programming formulation to reconstruct a floorplan from this representation at the end.

3.2 Triple-Branch Hybrid Design

Floornet consists of three DNN branches. We employ existing DNN architectures in each branch without modifications. Our contribution lies in its hybrid design: how to combine them and share intermediate features (See Fig. 1).

Fig. 1.
figure 1

FloorNet consists of three DNN branches. The first branch uses PointNet [26] to directly consume 3D information. The second branch takes a top-down point density image in a floorplan domain with a fully convolutional network [18], and produces pixel-wise geometry and semantics information. The third branch produces deep image features by a dilated residual network trained on the semantic segmentation task [44] as well as a stacked hourglass CNN trained on the room layout estimation [24]. The PointNet branch and the floorplan branch exchanges intermediate features at every layer, while the image branch contributes deep image features into the decoding part of the floorplan branch. This hybrid DNN architecture effectively processes an input RGBD video with camera poses, covering a large 3D space.

Full size image

PointNet Branch: The first branch is PointNet [26] with the original architecture except that each 3D point is represented by XYZ and RGB values without the normalized position. We randomly subsample 50,000 points for each data. We manually rectify the rotation and align the gravity direction with the Z-axis. We add translation to move the center of mass to the origin.

Floorplan Branch: The second branch is a fully convolutional network (FCN) [18] with skip connections between the encoder and the decoder, which takes a point-density image with RGB values in the top-down view. The RGB value in each cell is computed as the average over the 3D points. We compute a 2D axis-aligned bounding box of the Manhattan-rectified 3D points to define a rectangular floorplan domain, while ignoring the 2.5% outlier points and expanding the rectangle by 5% in each of the four directions. The rectangle is placed in the middle of the \(256\times 256\) square image in which the geometry and semantics feature maps are produced. The input to the branch is a point-density image in the same domain.

Image Branch: The third branch computes deep image features through two CNN architectures: (1) Dilated residual network (DRN) [44] trained on semantic segmentation using the ScanNet dataset [4]; and (2) stacked hourglass CNN (SH) [24] trained on room layout estimation using the LSUN dataset [46].

3.3 Inter-branch Feature Sharing

Different branches learn features in different domains (3D points, the floorplan, and images). Figure 2 shows three inter-branch feature sharing by pooling and unpooling operations, based on the camera poses and 3D information.

Fig. 2.
figure 2

FloorNet shares features across branches to exploit the best of all the architectures. PointNet features at 3D points are pooled into corresponding 2D cells in the floorplan branch. Floorplan features at 2D cells are unpooled to the corresponding 3D points in the PointNet branch. Deep image features are pooled into corresponding 2D cells in the floorplan branch, based on the depthmap and the camera pose information.

Full size image

PointNet to Floorplan Pooling: This pooling module takes features of unordered points from each layer of the PointNet branch and produces a 2D top-down feature map in the corresponding layer of the floorplan branch. The module simply projects 3D point features into cells in the floorplan feature map, then aggregates the features in each cell by either summation or maximum operation. We use the summation operation in the first three convolution layers to keep more information while taking the maximum in the rest layers to introduce competition. A constructed feature map has the same dimension as the feature map in the floorplan branch. We have explored several different aggregation schemes between the feature map from the pointnet branch and the feature map at the floorplan branch. We found that a sum-pooling (i.e., element-wise addition) works the best. The time complexity of the projection pooling module is linear in the number of 3D points.

Floorplan to PointNet Unpooling: This module reverses the above pooling operation. It simply copies and adds a feature of the floorplan cell into each of the corresponding 3D points that project inside the cell. The time complexity is again linear in the number of points.

Table 1. Dataset statistics. From left to right: the number of rooms, the number of icons, the number of openings (i.e., doors or windows), the number of room-corners, and the total area. The average and the standard deviation are reported for each entry.
Full size table

Image to Floorplan Pooling: The image branch produces two deep image features of dimensions \(512\times 32\times 32\) and \(256\times 64\times 64\) from DRN and SH for each video frame. We first unproject image features to 3D points by their depthmaps and camera poses, then apply the same 3D to floorplan pooling above. One modification is that we use max-pooling at all the layers in projecting 3D points onto the floorplan domain to be simple, instead of the mix of sum and max poolings. The reason is that we use pre-trained models for image feature encoding, and more complex hybrid pooling would have less effects. We conduct the image branch pooling for every 10 frames in the video sequence.

3.4 Loss Functions

Our network outputs pixel-wise predictions on the floorplan geometry and semantics information in the same resolution \(256\times 256\). For geometry heatmaps (i.e., room corners, object icon-corners, and door/window end-points), a sigmoid cross entropy loss is used. The ground-truth heatmap is prepared by putting a value of 1.0 inside a disk of radius 11 pixels around each ground-truth pixel. For semantic classification feature maps (i.e., room types and object icon types), a pixel-wise softmax cross entropy loss is used.

4 Floorplan Reconstruction Benchmark

This paper creates a benchmark for the vector-graphics floorplan reconstruction problem from RGBD videos with camera poses. We have acquired roughly two-hundreds 3D scans of residential units in the United States and China using Google Tango phones (Lenovo Phab 2 Pro and Asus ZenFone AR) (See Fig. 3). After manually removing poor quality scans, we have annotated the complete floorplan information for the remaining 155 scans: (1) room-corners as points, (2) walls as pairs of room-corners, (3) object icons and types as axis-aligned rectangles and classification labels, 4) doors and windows (i.e., openings) as line-segments on walls, and 5) room types as classification labels for polygonal areas enclosed by walls. The list of object types is {counter, bathtub, toilet, sink, sofa, cabinet, bed, table, refrigerator}. The list of room types is {living room, kitchen, bedroom, bathroom, closet, balcony, corridor, dining room}. Table 1 provides statistics of our data collections.

Fig. 3.
figure 3

Floorplan reconstruction benchmark. From left to right: subsampled video frames, colored 3D point clouds, and ground-truth floorplan data. The floorplan data is stored in a vector-graphics representation, which is visualized with a simple rendering engine (e.g., rooms are assigned different colors based on their types, and objects are shown as canonical icons). (Color figure online)

Full size image

Reconstructed floorplans are evaluated on three different levels of geometric and semantic consistency with the ground-truth. We follow the work by Liu et al. [17] and define the low- and mid-level metrics as follows.

  • The low-level metric is the precision and recall of room corner detections. A corner detection is declared a success if its distance to the ground-truth is below 10 pixels and the closest among all the other room corners.

  • The mid-level metric is the precision and recall of detected openings (i.e., doors and windows), object-icons, and rooms. The detection of an opening is declared a success if the largest distance of the corresponding end-points is less than 10 pixels. The detection of an object (resp. a room) is declared a success if the intersection-over-union (IOU) with the ground-truth is above 0.5 (resp. 0.7).

  • Relationships of architectural components play crucial roles in evaluating indoor spaces. For example, one may look for apartments where bedrooms are not connected to a kitchen. A building code may enforce every bedroom to have a quick evacuation route to outside through windows or doors in case of fire. We introduce the high-level metric as the ratio of rooms that have the correct relationships with the neighboring rooms. More precisely, we declare that a room has a correct relationship if (1) it is connected to the correct set of rooms through doors, where two rooms are connected if their common walls contain at least one door, (2) the room has an IOU score larger than 0.5 with the corresponding ground-truth, and (3) the room has the correct room type.

5 Implementation Details

5.1 DNN Training

Among the 155 scans we collected, we randomly sample 135 for training and leave 20 for testing. We perform data augmentation by random scaling and rotation every time we feed a training sample. First, we apply rescaling to the point-cloud and the annotation with a random factor uniformly sampled from a range [0.5, 1.5]. Second, we randomly apply the rotation around the z axis by either \(0^{\circ }, 90^{\circ }, 180^{\circ }\), or \(270^{\circ }\).

We have utilized the official code for the two image encoders DRN [44] and SH [24]. We pre-trained DRN on the semantic segmentation task with ScanNet database [4] and SH on the room layout estimation task with LSUN [46]. DRN and SH are fixed during the FloorNet training. We implemented the remaining DNN modules by ourselves in TensorFlow with the modern APIs, that is, PointNet [26] for the Pointnet branch and FCN [18] for the Floorplan branch.

Training of FloorNet takes around 2 h with a TitanX GPU. We set the batch size to 6. FloorNet has three types of loss functions. To avoid overfitting with the icon loss, we trained icon-loss separately for at most 600 epochs with early-stopping based on the testing loss. The others losses are trained jointly for 600 epochs.Footnote 1 The training consumes \(81,000 = 135 (\text {samples})\times 600 (\text {epochs})\) augmented training samples. It is initially to our surprise that FloorNet generalizes even from a small number of 3D scans. However, FloorNet makes pixel-wise low-level predictions. Each 3D scan contains about 10 object-icons, 10 openings, and a few dozen room corners, which probably lead to the good generalization performance together with data augmentation, where similar phenomena were observed by Liu et al. [17].

5.2 Enhancement Heuristics

We augment the Integer Programming Formulation [17] with the following two enhancement heuristics to deal with more challenging input data (i.e., large-scale raw sensor data) and hence more noise in the network predictions.

Primitive Candidate Generation: Standard non-maximum suppression often detects multiple room corners around a single ground-truth. After thresholding the room-corner heatmap by a value 0.5, we simply extract the highest peak from each connected component, whose area is more than 5 pixels. To handle localization errors, we connect two room-corners and generate a wall candidate when their corresponding connected components overlap along X or Y direction. We do not augment junctions to keep the number of candidates tractable.

Objective Function: Wall and opening candidates are originally assigned uniform weights in the objective function [17]. We calculate the confidence of a wall (resp. opening) candidate by taking the average of the semantic heatmap scores of type “wall” along the line with width 7 pixels (resp. 5 pixels). We set the weight of each primitive by the confidence score minus 0.5, so that a primitive is encouraged to be chosen only when the confidence is at least 0.5.

6 Experiments

Figure 4 shows our reconstruction results on some representative examples. Our approach successfully recovers complex vector-graphics floorplan data including room geometries and their connectivities through doors. One of the major failure modes is in the icon detection, as object detection generally requires more training data than low-level geometry detection [17]. We believe that more training data will overcome this issue. Another typical failures come from missing room corners due to clutter or incomplete scanning. The successful reconstruction of a room requires successful detection of every room corner. This is a challenging problem and the introduction of higher level constraints may reveal a solution (Fig. 5).

Fig. 4.
figure 4

Floorplan reconstruction results.

Full size image
Fig. 5.
figure 5

Intermediate results. For each example, we show raw outputs of the networks (room corners, icon corners, opening corners, and icon types) compared against the ground-truth. In the second example, we produce a fake room (blue color) at the top due to poor quality 3D points. In the third example, reconstructed rooms have inaccurate shapes near the bottom left again due to noisy 3D points, illustrating the challenge of our problem. (Color figure online)

Full size image
Fig. 6.
figure 6

Qualitative comparisons against competing methods. The top is OctNet [29], a state-of-the-art 3D CNN architecture. The next three rows show variants of our FloorNet, where only one branch is enabled. FloorNet with all the branches overall produce more complete and accurate floorplans.

Full size image
Table 2. Quantitative evaluations on low-, mid-, and high-level metrics against competing methods and our variants. The orange and cyan color indicates the best and the second best result for each entry.
Full size table

Figure 6 and Table 2 qualitatively and quantitatively compare our method against competing techniques, namely, OctNet [29], PointNet [26], and a few variants of our FloorNet. OctNet and PointNet represent state-of-the-art 3D DNNs. More precisely, we implement the voxel semantic segmentation network based on the official OctNet library,Footnote 2 which takes \(256\times 256\times 256\) voxels as input and outputs 3D voxels of the same resolution. We then add three separate \(5\times 3 \times 3\) convolution layers with strides \(4\times 1\times 1\) to predict the same pixel-wise geometry and semantics feature-maps with the same set of loss functions. PointNet is simply our FloorNet without the point density or the image input. Similarly, we construct a FloorNet variant by enabling only the 3D points (for the PointNet branch) or the point density image (for the floorplan branch) as the input.

The table shows that the floorplan branch is the most informative as it is the most natural representation for floorplan reconstruction task, while PointNet branch or Image branch alone does not work well. We also split the entire point clouds into \(1\,\text {m}\times 1\,\text {m}\) blocks, train the PointNet-only model that makes predictions per block separately, followed by a simple merging. However, this performs much worse. OctNet performs reasonably well across low- to mid-level metrics, but does poorly on the high-level metric, where all the rooms and relevant doors must be reconstructed at high precision to report good numbers.

To further evaluate the effectiveness of the proposed FloorNet architecture, we conduct ablation studies by disabling each of the inter-branch pooling/unpooling operations. The bottom of Table 2 shows that the feature sharing overall leads to better results, especially for mid- to high-level metrics.

Table 3 compares different inter-branch pooling/unpooling schemes for the PointNet to floorplan pooling. The table shows that the max operation in early layers loses too much information and leads to worse performance.

Finally, Fig. 7 compares against a build-in Tango Navigator App [11], which generates a floorplan image real-time on the phone. Note that their system does not (1) produce room segmentations, (2) recognize room types, (3) detect objects, (4) recognize object types, or (5) produce CAD-quality geometry. Therefore, we quantitatively evaluate only the geometry information by measuring the line distances between the ground-truth walls and predicted walls. More precisely, we (1) sample 100 points from each wall line segment, (2) for each sampled point, find the closest one in the other line segment, and (3) compute the mean distance over all the sampled points and line segments. The average line distances are 2.72 [pixels] and 1.66 [pixels] for Tango Navigator App and our FloorNet, respectively. This is a surprising result, because our algorithm drops many confident line segments during Integer Programming, when the corresponding room is not reconstructed. On the other hand, it is an expected result as our approach utilizes all the geometry and image information.

Table 3. In the PointNet to floorplan inter-branch pooling, we use a mix of sum and max pooling in projecting 3D points onto the 2D floorplan domain. To validate this hybrid scheme, we have also evaluated the performance when only the max-pooling or the sum-pooling is used at all the layers.
Full size table
Fig. 7.
figure 7

Comparison against a commercial floorplan generator, Tango Navigator App. Top: floorplan image from Tango. Bottom: our results.

Full size image

7 Conclusion

This paper proposes a novel DNN architecture FloorNet that reconstructs vector-graphics floorplans from RGBD videos with camera poses. FloorNet takes a hybrid approach and exploits the best of three DNN architectures to effectively process a RGBD video covering a large 3D space with complex camera motions. The paper also provides a new benchmark for a new vector-graphics reconstruction problem, which is missing in the recent indoor scene databases of Computer Vision. Two main future works are ahead of us. The first one is to learn to enforce higher level constraints inside DNNs as opposed to inside a separate post-processing (e.g., Integer Programming). Learning high-level constraints likely require more training data and the second future work is to acquire more scans.

More than 90% of houses in North America do not have floorplans. We hope that this paper together with the benchmark will be an important step towards solving this challenging vector-graphics reconstruction problem, and enabling the reconstruction of a floorplan just by walking through a house with a smartphone. We publicly share our code and data to promote further research.