Toward Mobile Monitoring of Cargo Compartment Using 3D Sensors for Real-Time Routing

Abstract

In this paper we present a system that monitors cargo compartments in transport vehicles in near real-time using RGB-D sensors (Asus Xtion Pro Live, Microsoft Kinect). The main component of this sensor system is a 3D analysis module that determines relevant logistical parameters of cargo compartment conditions such as free load meters, free cargo space and volume of cargo. Wireless GPRS technology transmits the results data to a central server along with GPS positioning. The latest information on a vehicle’s location and its cargo compartment parameters are always available to dispatchers, thus enabling them to route vehicles flexibly based on incoming transportation orders. The mobile sensor system’s easy and fast integration in the process make it extremely practicable. We also developed a calibration method based on simple planar markers.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The quality and quantity of the global flow of freight the shipping industry has to handle is increasing as the time for shipments decreases. Modern information technologies allow customers to access a range of goods quickly and directly. Real-time fleet scheduling and routing are therefore essential requirements for the efficient transportation of goods to customers. Earlier approaches to fleet scheduling and routing (Bodin et al. 1983) are less than optimal because they use a previously established quantity of freight as the basis for calculation. The aforementioned approaches cannot be applied to combined pickup and delivery traffic as order cycles are growing shorter and the diversity of goods is increasing. They require methods of fleet scheduling and routing using real-time information (Yang et al. 1999). Such solutions rely on the existence of relevant real-time information on available cargo capacity. At present, real-time information for pickup and delivery traffic is only available during order processing. For reasons of cost, direct measurements of cargo are unavailable during loading. The inaccuracy of the underlying data eventually leads to uneconomic runs that fail to fully utilize cargo capacity. Another problem is that statistics on cargo capacity utilization only reference measurements of weight. These measurements indicate that average utilization rates are relatively low (Seidel et al. 2013). A technical solution that automatically measures the volume of cargo in a truck to systematically quantify cargo capacity utilization does not exist yet. Neugebauer et al. (2010) introduced a system and technique to directly measure the operating rate of transportation vehicles. It monitors the cargo compartments in transportation vehicles with ultrasonic sensors. Reflection and interference among the individual ultrasonic sensors can cause low resolution and strong noise, however. A further problem is the measurement’s strong dependence on the cargo since potentially reflecting materials may be in it. The use of 3D image sensors can remedy these problems. In this paper, we propose a system for automatic real-time measurement of cargo compartments based on a particular type of 3D sensor, namely RGB-D sensors (Asus Xtion Pro Live, Microsoft Kinect). RGB-D sensors make it possible to take very close measurements and to detect small and slightly textured structures all over a large cargo compartment. Our proposed system is a refinement of an earlier prototype of ours (Voigt et al. 2011), which has been enhanced significantly, e.g. with an easy calibration procedure, real (metric) measurement based on point cloud processing and the option of configuring multiple sensors to monitor larger cargo compartments.

The paper is structured as follows. Section 2 presents the entire system. Section 3 describes the calibration method, Sect. 4 the analysis methods. Section 5 presents our evaluation concept. Section 6 is a short conclusion.

2 System Overview

The proposed approach to automatic cargo capacity monitoring consists of four components (Fig. 1). The RGB-D sensor module and the communication module are installed in a transport vehicle in an integrated unit termed a “mobile sensor unit”. The localization server and web clients, used for operational support of the end user (e.g. the dispatcher), form the database unit.

Fig. 1

Components of the complete system

The parameters of the cargo compartment are identified by the mobile sensor unit and transmitted to the localization server by mobile radio communication, i.e. GPRS. The localization server provides an information interface on loading status to web client applications used by dispatchers such as cargo monitoring systems. The cargo compartment parameters are: loading volume, cargo space and load meters. In combination with positioning data (GPS coordinates), they are important criteria for fleet management and vehicle routing. The parameter of structural change represents intolerable changes in the cargo’s structure while en route, and thus facilitates the securing of cargo. The occupancy matrix is a compact 3D model reconstruction of loaded cargo that enables a dispatcher to monitor it manually. In addition, a color image containing the state of the cargo compartment is transmitted to the localization server for general process documentation.

Based on information generated by the proposed system, the dispatcher can identify vehicles with available cargo capacity matching the requirements of incoming transportation orders. The dispatcher can flexibly reroute vehicles based on their current location and remaining transport orders. This form of ad hoc logistics (Poenicke et al. 2010) holds potential to make shipping more cost effective and greener by facilitating greater utilization of cargo capacities.

3 Methods of Calibration

One main challenge to the practicability and acceptance of the mobile sensor system is finding a simple calibration method that untrained operators can use in the application domain. In this section, we explain the calibration method used in the proposed approach.

The calibration of imaging sensor systems is far more elaborate than the calibration of other sensor systems (e.g. ultrasonic sensors) because a variety of parameters have to be determined (see Table 1).

Table 1 Parameters necessary for sensor system calibration (organized in groups)

The literature provides a variety of methods for, for instance, calibration of depth sensors (Yamazoe et al. 2012) or color image sensors (Zhang 2000) or full calibration of both depth and color sensors including co-registration of the sensors (Herrera et al. 2011; Khoshelham et al. 2012; Zhang and Zhang 2011; Mikhelson et al. 2013). These methods are too complex to be applied in the field by an untrained operator. Nevertheless, the calibration parameters must be determined in advance by experts to achieve the accuracies specified in the publications cited. Alternatively, the default calibration parameters available from OpenNI library (OpenNI 2013) can be used. Since these parameters are not adapted for the special characteristics of the individual sensors, the accuracy is affected. Since the default calibration parameters seems to be fairly good (Herrera et al. 2011), though, we use it in our approach. Indeed, we intend to perform the process of pre-field calibration ourselves in future research for purposes of comparison.

Practically, the most critical part of calibration is the determination of the extrinsic calibration parameters related to the cargo compartment (rotation, translation, range) because this type of calibration must be performed directly in the application field by a user without any experience in computer vision. There are many established methods for determining extrinsic parameters, e.g. Nobuhara (2012) or Rodrigues et al. (2010). Such methods are inapplicable to our needs since they have a laboratory character and do not exploit the RGB-D sensor’s properties, e.g. depth information. Our calibration method relies on the special property of the RGB-D sensors used, namely the availability of color image data, 3D point cloud data and validity data in registered form. This allows us to use simple black and white planar markers as local references for calibration. The planar markers placed in the cargo compartment by the user represent the origin, the x-axis and the z-direction of the cargo compartment coordinate system (Fig. 2).

Fig. 2

Definition of the cargo compartment coordinate system with the aid of 2D markers: 1 defines the origin, 2 defines the x-axis in conjunction with marker 2, and 3 defines the z-direction in conjunction with markers 1 and 2

We use an adaptation of a marker recognition algorithm (Fiala 2004) to find the center points $\varvec{c}_{1}^{i}$, $\varvec{c}_{2}^{i}$ and $\varvec{c}_{3}^{i}$ of the markers in the color image $\varvec{I}_{t}$ (Fig. 3).

Fig. 3

Pipeline of marker recognition algorithm: a original image, b binarized image, c code reading and d result

Since the color image is registered to the point cloud by means of pre-field calibration, we have direct access to the 3D position of the center points $\varvec{c}_{1}^{s}$ ,$\varvec{c}_{2}^{s}$ and $\varvec{c}_{3}^{s}$ in sensor coordinates. We could establish a common least squares problem, for which the user would have to measure the cargo compartment coordinates of the three planar markers $\varvec{c}_{1}^{w}$, $\varvec{c}_{2}^{w}$ and $\varvec{c}_{3}^{w}$. Instead, we use a different approach in which the user only has to determine the location of the origin marker in the cargo compartment. We do this as follows:

• Find a rotation that determines the z-component $\varvec{e}_{z}^{w}$ of the base vector $\varvec{e}^{w}$ of the cargo compartment coordinate system. (We applying the Rodrigues’ formula.)

• Find the remaining “2D-rotation” that determines the x- and y-components ($\varvec{e}_{x}^{w} ,\; \varvec{e}_{y}^{w}$) of the base vector $\varvec{e}^{w}$ of the cargo compartment coordinate system.

We apply the Rodrigues’ formula to determine the z-component,

$$\varvec{R}_{1} = { \cos }\upalpha \cdot \varvec{I} + (1 - { \cos }\upalpha) \cdot \varvec{rr}^{T} + { \sin }\alpha \cdot \left[ {\begin{array}{*{20}c} 0 & { - r_{z} } & {r_{y} } \\ {r_{z} } & 0 & { - r_{x} } \\ {r_{y} } & {r_{x} } & 0 \\ \end{array} } \right]$$

(1)

where

$\varvec{R}_{1}$ :

rotation matrix

$\varvec{I}$ :

identity matrix

$\upalpha$ :

magnitude of the rotation

$\varvec{r} = \left( {\begin{array}{*{20}c} {r_{x} } & {r_{y} } & {r_{z} } \\ \end{array} } \right)^{\varvec{T}}$ :

rotation vector

When the Rodrigues’ formula is applies, the unknown parameters $\upalpha$ and $\varvec{r}$ that determine $\varvec{e}_{z}^{w}$ have to be calculated

$$\upalpha = \cos^{ - 1} \left( {\frac{{\varvec{n}^{s} }}{{\left\| {\varvec{n}^{s} } \right\|}} \times \varvec{e}_{z}^{w} } \right)$$

(2)

$$\varvec{r} =\upalpha \cdot \left( {\frac{{\varvec{n}^{s} }}{{\left\| {\varvec{n}^{s} } \right\|}} \times \varvec{e}_{z}^{w} } \right)$$

(3)

We calculate the normal vector $\varvec{n}^{s}$ thusly:

$$\varvec{n}^{s} = \left( {\varvec{c}_{2}^{s} - \varvec{c}_{1}^{s} } \right) \times \left( {\varvec{c}_{3}^{s} - \varvec{c}_{1}^{s} } \right)$$

(4)

As mentioned before, the calculated rotation matrix $\varvec{R}_{1}$ only links the sensor coordinate system to the component $\varvec{e}_{z}^{w}$. The base vector’s components $\varvec{e}_{x}^{w}$, $\varvec{e}_{y}^{w}$ are unassigned but are interrelated by $\varvec{e}_{z}^{w} = \varvec{e}_{x}^{w} \times \varvec{e}_{y}^{w}$. This means, we only have to calculate a rotation matrix $R_{2} \in \, {\mathbb{R}}^{3 \times 3}$ that determines $\varvec{e}_{x}^{w}$ or $\varvec{e}_{y}^{w}$. We generate $\varvec{R}_{2}$ as the rotation matrix representing the rotation angle $\gamma$ between $\varvec{e}_{x}^{w}$ and position $\varvec{R}_{1} \varvec{c}_{2}^{s}$ of the rotated marker 2:

$$\gamma = \cos^{ - 1} \left( {\left( {\varvec{e}_{x}^{w} \cdot \varvec{R}_{1} \varvec{c}_{2}^{s} } \right)/\left\| {\varvec{R}_{1} \varvec{c}_{2}^{s} } \right\|} \right)$$

(5)

The calibration procedure leads to a transformation formula and, thus, to a point cloud $\varvec{P}_{t}^{w}$ in the cargo compartment coordinates:

$$\varvec{p}_{i,t}^{w} = \varvec{R}_{2} \varvec{R}_{1} \left( {\varvec{p}_{i,t}^{s} - \varvec{c}_{1}^{w} } \right)$$

(6)

where

$\varvec{p}_{i,t}^{w}$ :

ith point of point cloud $\varvec{P}_{t}^{w}$ at time index t

$\varvec{p}_{i,t}^{s}$ :

ith point of point cloud $\varvec{P}_{t}^{s}$ at time index t

$\varvec{R}_{1}$ :

rotation matrix that determines $\varvec{e}_{z}^{w}$

$\varvec{R}_{2}$ :

rotation matrix that determines $\varvec{e}_{x}^{w}$ and $\varvec{e}_{y}^{w}$

$\varvec{c}_{1}^{w}$ :

translation vector (position of the origin marker)

Figure 4 presents the result of a transformation from the sensor coordinate system to the cargo compartment coordinate system using formula (6).

Fig. 4

Transformation from sensor to cargo compartment coordinate system. The colors indicate the x-axis (red), y-axis (green) and z-axis (blue)

4 Methods of Analysis

The logistical parameters loading volume, cargo space, load meters and structural change are determined on the basis of the reconstructed 3D cargo compartment, which we obtain by applying formula (6) to the measured point cloud data. The distinctive feature of the analysis module’s analyses are its conversion of point clouds (big data) into easily manageable, scalar or vector values (with the form of $func: {\mathbb{R}}^{3 \times m \times n} \to {\mathbb{R}}^{k}$), which can be transmitted easily by GPRS. We combine them into a 3D analysis module and integrate it in a modular system concept (Fig. 5). The need for such a modular software concept has been made manifest by different requirements in different projects of ours, e.g. hardware independence, independence of communication interfaces and independent development of analysis and reconstruction methods. The most important advantage is that the methods of analysis can be applied directly to merged point ($\varvec{P}_{t}^{w} = \varvec{P}_{t,sensor\,1}^{w} \cup \varvec{P}_{t,sensor\,2}^{w}$ ), thus allowing multi-view configurations that monitor larger cargo compartments to be built upon this concept.

Fig. 5

Modular diagram of the proposed approach

The idea behind the methods of analysis is to project the point cloud $\varvec{P}_{t}^{w}$ onto an equidistant grid $\varvec{G}_{t} \in {\mathbb{R}}^{{\tilde{m} \times \tilde{n}}}$ and thus allow the use of fast 2D image processing methods. We define a $\tilde{m} \times \tilde{n}$ grid that lies in the xy-plane of the cargo compartment coordinate system with an offset $\varvec{O} = \left( {x_{0} , y_{0} } \right)$ and the grid spacing parameters of $\Delta x$, $\Delta y$. We determine the points from $\varvec{P}_{t}^{w}$ that are included in each grid area and assign a mean z value to $g_{ij}$. After projection, we can use a simple difference-image technique to calculate the difference at time t as follows:

$$\varvec{D}_{t} = \varvec{G}_{t} - \varvec{G}_{t = 0}$$

(7)

where

$\varvec{D}_{t} \in {\mathbb{R}}^{{\tilde{m} \times \tilde{n}}}$ :

difference matrix at time index t

$\varvec{G}_{t} \in {\mathbb{R}}^{{\tilde{m} \times \tilde{n}}}$ :

equidistant grid at time index t

$\varvec{G}_{t = 0} \in {\mathbb{R}}^{{\tilde{m} \times \tilde{n}}}$ :

equidistant grid at calibration time

We determine the occupied volume from $\varvec{D}_{t}$ by simple integration:

$$V_{t} =\Delta x \cdot\Delta y \cdot \mathop \sum \limits_{i = 0}^{{\tilde{m}}} \mathop \sum \limits_{j = 0}^{{\tilde{n}}} d_{ij}$$

(8)

where

$d_{ij}$ :

matrix elements of $\varvec{D}_{t}$

$\Delta x$ :

length of grid element in x-direction

$\Delta y$ :

length of grid element in y-direction

We determine the occupied area $A_{t}$ by applying a threshold function $f: \varvec{D}_{t} \in {\mathbb{R}}^{{\tilde{m} \times \tilde{n}}} \to \varvec{A}_{t} \in {\mathbb{R}}^{{\tilde{m} \times \tilde{n}}}$ followed by a simple integration:

$$a_{ij} = f\left( {d_{ij} } \right) = \left\{ {\begin{array}{*{20}c} {0,} & {d_{ij} < h} \\ {1,} & {else} \\ \end{array} } \right.$$

(9)

$$A_{t} =\Delta x \cdot\Delta y \cdot \mathop \sum \limits_{i = 0}^{{\tilde{m}}} \mathop \sum \limits_{j = 0}^{{\tilde{n}}} a_{ij}$$

(10)

where

$d_{ij}$ :

matrix elements of $\varvec{D}_{\varvec{t}}$

$a_{ij}$ :

matrix elements of $\varvec{A}_{\varvec{t}}$

$h$ :

height threshold

$\varvec{D}_{t}$ and $\varvec{A}_{t}$ are two variants of the occupancy matrix, which is sent to the database as a control parameter for manual routing. Although $\varvec{A}_{t}$ contains less information than $\varvec{D}_{t}$, it also contains a significantly smaller quantity of data. This is relevant since we transmit the data by a mobile communication network.

We determine the load meter by first defining a vector $\varvec{b} = \left( {b_{0} , \, \ldots ,b_{m - 1} } \right)$ with $b_{i} = \sum\nolimits_{j = 0}^{n} {a_{ij} }$ representing the row occupancy of $\varvec{A}_{t}$. This yields the following for one load meter $L_{t}$:

$$L_{t} =\Delta y \cdot { \hbox{max} }\left( {i|b_{i} > 0} \right)$$

(11)

Finally, the structural change parameter c is specified by:

$$c = f\left( {\varvec{D}_{\varvec{t}} } \right) = \left\{ {\begin{array}{*{20}c} {0,} & {\left( {\hbox{max} \left( {\varvec{D}_{\varvec{t}} } \right) < d_{ \hbox{max} } } \right) } \\ {1,} & {else} \\ \end{array} } \right.$$

(12)

where

$d_{ \hbox{max} }$ :

maximum tolerable difference

5 Evaluation Concept

The proposed approach is currently being tested using a dual evaluation strategy. First, we are evaluating the methods based on standard loading of a cargo compartment in a laboratory setting. Second, we are using the data acquired from the mobile sensor unit, i.e. color images, occupancy matrix, etc., to verify the results of analysis from the application field manually. This rigorous evaluation is necessary because a variety of complex loading structures appear in practice, which cannot be simulated in the laboratory (Fig. 6, top). We have a research prototype in operation in a logistics partner’s vehicle and, thus, access to a constant stream of real data, which we use for manual evaluation (Fig. 6, bottom).

Fig. 6

Complex loading structures: (left) three of many possible loading structures, (right) real evaluation example

In parallel, we are currently performing the evaluation in a laboratory using standard bodies and standard loading of a cargo compartment. We have equipped our own test vehicle with the sensor system and have defined standard bodies, which mainly differ in size and shape. Figure 7 pictures an example of standard loading of a cargo compartment with standard bodies. We expect our future research to deliver concrete findings on both evaluation strategies.

Fig. 7

(Top) Standard bodies and standard loading for evaluation, (bottom) loading structure reconstructed with the aid of the occupancy matrix

6 Conclusions

We have presented a system for cargo compartment monitoring using low cost RGB-D sensors, which reconstructs and analyze the cargo compartment. Based on the depth data, we determine several parameters that help a dispatcher to identify vehicles with available cargo capacity matching the requirements of incoming transportation orders. We have presented a calibration method, practicability and user friendliness being emphases of the related research and development. Our calibration method employs 2D-markers and renders the manual calibration step dispensable. The initial results of the evaluation have been very promising when tested under laboratory and real conditions. In particular, the parameters of cargo space and load meter are measured very robustly. At present, the system has some limitations, however, such as the low robustness of the parameter of cargo volume because of obscured areas in space.

In our future work, we will be focusing on two tasks. First, we will finish our work on our dual evaluation concept in order to clearly define the system’s advantages and limits. Second, we will address the commercialization of the system developed. These findings will give rise to modified technical requirements such as the development of a fully integrated ARM embedded system based on our prototype mobile unit with an interface to trucks’ onboard computers.