Benchmarking and Movement Preservation: Evidences from Real-Life and Simulated Series

Abstract

The benchmarking problem arises when time series data for the same target variable are measured at different frequencies with different level of accuracy, and there is the need to remove discrepancies between annual benchmarks and corresponding sums of the sub-annual values. Two widely used benchmarking procedures are the modified Denton Proportionate First Differences (PFD) and the Causey and Trager Growth Rates Preservation (GRP) techniques. In the literature it is often claimed that the PFD procedure produces results very close to those obtained through the GRP procedure. In this chapter we study the conditions under which this result holds, by looking at an artificial and a real-life economic series, and by means of a simulation exercise.

The views expressed herein are those of the authors and should not be attributed to the IMF, its Executive Board, or its management

1 Introduction

Benchmarking monthly and quarterly series to annual series is a common practice in many National Statistical Institutes. The benchmarking problem arises when time series data for the same target variable are measured at different frequencies with different level of accuracy, and there is the need to remove discrepancies between annual benchmarks and corresponding sums of the sub-annual values. The most widely used benchmarking procedures are the modified Denton Proportionate First Differences (PFD) technique [4, 6], and the [3] Growth Rates Preservation (GRP) procedure (see also Trager [11], and Bozik and Otto [2]). The PFD procedure looks for benchmarked estimates aimed at minimizing the sum of squared proportional differences between the target and the unbenchmarked values, and is characterized by an explicit benchmarking formula involving simple matrix operations. The GRP technique is a nonlinear procedure based on a “true” movement preservation principle, according to which the sum of squared differences between the growth rates of the target and of the unbenchmarked series is minimized. As in the literature [1, 4, 5] it is often claimed that the PFD procedure produces results very close to those obtained through the GRP procedure, in this chapter we study the conditions under which this result holds. We do that by showing how the two procedures work in practice, by looking at an artificial and a real-life economic series. Then a simulation exercise is performed in order to appreciate the impact on the benchmarked series of the variance of the observational error and of possible “steps” in the annual benchmarks.

The chapter is organized as follows. In Sect. 45.2 the two benchmarking procedures are described, and the way they take into account a “movement preservation principle” is discussed. In Sect. 45.3 the artificial time series of Denton [6] and a quarterly preliminary series of the EU Quarterly Sector Accounts [7] are benchmarked to their annual counterparts, using both modified Denton PFD and Causey and Trager GRP benchmarking procedures, and the results are discussed. In Sect. 45.4 we design a simulation exercise in order to analyze the distinctive features of the two procedures.

2 Two Benchmarking Procedures

Let Y _T , T = 1, …, N, and p _t , t = 1, …, n, be, respectively, the (say annual) totals and the (say quarterly) preliminary values of an unknown quarterly target variable y _t . The preliminary values being not in line with the annual benchmarks, i.e., $\sum _{t\in T}p_{t}\neq Y _{T}$, T = 1, …, N, we look for benchmarked estimates y _t ^b such that $\sum _{t\in T}y_{t}^{b} = Y _{ T}$.

As Bozik and Otto [2, p. 2] stress, “Just forcing a series to sum to its benchmark totals does not make a unique benchmark series.” Some characteristic of the original series p _t should be considered in addition, in order to get benchmarked estimates “as close as possible” to the preliminary values. In an economic time series framework, the preservation of the temporal dynamics (however defined) of the preliminary series is often a major interest of the practitioner. Thus in what follows we consider two procedures designed to preserve at the best the movement of the series p _t : modified Denton PFD and Causey and Trager GRP.¹

Denton [6] proposed a benchmarking procedure grounded on the Proportionate First Differences between the target and the original series. Cholette [4] slightly modified the result of Denton, in order to correctly deal with the starting conditions of the problem. The PFD benchmarked estimates are thus obtained as the solution to the constrained quadratic minimization problem

$$\displaystyle{ \min _{y_{t}}\sum _{t=2}^{n}{\left (\frac{y_{t}} {p_{t}} -\frac{y_{t-1}} {p_{t-1}}\right )}^{2}\qquad \mbox{ subject to}\quad \sum _{ t\in T}y_{t} = Y _{T},\quad T = 1,\ldots ,N. }$$

(45.1)

In matrix notation, denoting p and Y the (n ×1) and (N ×1), respectively, vectors of preliminary and benchmark values, the PFD benchmarked series is contained in the (n ×1) vector y ^PFD solution of the linear system [4, p. 40]

$$\displaystyle{ \left [\begin{array}{cc} \mathbf{Q}&\mathbf{C}\prime\\ \mathbf{C} & \mathbf{0} \end{array} \right ]\left [\begin{array}{c} {\mathbf{y}}^{\mathrm{PFD}}\\ \boldsymbol{\lambda } \end{array} \right ] = \left [\begin{array}{c} \mathbf{Q}\mathbf{p}\\ \mathbf{Y} \end{array} \right ], }$$

(45.2)

where $\boldsymbol{\lambda }$ is a (N ×1) vector of Lagrange multipliers, $\mathbf{Q} ={ \mathbf{P}}^{-1}\boldsymbol{\Delta }^{\prime}_{n}\boldsymbol{\Delta }_{n}{\mathbf{P}}^{-1}$, P = diag(p), C is a (N ×n) temporal aggregation matrix converting quarterly values in their annual sums, and $\boldsymbol{\Delta }_{n}$ is the ((n − 1) ×n) first differences matrix.

Notice that $\boldsymbol{\Delta }^{\prime}_{n}\boldsymbol{\Delta }_{n}$ has rank n − 1, so Q is singular. However, provided no preliminary value is equal to zero, the coefficient matrix of system (45.2) has full rank (see the Appendix). After a bit of algebra, the solution of the linear system (45.2) can be written as

$$\displaystyle{ \left [\begin{array}{c} {\mathbf{y}}^{\mathrm{PFD}}\\ \boldsymbol{\lambda } \end{array} \right ] = \left [\begin{array}{cc} \mathbf{P}& \mathbf{0}\\ \mathbf{0} &\mathbf{I} _{N} \end{array} \right ]{\left [\begin{array}{cc} \boldsymbol{\Delta }^{\prime}_{n}\boldsymbol{\Delta }_{n}&\mathbf{P}\mathbf{C}^\prime\\ \mathbf{C} \mathbf{P} & \mathbf{0} \end{array} \right ]}^{-1}\left [\begin{array}{c} \mathbf{0} \\ \mathbf{Y}\end{array} \right ]. }$$

(45.3)

Causey and Trager [3] consider a different quadratic minimization problem, in which the criterion to be minimized is explicitly related to the growth rate, which is a natural measure of the movement of a time series:

$$\displaystyle{ \min _{y_{t}}\sum _{t=2}^{n}{\left ( \frac{y_{t}} {y_{t-1}} - \frac{p_{t}} {p_{t-1}}\right )}^{2}\qquad \mbox{ subject to}\quad \sum _{ t\in T}y_{t} = Y _{T},\quad T = 1,\ldots ,N. }$$

(45.4)

Looking at the criterion to be minimized in (45.4), it clearly appears that, differently from (45.1), it is grounded on an “ideal” movement preservation principle, “formulated as an explicit preservation of the period-to-period rate of change” of the preliminary series [1, p. 100].

It should be noted that while problem (45.1) has linear first-order conditions for a minimum, and thus gives rise to an explicit solution as shown in (45.3), the minimization problem in (45.4) is inherently nonlinear. Trager [11, see Bozik and Otto [2]] suggests to use a technique based on the steepest descent method,² using y ^PFD as starting values, in order to calculate the benchmarked estimates y _t ^GRP, t = 1, …, n, solution to problem (45.4).

We employ the Interior Point method of the Optimization Toolbox of MATLAB^​®; (version 2009b). It consists of an iterative procedure that solves a sequence of approximate unconstrained minimization problems by standard (quadratic) nonlinear programming methods. In each iteration the procedure exploits the exact gradient vector and hessian matrix of the Lagrangian function [8], which enables to make informed decisions regarding directions of search and step length. This fact makes the procedure feasible and robust, in terms of reduced numbers of iterations required for the convergence, as far as of quality of the found minimum.

It is interesting to go deep into the relationship between the criteria optimized by the two alternative procedures. Let

$$\displaystyle{C_{\mathrm{PFD}} =\sum _{ t=2}^{n}{\left (\frac{y_{t}} {p_{t}} -\frac{y_{t-1}} {p_{t-1}}\right )}^{2}\quad \mbox{ and}\quad C_{\mathrm{ GRP}} =\sum _{ t=2}^{n}{\left ( \frac{y_{t}} {y_{t-1}} - \frac{p_{t}} {p_{t-1}}\right )}^{2}}$$

be the objective functions of the PFD and GRP benchmarking procedures, respectively. We can write (U.S. Census Bureau [12], p. 96):

$$\displaystyle{ C_{\mathrm{PFD}} =\sum _{ t=2}^{n}{\left [\frac{y_{t-1}} {p_{t}} \left ( \frac{y_{t}} {y_{t-1}} - \frac{p_{t}} {p_{t-1}}\right )\right ]}^{2}. }$$

(45.5)

Expression (45.5) makes clear the relationship between C _PFD and C _GRP. The term in parentheses is the difference between the growth rates of the target and the preliminary series, namely the addendum of C _GRP. In C _PFD these terms are weighted by the ratio between the target series at t − 1 and the preliminary series at t. When these ratios are relatively stable over time, which is the case when the “benchmark-to-indicator ratio”  [1]

$$\displaystyle{ \frac{Y _{T}} {\sum _{t\in T}p_{t}},T = 1,\ldots ,N}$$

is a smooth series, C _PFD and C _GRP are very close to each other. On the contrary, when the ratios ($y_{t-1}/p_{t}$) behave differently each term in the summation is over-(under-)weighted according to the specific relationship between target and preliminary series in that period. For example, sudden breaks in the movements of $y_{t-1}/p_{t}$ might arise in case of large differences between the annual benchmarks and the annually aggregated preliminary series. The situation is rather similar to the one described by Dagum and Cholette [5, p. 121], when the risk of producing negative benchmarked values is discussed: “These situations occur when the benchmarks dramatically change from one year to the next, while the sub-annual series changed very little in comparison; or, when the benchmarks change little, while the annual sums of the sub-annual series change dramatically.”

Keeping in mind this relationship, we move to investigate on the differences between the PFD and the GRP benchmarking solutions in simulated and real-life cases.

3 Evidences from Artificial and Real Time Series

In this section we apply both the PFD and GRP benchmarking procedures to two illustrative examples, in order to show to what extent the former solution can be used effectively to approximate the “ideal” movement preservation criterion based on growth rates. We consider also a distance measure between the growth rates of the preliminary and target series given by the absolute, rather than the squared, value of their difference. The results are thus evaluated looking at the two ratios

$$\displaystyle{ r_{\alpha } ={ \left (\frac{\sum _{t=2}^{n}{\left \vert \frac{y_{t}^{\mathrm{GRP}}} {y_{t-1}^{\mathrm{GRP}}} - \frac{p_{t}} {p_{t-1}}\right \vert }^{\alpha }} {\sum _{t=2}^{n}{\left \vert \frac{y_{t}^{\mathrm{PFD}}} {y_{t-1}^{\mathrm{PFD}}} - \frac{p_{t}} {p_{t-1}}\right \vert }^{\alpha }}\right )}^{\frac{1} {\alpha } }\qquad \alpha = 1,2. }$$

(45.6)

When α = 2, this index is simply the square root of the ratio between the Causey and Trager “Growth Rate Preservation” criteria computed from the two solutions. Obviously, we expect that the GRP technique always reaches a lower value of the chosen criterion than PFD, and thus the ratio r ₂ should be never larger than 1. Put in other words, r ₂ is the ratio between the Root Mean Squared Adjustments to the preliminary growth rates produced by the Causey and Trager GRP and the Denton PFD benchmarking procedures. On the other hand, r ₁ can be seen as the ratio between the Mean Absolute Adjustments: sometimes this index can be larger than 1, thus indicating a better performance of Denton PFD when the size of the corrections to the preliminary growth rates is measured according to an absolute rather than a squared form.

The first example we consider is the artificial preliminary series used in the seminal paper of Denton [6]. It consists of a 5-year artificial quarterly series, with a fixed seasonal pattern invariant from year to year. The values are 50, 100, 150, and 100 in the four quarters, for a total yearly amount of 400. The annual benchmarks are assumed to be 500, 400, 300, 400, and 500 in the 5 successive years. The corresponding discrepancies (i.e., the differences between the known benchmarks and the annual sums of the preliminary series) are therefore 100, 0, − 100, 0, and 100, respectively. As expected, the minimum C _GRP is achieved by the GRP procedure (0.04412 against 0.14428 of PFD, with r ₂ = 0. 553). The GRP procedure shows better results as regards the movement preservation, also when the distance between the preliminary and the target growth rates is measured by the absolute differences (r ₁ = 0. 539). Figure 45.1 shows the adjustments to the levels of the original series in the two cases. The horizontal lines in each year denote the (average) annual discrepancy to be distributed.

Fig. 45.1

Adjustments to the artificial series produced by the PFD and GRP procedures

The second example is a real-life economic series coming from the European Quarterly Sector Accounts (EU-QSA). The EU-QSA system has been dealt with by Di Fonzo and Marini [7] in a reconciliation exercise, where several time series have to be adjusted in order to be in line with both temporal and contemporaneous known aggregates [5]. In this chapter we consider the series “Other Property Income” of the Financial Corporation sector, showing a considerable amount of temporal discrepancies. Figure 45.2 shows the large discrepancy in 2002, when the original series accounts for just 65% of the annual target. From 2003 onwards the discrepancies are much more contained. This is a typical practical situation where the preservation of the original growth rates can be better guaranteed by the GRP procedure (r ₂ = 0. 579 and r ₁ = 0. 615). The quarterly adjustments in the two cases are also displayed in Fig. 45.2. The differences are large in the years with large discrepancies (2001–2002), but they are also remarkable in 2003, when the discrepancy is limited. In this case the smoother distribution produced by the GRP procedure is clearly visible.

Fig. 45.2

Adjustments to the real-life series produced by the PFD and GRP procedures

4 The Simulation Exercise

By means of this experiment we wish to shed light on the conditions under which the PFD benchmarking procedure produces results “close” to the GRP technique in terms of differences between the growth rates of the benchmarked and preliminary series. We consider quarterly series covering a period of 7 years (n = 28).

Let $\theta _{t} =\theta _{t-1} +\varepsilon _{t}$ be a random walk process, where $\varepsilon _{t}$ is a Gaussian white noise with unit variance ($\sigma _{\varepsilon }^{2} = 1$) and $\theta _{0} =\varepsilon _{0}$. The target series of the exercise, y _t , is derived as

$$\displaystyle\begin{array}{rcl} y_{t} =\theta _{ t}^{{\ast}} +\mu _{ t},\qquad t = 1,\ldots ,n& & {}\\ \end{array}$$

with $\theta _{t}^{{\ast}} = 100 +\theta _{t}$, where the constant term 100 is large enough to prevent negative values, and μ _t is given by

$$\displaystyle\begin{array}{rcl} \mu _{t} = \left \{\begin{array}{l@{\quad }l} \mu \quad &t = 9,\ldots ,16 \\ -\mu \quad &t = 17,\ldots ,24 \\ 0 \quad &\text{elsewhere}\\ \quad \end{array} \right..& & {}\\ \end{array}$$

The preliminary series p _t is related to y _t as follows:

$$\displaystyle\begin{array}{rcl} p_{t} =\theta _{ t}^{{\ast}} + e_{ t},\qquad t = 1,\ldots ,n& & {}\\ \end{array}$$

where e _t is a Gaussian white noise with variance σ _e ². It is clear that preliminary and target series are different for the effects of μ _t and e _t . The former is introduced in the model for y _t in order to simulate yearly biases of the preliminary series. The first control parameter of the experiment is thus μ. When μ > 0, the target series contains a positive drift from p _t in years 3 and 4, followed by a negative step (of the same amount) in years 5 and 6. We set μ = 0, 15, 30, 45, 60. The second control parameter is σ _e , the standard deviation of the innovation process e _t . The larger this parameter is, the larger the observational error in the preliminary series will be. We set σ _e  = 5, 10, 15, 20, 25.

We drew two sets of 1,000 n-dimensional vectors as N(0, 1). One set is used to simulate $\varepsilon _{t}$; the other is used to derive the innovation e _t according to the five levels of σ _e . By using the five values of μ, we achieved 1,000 experiments for each of the 25 combinations. For each combination, we computed summary statistics on the ratios r ₁ and r ₂ obtained over the 1,000 experiments.

Tables 45.1 and 45.2 show, respectively, the median and the maximum values³ of r ₂ under different values of σ _e (rows) and μ (columns).

Table 45.1 Median of r ₂ for different values of σ _e and μ (across 1,000 experiments)

Table 45.2 Maximum of r ₂ for different values of σ _e and μ (across 1,000 experiments)

According to Bozik and Otto [2], we used the series benchmarked via modified Denton PFD as starting values of the GRP procedure, and this turned out to be a good choice⁴: as one would expect, from Table 45.2 it appears that in all cases the GRP procedure improves on the modified PFD starting values and reaches a lower value of the criterion.

However, from Table 45.1 we observe that the PFD procedure provides very similar results to GRP when discrepancies are small and unsystematic (median r ₂ ≥ 0. 9 when σ _e  ≤ 10 and μ ≤ 15). The reduction is stronger as both σ _e and μ increase.

These results are confirmed by r ₁, whose median and maximum values are shown in Tables 45.3 and 45.4, respectively, with an important remark: from Table 45.4 we observe that if the absolute differences between preliminary and target growth rates are considered, and when the bias is either absent or small (μ ≤ 15), there are cases where Denton PFD gives benchmarked estimates whose dynamics is “closer” to the preliminary series than Causey and Trager GRP does.⁵

Table 45.3 Median of r ₁ for different values of σ _e and μ (across 1,000 experiments)

Table 45.4 Maximum of r ₁ for different values of σ _e and μ (across 1,000 experiments)

A visual, more comprehensive overview of the results of the simulation experiment is given by the boxplots of r ₂ (Fig. 45.3), and r ₁ (Fig. 45.4). The 25 boxplots in each figure are ordered (from left to right) so that the first group of five boxplots corresponds to σ _e  = 5 and μ = 0, 15, 30, 45, 60, respectively, the second group to σ _e  = 10 and μ = 0, 15, 30, 45, 60, and so on.

Fig. 45.3

Boxplots of r ₂

Fig. 45.4

Boxplots of r ₁

From these evidences, and for this dataset, we conclude that the modified Denton PFD benchmarking procedure can be viewed as a sensible approximation of the Causey and Trager benchmarking procedure when the variability of the preliminary series and/or its bias are low with respect to the target variable. When this is not the case (high variability and/or large bias), the quality of the approximation clearly worsens. In addition, as regards the “movement preservation,” we have found that generally the Causey and Trager GRP benchmarking procedure gives better performances as compared to Denton PFD. This turned out to be always the case (as one would expect) when the comparison criterion is the one optimized by the Causey and Trager procedure, and in the very largest amount (more than 98%) of the 25,000 series of our simulation experiment, when the distance between the growth rates of the preliminary and the benchmarked series is measured by the absolute difference.

Appendix: Non-singularity of the Coefficient Matrix of System (45.2)

Luenberger [9, p. 424] shows that a unique solution to the problem

$$\displaystyle{\min _{\mathbf{y}}\mathbf{y}\prime\mathbf{Qy}\qquad \mbox{ subject to}\quad \mathbf{Cy} = \mathbf{Y}}$$

exists if the matrix C is of full rank, and the matrix Q is positive definite on the null space of matrix C: $\mathcal{N}(\mathbf{C}) =\{ \mathbf{x} \in {\mathbb{R}}^{n} : \mathbf{Cx} = \mathbf{0}\}$.

Let us consider the matrix $\mathbf{Q} ={ \mathbf{P}}^{-1}\boldsymbol{\Delta }^{\prime}_{n}\boldsymbol{\Delta }_{n}{\mathbf{P}}^{-1}$, and let the vector y belong to $\mathcal{N}$(C). We assume p _t ≠0, t = 1, …, n (otherwise the objective function is not defined), and Cp≠Y, which corresponds to exclude the trivial solution y ^∗ = p, valid when there is no benchmarking problem. Given that

$$\displaystyle{\mathbf{y}\prime\mathbf{Qy} =\sum _{ t=2}^{n}{\left (\frac{y_{t}} {p_{t}} -\frac{y_{t-1}} {p_{t-1}}\right )}^{2},}$$

it is immediately recognized that the expression above is strictly positive, which means that matrix $\mathbf{Q} ={ \mathbf{P}}^{-1}\boldsymbol{\Delta }^{\prime}_{n}\boldsymbol{\Delta }_{n}{\mathbf{P}}^{-1}$ is positive definite on the null space spanned by the columns of matrix C, and thus the coefficient matrix of system (45.2) is nonsingular.