Quantification of energy savings from Ireland’s Home Energy Saving scheme: an ex post billing analysis

Abstract

This paper quantifies the energy savings realised by a sample of participants in the Sustainable Energy Authority of Ireland’s Home Energy Saving (HES) residential retrofit scheme (currently branded as the Better Energy Homes scheme), through an ex post billing analysis. The billing data are used to evaluate: (1) the reduction in gas consumption of the sample between pre- (2008) and post- (2010) scheme participation when compared to the gas consumption of a control group, (2) an estimate of the shortfall when this result is compared to engineering-type ex ante savings estimates and (3) the degree to which these results may apply to the wider population. All dwellings in the study underwent energy efficiency improvements, including insulation upgrades (wall and/or roof), installation of high-efficiency boilers and/or improved heating controls, as part of the HES scheme. Metered gas use data for the 210 households were obtained from meter operators for a number of years preceding dwelling upgrades and for a post-intervention period of 1 year. Dwelling characteristics and some household behavioural data were obtained through a survey of the sample. The gas network operator provided anonymised data on gas usage for 640,000 customers collected over the same period as the HES sample. Dwelling type data provided with the population dataset enabled matching with the HES sample to increase the internal validity of the comparison between the control (matched population data) and the treatment (HES sample). Using a difference-in-difference methodology, the change in demand of the sample was compared with that of the matched population subset of gas-using customers in Ireland over the same time period. The mean reduction in gas demand as a result of energy efficiency upgrades for the HES sample is estimated as 21 % or 3,664 ± 603 kWh between 2008 and 2010. An ex ante estimate of average energy savings, based on engineering calculations (u value reductions and improved boiler efficiency and use through heating controls), suggests a technical reduction potential of 5,676 kWh per dwelling. Equating this with the gas reduction in the sample suggests a shortfall of approximately 36 ± 8 % between technical potential and measured savings. This shortfall includes the effects of direct and indirect rebound effects, variations in ex ante assumptions and achieved u values and efficiencies for upgraded dwellings. The profile of household characteristics in the HES sample is influenced by the self-selected nature of scheme participants. Self-selection bias and other possible biases in the sample data impact on the validity of the comparison. Data limitations for individual households across explanatory variables in the control and treatment groups precluded corrections for these biases in the sample; however, the profiles of separate comparable data sets were used where possible to quantify the differences in the explanatory variables and how these might impact on the measured energy saving with reference to the relevant effects identified in the literature.

Introduction

This paper presents the results of a study seeking to quantify energy savings achieved by participants in the Sustainable Energy Authority of Ireland’s (SEAI’s) Home Energy Saving scheme (HES).¹ The HES scheme aims to improve the energy efficiency of residential dwellings by providing financial support for homeowners to carry out upgrades to reduce energy demand in homes built before 2006. Technologies supported include wall and roof insulation, high-efficiency boilers and heating controls. Typically 30–35 % of the installed costs of measures are grant aided to participants.

The Energy Savings Directive (European Commission 2006) stipulates that member states of the European Union must extend the degree to which they undertake ex post empirical quantification of the impact of their energy efficiency policies and measures. Ireland submitted its first National Energy Efficiency Action Plan (NEEAP) to the European authorities in July 2009, outlining the policy measures aimed at reducing energy use by 20 % below average use in the period 2001–2005 by 2020 (Department of Communications, Energy and Natural Resources 2009). Each of the measures presented in the NEEAP has an ex ante estimate of the associated energy reductions. Ex ante estimations of energy savings are in the main based on bottom-up engineering calculations.²

For a number of policies and programmes, the measurement of impacts is being undertaken to refine the original savings estimates. Measurement methods include analysis of participants’ billing data, surveys of scheme participants, third party assessment of savings by energy advisors and programme delivery agents and installation of energy monitoring devices. This study contributes to the ex post quantifications of energy savings due to the HES scheme by:

• Establishing the energy savings associated with a sample of HES scheme participants

• Estimating the shortfall (potential savings offset) for the sample

• Considering these results in the context of other HES scheme and population data to assess the biases in the HES sample and to test the potential to extrapolate the results to broader cross sections of Irish dwellings

Engineering-estimated data are used; that is, a calculation to determine a baseline, or the situation before improvement. This is compared to an after improvement calculation, where the difference represents the energy savings achieved. By comparing the results of (ex ante) energy savings estimates based on theoretical calculations and actual energy savings based on ex post billing data, an estimate of the shortfall is made (Sorrell et al. 2009).

This shortfall results from a number of sources. Improvements in energy efficiency can reduce the cost of energy services which can in turn lead to increased energy consumption (Greening et al. 2000). An example is the installation of a more efficient heat source in a dwelling which leads to the household heating more rooms. This increased consumption can offset the energy savings that may have been achieved. Post-upgrade householders might choose increased internal temperatures in lieu of energy savings. Such behavioural changes are commonly referred to as the direct rebound effect, which can give rise to an overestimate of the real savings achieved. Indirect rebound effects result from consumers spending money saved from improved energy efficiency on the purchase of other goods and services that result in an increased energy demand elsewhere in the economy. Economy wide rebound effects can also occur when a fall in the cost of energy services induces reductions in the price of goods throughout the economy (Sorrell et al. 2009). In addition to these rebound effects, poorly installed equipment can contribute to a reduction in the measured effect of energy savings measures when compared to estimates of technical potential savings.

An estimation of the energy savings achieved by HES scheme participants is made by comparing before-and-after upgrade billing data for a sample of scheme participants with a control group from the population. Individual data for all central explanatory variables were not available in both the HES sample and the gas-connected consumer population. This influenced the method used to evaluate the energy savings realised in the HES sample treatment group when compared to the matched population control—the data sets were matched where possible before applying the difference in difference evaluation. An assessment of the biases prevalent in the HES sample is determined through the use of other available data sets on gas usage and dwelling characteristics.

The experimental used to estimate energy savings is outlined in ‘Experimental approach’ section of this paper. Details of the available data and how it is employed are included in ‘Data sources’ section. ‘Method’ section includes a description of the method employed and presents results of the analysis and surveys. ‘Results’ section includes a presentation and discussion of the results of the analysis and assessment of the impact of sample bias. ‘Conclusion’ section concludes.

Experimental approach

The participation of a household in the HES scheme represents a ‘natural experiment’. That is, homes receiving an upgrade are part of the treatment group with the remainder of the population potentially representing the control group. Natural experiments are used to measure the effect of a treatment for a given time period on different groups (Meyer 1995). A quasi-experimental approach allows an ex post evaluation of energy savings between these two groups. The quasi-experimental approach is most appropriate for the data set available as it allows different outcomes between a treatment group and a control group to be evaluated (Greenstone and Gayer 2009). It has been used in various observational studies with similar data availability as in the study presented here.

Frondel and Schmidt (2005) detail four experimental methods to determine the impact of programme implementation on actual energy use in households. These include before–after comparisons, cross-section estimators, difference-in-difference estimators and matching estimators. Before–after comparisons allow programme participants to serve as their own controls. This experimental approach may not be possible if environmental and economic conditions vary substantially over time as this will contaminate the experiment’s assumption. It is considered that the significant reduction in economic activity and the unusually cold weather in Ireland over the period of analysis for this study could result in such contamination. Cross-section estimators require that the selection groups are statistically independent of the non-treatment outcome. In this study, environmental consciousness may have motivated households to participate. This leaves the results open to selection bias (Hartman 1988; Meyer 1995; Greenstone and Gayer 2009).

The difference-in-difference method was used to analyse the available data. It is used to measure the effect of a treatment for a given time period on different groups. It involves comparing the change in the consumption of participants in an efficiency upgrade scheme with the change in consumption of a cohort of non-participants. The non-participant consumption change over the same period is the counterfactual—what the consumption change would have been in the absence of the application of efficiency measures (Frondel and Schmidt 2005). This method requires that, in the absence of the treatment, the outcome for the treatment and control groups would have followed parallel paths over time (Sorrell et al. 2009). This approach has the advantage that differences in the characteristics of the sample and the control group do not impinge on the outcome to the same degree as in simple cross-sectional studies. The behavioural, economic and environmental factors that impact on both the sample and the control group between periods are captured (Frondel and Schmidt 2005; Sommerville and Sorrell 2007; Sorrell et al. 2009; Hartman 1988)

Cook and Campbell (1979) detail the issues involved in identifying a true estimate of a treatment intervention under three categories: the internal validity, the external validity and the construct validity (Cook and Campbell 1979; Meyer 1995). An inference is internally valid if the change in the dependent variables is caused by changes in the explanatory variables. A drawback of the quasi-experimental approach is that oftentimes the results from these experiments cannot be generalised beyond the individuals or group of participants in the study (Meyer 1995). This is known as external validity. People, place and time are three major threats to external validity because the effect of policy intervention may differ for individuals and the overall population, across geographical settings and across different years (Meyer 1995; Greenstone and Gayer 2009)

Meyer (1995), whose notation is replicated here, describes a general methodology for determining the impacts of the treatment. The chosen approach should first seek to maximise the internal validity before drawing conclusions on the external validity of the model. Several methods are possible depending on the available data; all of which are based on some measurements of before intervention demand compared to post-intervention. As described by Meyer (1995), the one group before and after design is given by:

$$ {y_{{it}}} = \alpha + \beta {d_t} + { \in_{{i,t}}} $$

where y _it is the energy demand of individual i (where i = 1… N _t ), in period t = 0 and 1. d _t  = 1 if the individual is in the treatment group (i.e. undertakes an energy efficiency upgrade) and 0 otherwise (i.e. has had no upgrade). β is the estimated impact of the energy efficiency upgrade. In the absence of treatment, β = 0. The conditional mean of the error term does not depend on the treatment, $E\left[ {{ \in_{{i,t}}}\left| {{d_t}} \right.} \right] = 0.$ There is no difference in the mean energy consumption between individuals specified as being in group 1 or group 0. An unbiased estimate of the impact of the energy efficiency measures can be given by:

$$ {\widehat{\beta }_d} = \Delta \overline y = {\overline y_1} - {\overline y_0}. $$

The bar in the equation indicates an average over all individuals (i) and the subscript denotes the pre-intervention period, 0, or the post-intervention period, 1. This will only fully hold in the hypothetical situation where an individual household’s consumption can be measured with and without upgrade in time period t = 1. Frondel and Schmidt (2005) define this as the evaluation problem. It is impossible for a household’s consumption to be measured in both situations at the same time. Some studies have employed the simple before and after treatment comparison of the same group. The change in energy usage in these cases is due to the intervention but can also be as a result of other changes in external variables such as fuel price and economic changes across the measurement period. The use of a control and treatment group is employed to get around this difficulty.

The identifying equation from Meyer (1995) for the difference-in-difference method is:

$$ y_{{it}}^j = \alpha + {\alpha_1}{d_t} + {\alpha^1}{d^j} + \beta d_t^j + \in_{{it}}^j. $$

In the equation, j = 1 denotes a household belonging to the treatment group and $d_t^j$ is a dummy variable for being in the experimental group after an efficiency upgrade. For t = 0, before intervention, the mean energy usage, $\overline y$, is the same for all households. At t = 1, after the intervention, the mean energy consumption of both groups will differ if the intervention has had an effect. As before, = 0, in the absence of treatment. The difference in difference is given by:

$$ {\widehat{\beta }_{{dd}}} = \Delta \overline y_0^1 - \Delta \overline y_0^0 = \overline y_1^1 - \overline y_0^1 - \left( {\overline y_1^0 - \overline y_0^0} \right) $$

The bar is the average over i, the subscript identifies pre- and post-intervention periods and the superscript identifies the control group or the treatment group. The rationale behind this approach is that α ₁ estimates how time influences the control group and the treatment group. This approach assumes that the impact of gross domestic product (GDP), energy prices and weather on both groups is the same. The implicit assumption is that the makeup of the treatment and control groups are statistically similar across the explanatory variables. α ₁ captures the impact of time invariant differences between the two groups with α ₁ = 0 implying that the groups are similar.

The difference-in-difference method improves internal validity if both the treatment and the control groups can be verifiably shown to be similar across the explanatory variables. The lack of data on several explanatory variables in this study opens up the threat of omitted variable bias while the voluntary nature of enrolling in the HES scheme implies self-selection bias. The matching of estimators can further reduce these threats to internal validity. The method involves assigning households with a set of identifying characteristics in the treatment group with households in the control group. This increases internal validity but at the cost of external validity; that is the outcome of the study cannot be generalised to a wider group with confidence.

Participants in voluntary efficiency schemes are, by definition, self-selected. This introduces bias into the sample; for example, these participants may already be more aware of their energy consumption than the general population and hence may experience proportionally smaller savings from an efficiency scheme (Sommerville and Sorrell 2007; Sorrell et al. 2009). Differences in the physical characteristics of the dwellings as well as differences in the demographics and income profiles of the occupants are also likely to result (Hartman 1988; Sommerville and Sorrell 2007). An un-representative sample is likely to have a different average consumption to the population, but both may be affected in similar ways by external variables between time periods. If the sample shows similar past changes between time periods as the population, then any change in average energy consumption can be hypothesised to be due to the efficiency measures (Frondel and Schmidt 2005; Meyer 1995). Methods can be employed to correct for self-selection bias but are dependent on data availability and as such have not been employed here (Hartman 1988).

Data sources

HES scheme sample

In order to obtain accurate billing data for heating fuels, the cohort of HES scheme participants connected to the metered gas grid was considered, and unmetered oil-heated homes were excluded from the study. To ensure sufficient data were available on house type, size and specific energy characteristics (such as the thermal properties of all dwellings in the study), the sample size was further limited to participants who had undertaken a before-and-after building energy rating (BER).

Over 500 dwellings met these criteria at the time the study was initiated, and correspondence was sent seeking permission from the householders to collect historical energy use data for their dwellings directly from the gas and electricity meter operators.³ A total of 250 positive responses were received. For these households, metered gas data were obtained from the Gas Point Registration Operator (GPRO)⁴ which records the meter readings for all unique gas meter points in Ireland. Details of occupancy and behaviour (such as the use of secondary heating sources including plug-in electric heaters) were gathered from the sample directly by way of a postal survey (‘HES sample survey’). The data were cleaned to remove dwellings which were vacant and that had a change in the name on the gas account over the period 2007–2010. In addition, outliers were removed from the population on the basis of an upper limit set at 1.5 times the inter-quartile range for each year (between 32,000 and 36,000 kWh/year depending on the year). The HES sample was reduced from 250 to 210 data points.

The gas consumption data are based on actual meter readings and estimates (typically bi-monthly). For each data point, bi-monthly meter readings^{5 , 6} were sourced and converted to annual consumption figures using a year start and end adjustment of first and last reads for the year to account for a calendar year of gas usage data. Meter readings are a combination of actual physical reads by a GPRO representative, phone reads provided by customers and estimated reads calculated by the GPRO based on a historical use algorithm applied on a per dwelling basis. Over 57 % (average of 3.4 reads in 6 for a given year) could be categorised as accurate on the basis that they are actual reads by the GPRO or figures reported by homeowners over the phone or online for the sample during the baseline period (2007/2008). The remainder are estimates based on the GPRO algorithm. For the 2010 comparison period, the number of accurate reads increases to 64 % (3.7 reads in 6 per annum) for the sample. All dwellings had at least two actual reads per annum.

Tables 6 and 7 in the Appendix present a summary of measures installed in the sample dwellings. Roof insulation (152 households), cavity wall filling (101 households) and boiler upgrades with heating controls (87 households) are the most common measures chosen by the respondents (Table 6 in the Appendix). The coincidence of roof and cavity wall insulation was the most popular measure combination in the sample (74 households), followed by high-efficiency boiler and heating control installation (33 households). Savings estimates were considered at a household (measure package) level, rather than by individual measures given the low frequency of uptake of some measures, notable external wall insulation (15 households) and heating controls only (20 households).

Population gas data

Population gas data were provided separately by the GPRO as an anonymised dataset. This contains information on the year of connection of the dwelling to the gas grid (staring in around 1990), if there was a change in consumer linked to a specific meter point in a given year and, for some data points (approximately 50 %), the type of house and number of bedrooms. From these data, annual consumption for 2007 to 2010 was extracted. These data were cleaned for outliers above 1.5 times the inter-quartile range in each year 2007 to 2010 and for dwellings connected before 2006 leaving a total of n = 477,992. In addition, vacant homes were identified and removed.⁷ The final matched population subset excludes homes that have had a change of owner between 2007 and 2010 and matches the remaining data points to the sample on the basis on house type leaving data for 153,928 households.

Bringing Energy Home survey data

In addition to the HES sample survey, a broader HES scheme participant survey with data for over 10,000 scheme participants was available (‘Bringing Energy Home (BEH) survey’) (SEAI 2010). Almost 3,000 (2,986) households in this survey sample reported using gas as their primary heating fuel. This survey has details of occupation, income levels and age of primary household member for scheme participants.

Matching of this data at an individual dwelling level was not possible as there was no overlap between the HES sample and the BEH sample. However, a unique identifier meant that a limited set of this data could be matched to BER data enabling dwelling-specific data including floor area and dwelling type to be matched and compared to the HES sample and population statistics for the same metrics. This allows those variables mentioned to act as a proxy for missing variables in the HES sample and enables a comparison with larger samples.

2006 Census small area population statistics

The 2006 data are available at an electoral district (ED) resolution from the CSO (Central Statistics Office 2006). As the gas consumption data for the population have no information on occupancy, these data were used to establish occupancy for the entire population as well as information on average occupancy across the EDs where the HES sample respondents are located. Other variables available from the census were not employed as it was considered that the changes in these variables to 2010 would likely reduce the meaningfulness of any interpretation.

Building Energy Rating

The BER database at the time of writing contained detailed information on over 250,000 dwellings each with a unique identifier. This allowed cross-referencing with both the HES and those homes in the BEH survey that had BERs carried out, to extract variables such as floor area and house type.

A BER is an indication of the energy performance of a home. Over 120 data points are collected pertaining to energy use for space and water heating, ventilation and lighting. The resultant rating (expressed as kilowatt-hour per square meter per annum) is calculated on the basis of these data and standardised assumptions relating to occupancy (based on square meter) and heating patterns based on an assumption that the dwelling is heated to 21 °C in living areas and 18 °C in other rooms. A rating is provided to homeowners together with an advisory report identifying possible measures for home improvements, their indicative costs (high, medium and low) and impacts (high, medium and low). Assessors are certified by Ireland’s national energy agency, the SEAI, who manages a quality control and inspection programme. At the time of writing, costs for BER assessments were estimated at an average of €150/dwelling.⁸

The BER dataset was matched across dwelling type with the matched population subset. This allows for an indicative mean floor area for this cohort to be established as a means for comparison with the HES sample.

Quarterly National Household Survey

The Quarterly National Household Survey is a large-scale, nationwide survey of households in Ireland (CSO 2010). These data were used as a basis for comparing the occupation of homeowners in the HES sample with the wider population.

Matching data across explanatory variables

In order to investigate the impact of sample bias and consider the extent to which the results observed within the sample for energy savings could be extrapolated to broader cross sections of dwellings, matched samples were developed from both the BEH survey data and population gas usage data. An optimisation model that maximised the number of observations subject the constraint that a chi-squared test produced a statistically significant matched distribution. Observations were removed from the required categories on the basis of a randomly generated number simulation. This process was repeated for the population of gas consumers, the BEH dataset and the BER data set.

Method

The difference-in-difference calculation compares the change in the HES sample annual gas consumption with that of a matched population subset between a pre-intervention period (2008) and a post-intervention period (2010). The difference in the change in consumption between the HES sample (treatment) and the matched population subset (control) is interpreted as the reduction due to the HES upgrades. Internal validity between the two groups is improved by the matching of the gas consumption population with the HES sample over the explanatory variables of house type and tenure. This increases the likelihood that exogenous variables that change over time impact on both groups in a similar way. Three stages in the matching process are compared in Table 2 to the sample to illustrate the improvement.

To determine the shortfall, the difference between the average ex ante estimate of savings is compared to the average measured savings. Engineering estimates are based on the methodology used in the Dwelling Energy Assessment Procedure (DEAP) software used to generate BERs for dwellings in Ireland.⁹ The three typical dwelling types were modelled in DEAP, and the impact of application of measure combinations analysed on the basis of improved u values for insulation measures, and improved efficiency of heating supply for high-efficiency boilers and heating controls, to determine estimates for energy savings per measure. These are shown to be additive for insulation measures by the software, with a reduction made to unitary savings estimates for heating supply efficiency measures to account for instances where these are installed in combination with insulation measures. The savings are applied on a unitary savings per measure basis for three dwelling types, as outlined in Appendix, Table 8.

While the shortfall between engineering estimates for gas demand reduction and billing data analysis cannot be disaggregated into the potential influences on the difference, an estimate is made of their combined impact. The difference between expected savings and observed savings is attributable to behavioural change, poor initial estimates of achievable savings, poorly performing equipment and potential inefficiencies in the systems installed and the way these systems are used (Sorrell et al. 2009). Behavioural change is a major contributing factor to the direct rebound effect; however, other factors such as the physical makeup of the building are also influential, but these can be hard to isolate or quantify. Other factors include the climate, the health and well-being of occupants and possibly household income.

Further comparisons of the sample with available data are undertaken to quantify the degree to which the sample may be biased. Firstly, the mean annual consumption in the pre-intervention period is compared across the treatment and the control to assess their similarity. Secondly, the change in gas consumption between 2007 and 2008 for the HES sample and the matched population subset is compared. This establishes if the two groups are likely to respond to changes in exogenous variables in a similar way. Lastly, the HES sample is compared to various population data sources to assess the degree to which various explanatory variables may be contributing to bias in the result. The data collected in the HES sample allow for some direct comparisons with the population and some qualitative assessments of the likely nature and direction of sample bias on the result. Where direct comparisons between the HES sample and the wider population were not possible, other data sets were matched with the HES sample across the available explanatory variables—dwelling type in most instances. This also allows for a qualitative statement on how the findings of this analysis transfer to a wider number of gas users. Table 1 shows the data available for comparison over the relative explanatory variables in each dataset.

Table 1 Summary of data sources and key variables

Results

Sample and population comparisons

Table 2 shows the mean annual energy usage across the years examined in the study¹⁰ as well as the mean change in consumption across pre-intervention years (2007–2008) and between pre-and post-intervention years (2008–2010). The HES sample is compared to each population subset for statistical similarity using the appropriate t test. Whilst the mean HES sample consumption is higher than the matched population subset in the pre-intervention years (2007 and 2008), the change in demand across these years is statistically similar.

Table 2 Annual consumption and change in consumption in pre- and post-upgrade periods for HES sample and population subsets

As expected, the population sample matched for house type displays the most similarly in the change in consumption in pre-intervention periods. The results suggest that while the HES sample characteristics result in a significantly higher demand, both the HES sample and the population have been impacted in similar ways by exogenous factors such as fuel price and reductions in GDP in the pre-intervention period. This is especially true for the matched population data.

The mean change in the energy use of HES treatment sample over the period is 3,698 kWh, while the matched by house type population reduces by 34 kWh over the same period. Applying the difference-in-difference method, the mean energy reduction attributable to the HES intervention is then 3,664 kWh or 21 % (in the range 3,061 and 4,387 kWh or 17–24 % at a confidence interval of 95 %).

The ex ante engineering estimate predicts a saving of 5,676 kWh. A mean shortfall of 36 ± 8 % is implied for the HES sample. The shortfall results of this study are similar to the findings of work reviewed by Sorrell et al. (2009). The study estimates that the direct rebound effect for household heating is 30 % or less. Hausman (1979) and Dubin and McFadden (1984) carried out studies that provided direct rebound estimates of 1–26 % for household cooling, which is comparable to household heating (Hausman 1979; Dubin and McFadden 1984).

Sample bias

The HES scheme is open to all householders with dwellings built before 2006. Participants must produce some up-front capital; generally 65–70 % of the cost of measures installed. This potentially rules out low income households from the scheme. Hence, the sample is more likely to include those that can afford sufficient energy to comfortably heat their homes and for which fuel costs may represent a smaller proportion of their total expenditure compared to fuel-poor groups.

While there has been widespread advertising of the HES scheme, a survey undertaken by SEAI (BEH survey) suggests that the socio-economic profile of participants is not comparable with that of the population (SEAI 2010). For example, while participants’ incomes are somewhat representative of national patterns, low participation rates of single adults and high participation rates of retired householders have been observed. Demographic characteristics of a household such as age profile, home ownership and employment status are contributing factors to variations in energy demand per dwelling.

The sample includes only participants who opted voluntarily for a before-and-after BER, for which there was a small additional fee. This might indicate that sample householders had a specific interest in the impact of the planned upgrades on their asset rating (BER). On this basis, sample householders may have been more aware of their energy use prior to scheme participation, compared with participants who were not interested in the BER before or after an upgrade.

Dwellings that use heating fuels other than gas were excluded. A comparison of average dwelling fuel use data suggests that gas customers use less kilowatt-hour per annum than those who use oil, the other dominant main heating source in Ireland (SEAI 2008). Given that oil-heated homes tend to be larger and more often located in rural areas compared to gas grid-connected dwellings, the per measure savings could be higher for this cohort of dwellings.

Table 3 compares the HES sample explanatory variables with the available data in other available data sets. House type was available across all available data sets and Table 3 shows the before and after matching p values. The full available sample of 256 respondents to the HES survey was used for house type.

Table 3 Before and after matching by house type across available data sets

As evidenced by Fig. 1 and Table 4, the HES sample dwellings were less energy efficient than the BER sample matched by house type¹¹ prior to undertaking upgrades, as measured by BER ratings. Post-upgrades, the same dwellings rated better than the BER-matched sample in terms of efficiency, thus all else being equal, and by extrapolation to the population,¹² it could be expected that energy demand before upgrades would be higher than the population (as observed), and after upgrades lower than the population. However, as per Table 3, energy demand of the HES sample after upgrade (2010) was approximately equal to that of the matched population subset (matched on the basis of house type); confirming the sample bias as described above. There are a number of factors that could influence this result as borne out by this study and in the literature.

Fig. 1

BER before and after scheme participation for HES sample and BER-matched sample

Table 4 Before and after HES scheme participation BER comparison for the HES sample and the matched BER data

Factors likely to result in a higher than average energy demand include dwellings with larger floor areas and older homes.¹³ The comparison of the mean floor area with a matched sample of BER data (Table 5) shows that the dwellings in the sample have a significantly larger floor area. This can account for some of the tendency towards higher energy use in the sample. This is confirmed by Leahy and Lyons (2009) who demonstrated on the basis of OLS regression of Irish Household Budget Survey data (2004/2005) a significant association between greater floor area (using number of bedrooms as a proxy) and greater energy demand (Leahy and Lyons 2009).

Table 5 Statistical test result (p values for chi-square and t test) for explanatory variable similarity between data sets

The HES sample dwellings are significantly different to population data for dwelling age (period of construction). The population has a significantly higher proportion of dwellings in the newest age band (2001–2006), with the HES sample having a higher proportion of dwellings built between 1940 and 1980. As per Leahy and Lyons (2009), dwellings built after 1980 use significantly less energy (fuels) compared to reference case dwellings built between 1918 and 1960. This evidence supports the observation of higher energy use in the HES sample compared to the population before upgrades. The higher proportion of dwellings built pre-1919 in the population compared to HES sample dwellings is noted and may be a function of the potential to apply supported insulation technologies (in particular wall insulation) to this period of dwellings. No significant dwelling age difference is evident between the HES sample and the matched BEH data (Fig. 2).

Fig. 2

Comparison across period of construction for relevant data sets

A comparison of occupancy indicates that the HES sample dwellings have significantly lower occupancy compared to the population. When compared to the average occupancy of homes in the same electoral districts, the HES sample still has a significantly lower occupancy. This factor is likely to lead to a reduced energy demand in the HES sample when compared to population as indicated by Leahy and Lyons’s finding that higher occupancy is associated with increased energy demand.¹⁴

Whilst occupation data were not available for the HES sample, the matched BEH sample can be used as a proxy. Differences between the population and the matched BEH sample indicate significantly greater proportions of working households, lower proportions of both unemployed and working from home households when the status of the household chief economic supported (CES) is considered. The greater proportion of working households could be results of greater ability to pay for upgrades (participate in the scheme). Further, these three factors could result in households for scheme participants being occupied for less hours per day compared to the population in general. The higher proportion of retired households is likely to influence demand in the other direction. Leahy and Lyons indicate greater energy demand for retired households compared to those with a full time employed CES (Fig. 3).

Fig. 3

Comparison of occupation of CES

Along with the observable explanatory variables described above, another significant unobservable variable impacting on the usage of the sample is that of self-selection. Hartman highlights how households who enter voluntary energy efficiency schemes are more likely to be more conscious and knowledgeable about energy conservation and thereby be more proactive in reducing energy usage (Hartman 1988).

The HES sample includes only participants who opted voluntarily for a before-and-after BER, for which there was a small additional fee. This might indicate that sample householders had a specific interest in the impact of the planned upgrades on their asset rating (BER). On this basis, sample householders may have been more aware of their energy use prior to scheme participation, compared with participants who were not interested in the BER before or after an upgrade. Hartman’s analysis of self-selection showed how voluntary participation can account for a significant portion recorded savings (Hartman 1988). A similar effect is likely to be in effect in the sample analysed here.

Conclusion

The mean (climate-corrected) reduction in gas demand for the HES sample as a result of energy efficiency upgrades is estimated as 21 % or 3,664 ± 603 kWh between 2008 and 2010. Ex ante engineering calculations of energy saving for the sample dwellings estimate an average reduction of 5,676 kWh. Equating this with the ex post billing analysis results suggests a shortfall of approximately 36 ± 8 %.

Given the statistical differences between the sample and the population for dwelling and occupant characteristics that affect energy demand, it is not possible to draw conclusions that can be applied to the population of dwellings in Ireland with the potential for energy efficiency improvements. The results do, however, verify that substantial energy savings (>21 % per dwelling) are being realised through the scheme. In addition to energy savings, surveys of HES participants indicated a range of other benefits realised as a result of energy efficiency upgrades, including: well-being, improved internal climate (increased comfort and reduced dampness) and perceived increase in the value of dwellings.

Given the non-random nature of the sample and based on the tests outlined above, it is clear that the sample is not fully representative of the population as a whole. This limits these specific findings to the impacts on energy demand for energy consumers with similar characteristics (dwelling and homeowner) to those of the sample analysed. Sample bias impacts the results in different directions and by different orders of magnitude. Whilst these have not been specifically quantified, in accordance with the available literature, the original energy efficiency standard of the dwelling, differences in floor area and occupation of the CES all have the potential to lead to an overestimate of savings at the population level on the basis of the sample results. However, differences in occupancy, main heating fuel source (gas) and self-selection for the scheme are likely to influence the result towards an underestimate of population wide potential.

A statistically significant sample, representative of dwellings in Ireland that have undergone an upgrade through HES, could be more usefully compared to a control group of dwellings that had not undergone an upgrade. Such a study could provide sufficient data to assess the overall potential for improvement in Ireland’s dwelling stock for a given suite of technologies.

Appendix

Table 6 Efficiency upgrades undertaken in the sample

Table 7 Measure combinations

Table 8 Unitary savings estimates