How Poverty Indicators Confound Poverty Reduction Evaluations: The Targeting Performance of Income Transfers in Europe

Abstract

This paper investigates whether two popular poverty indicators, namely income poverty and material deprivation, reach similar conclusions about the poverty reduction effects of income transfers. Such evaluations generally use income poverty. It is well-known, however, that poverty indicators regularly disagree about a person’s poverty status. What is less known is whether such disagreement also confounds estimates of a program’s poverty reduction effects. This paper compares the targeting performance of social assistance, housing and family transfers in countries with different welfare states namely Germany, France, Ireland, the Netherlands, Sweden and the United Kingdom. It finds that a transfer’s targeting performance does not differ much when defining the transfer’s target group either as the poorest income quintile or the poorest material deprivation quintile. Yet, when combining the information from both indicators, transfers appear much more effective in reaching those groups that both poverty indicators identify as part of the target group. Transfers also appear much more efficient in excluding non-target populations. For the groups on which the poverty indicators disagree, more analysis is needed. Triangulation between poverty indicators thus improves the validity of program evaluations as it enables a better separation between (potential) poverty measurement issues and the measurement of a program’s (potential) effects.

1 Introduction

Income transfers reduce poverty. Governments spend a significant part of their budgets on income transfers but are increasingly under pressure to do more with less. Evaluating whether transfers are efficient and effective in reducing poverty is thus important. Such assessments are typically based on only one poverty indicator, and income poverty indicators are by far the most popular. However, a large body of research on poverty measurement argues that income poverty indicators should be complemented with other poverty indicators (i.e. Battiston et al. 2013; Bossert et al. 2013; Cancian and Meyer 2004; Fusco et al. 2011; Nolan and Whelan 2010). Using income poverty and material deprivation indicators, this paper investigates whether the insights from this literature also apply to the practice of evaluating poverty reduction.

The argument for complementing income poverty indicators with direct indicators of poverty such as material deprivation has a conceptual and a measurement reason. Direct (material deprivation) and indirect (income) poverty indicators are related but conceptually distinct proxies of material wellbeing (Ringen 1988). Income poverty indicators document, what is for many, the most important financial resource to attaining material wellbeing; Material deprivation indicators document adverse material outcomes resulting from insufficient financial resources. Each poverty indicator further has its own specific strengths and weaknesses when it comes to measuring poverty. For instance, income poverty indicators do not account for savings or debts; material deprivation indicators assume that everyone has the same prioritization of needs.

Consequently, income poverty and material deprivation indicators regularly disagree about a person’s poverty status. In Western Europe, the positive but relatively modest correlation between (adult equivalent) income and the number of material deprivations ranges from 0.17 in Denmark to 0.36 in Belgium resulting in uncertainty around the poverty status of 12–20 % of the population (Fusco et al. 2010, Table 6.1; Fusco et al. 2011, Table A4). Similar results are found for different indicator definitions (i.e. Cancian and Meyer 2004; Nolan and Whelan 2010). The European Union (EU) therefore uses multiple indicators to monitor poverty (Marlier et al. 2007).

Does such disagreement also confound the estimates of programs’ poverty reduction effects? Income poverty is the most popular indicator used (i.e. Backman and Ferrarini 2010; Casalone and Sonedda 2013; Figari et al. 2013; Marx et al. 2012; Salanauskaite and Verbist 2013; Stewart and Huerta 2009; Yakut-Cakar et al. 2012). Some studies use material deprivation: Nelson (2012) studies the effect of social assistance on material deprivation; Saunders and Wong (2011) use material deprivation to assess the adequacy of Australian benefits to single pensioners. Other studies use a combination of poverty indicators. Cancian and Meyer (2004) evaluate a social assistance reform in Wisconsin using different indicators of success (defined as a former client’s ability to avoid income poverty, material deprivation and dependence on governmental assistance) and find that different indicators appear to capture different aspects of a program’s success. Chzhen and Bradshaw (2012) evaluate the effects of income transfers on lone parent families and find that more generous transfers reduce child poverty but not material deprivation.

The above suggests there is a potentially relevant yet unanswered question: Do program evaluations show that a program can have different poverty reduction effects or do different poverty indicators evaluate a program’s poverty reduction effect differently? Departing from the view that a program evaluation ought to measure a program’s effect, I investigate the degree to which poverty indicators confound the evaluation of a program’s poverty reduction effect. I compare two popular poverty indicators, income poverty and material deprivation, in the context of income transfer programs across six European countries. In these countries, income transfers are a key component of poverty reduction policies (Barr 2012) and their governments use both income poverty and material deprivation indicators to monitor poverty (Marlier et al. 2007). Acknowledging that transfer programs may have goals other than poverty reduction (Barr 2012), I focus here on the targeting performance of income transfers defining the poorest 20 % of the population as the target group (Atkinson 1998; Coady et al. 2004). I ask two questions: (1) If only one poverty indicator is used, does it matter if that is income poverty or material deprivation? (2) Does the evaluation change when using both indicators? I answer the first question by comparing the targeting performance of two target groups namely those who are income poor and those who are materially deprived. I answer the second question by comparing three target groups: those who are income poor and materially deprived; those who are only income poor; and those who are only materially deprived. To assess a transfer’s targeting performance I compare program indicators between the target group and the non-target reference group: a transfer is successful when the target group is more likely to receive the transfer (coverage); when they receive larger transfers (generosity); and, when they receive a higher share of total benefit expenditures (distribution).¹

The analysis compares three transfer categories in six countries. It includes transfers for which poverty reduction is often an explicit policy goal: social assistance, housing and family allowances. It includes six European countries with similar resources but different welfare states: Germany (DE), France (FR), Ireland (IE), Netherlands (NL), Sweden (SE) and the United Kingdom (UK). This sample thus offers a desirable source of cross-national variation in transfer programs (different design and implementation) while it excludes a non-desirable source (different program effects due to (large) differences in fiscal resources and implementation capacities). The operationalization of poverty indicators follows the EU methodology (i.e. Guio 2009; Marlier et al. 2007) with the exception that I define the target group as the poorest 20 % of the population. Using the official EU thresholds would artificially increase the level of disagreement between poverty indicators because the thresholds embed different benchmarks (i.e. national median income versus a European deprivation threshold). I use the 2007 wave of the EU Statistics on Income and Living Conditions (EU-SILC) because I compiled detailed qualitative information about the transfer programs included in this wave, which facilitates the interpretation of the quantitative results.

Poverty measurement issues challenge the identification of the target group. This paper shows that this challenge leads to a downward bias in the estimated targeting performance of income transfers. It further shows that combining information from different indicators results in a more nuanced assessment of income transfers.

Section 2 delineates a theoretical framework for assessing how poverty measurement issues challenge the practice of evaluating targeting performance. Section 3 explains the methodology and Sect. 4 presents the results. The conclusion discusses the implications for program evaluations more generally.

2 Measuring Poverty and Measuring Poverty Reduction

A process evaluation is a first step in measuring whether a program has the intended effects (Mertens and Wilson 2012) and, when it comes to assessing poverty reduction, a key element of a process evaluation is to assess whether a program reaches its intended beneficiaries i.e. the poor (Coady et al. 2004). Following Atkinson’s (1998) definition of vertical efficiency, a program is targeted successfully when poor persons receive transfers and non-poor persons do not. Not reaching someone in the target group reduces a program’s effectiveness. Distributing resources to someone who is not in the target group, reduces a program’s efficiency and effectiveness (assuming a fixed budget). This is illustrated in Fig. 1a. The poverty indicator identifies groups A and B as poor. Yet, the program reaches groups A and C. The program can increase efficiency and effectiveness by shifting resources spent on group C to groups A and B.

Fig. 1

Measuring targeting performance. a One poverty indicator. b Two poverty indicators Notes: Designed by author, inspired by Atkinson’s (1998) definition of vertical efficiency

The concept of vertical efficiency forms the basis for evaluating a program’s targeting performance but it abstracts from a range of poverty measurement issues that result into a substantial disagreement about the target group, which may confound the evaluation. In the poverty measurement literature, poverty indicators are classified as monetary or non-monetary, direct or indirect, and resource-based versus outcome-based (Alkire 2008; Ringen 1988; Townsend 1979). An income poverty indicator focuses on, for many, a key resource to finance material wellbeing and is thereby a monetary, indirect, resource-based indicator. A material deprivation indicator focuses on an adverse material outcome resulting from a lack of financial means and is thus a non-monetary, direct, outcome-based indicator.

Not only are income poverty and material deprivation indicators based on different poverty definitions, each indicator also has its own specific measurement errors. Disagreement between poverty indicators thus arises for conceptual and measurement reasons. An income poverty indicator may underestimate poverty because it may not identify persons that are poor due to debts or higher needs (Pressman and Scott 2009; She and Livermore 2007). It may overestimate poverty because it mistakenly identifies persons as poor because they have alternative resources (Brandolini et al. 2010; Kalil and Ryan 2010), access to subsidized public services (Paulus et al. 2010), or their income is underreported (Rendtel et al 2004). A material deprivation indicator may underestimate poverty if persons are ashamed to reveal hardship (Breunig and McKibbin 2011) or have lowered their expectations as a response to cope with hardship (Guio 2009). It may over- or under-estimate deprivation if persons have a different prioritization of needs than the general population (McKay 2004).

These poverty measurement issues challenge program evaluations: Does a poor person not receive a transfer because the program fails or is that person mistakenly identified as poor? At first glance, using multiple poverty indicators appears to compound the analytical problem. Relative to Fig. 1a, the potential target group is larger in Fig. 1b: poverty indicator 1 identifies groups E, F, G, I as poor and poverty indicator 2 identifies groups E, F, H and J as poor. Yet, combining indicators offers an opportunity for triangulation as it yields four comparator groups: one that both indicators identify as not poor, one that both identify as poor and two on which they disagree. In Fig. 1b the indicators agree about the status of groups E, F, K and L.² They disagree about the status of groups G, H, I and J. Because the likelihood of a poverty measurement issue is lower when the poverty indicators agree, the estimated targeting performance is more reliable for these groups. Reversely, for the groups on which the indicators disagree it is more likely that poverty measurement issues confound the targeting estimates.

By comparing the targeting performance of three types of income transfers in six European countries, this paper empirically tests the degree to which these measurement issues confound evaluations of a program’s targeting performance. It should be noted though that income transfers are typically not exclusively targeted at the poor and that poverty reduction may not be the exclusive or primary policy goal. Even so-called last-resort transfers often explicitly include the ‘near poor’ to prevent the creation of welfare traps (Atkinson 1998). Moreover, escaping poverty is a gradual process rather than a discrete event providing another rationale for a gradual phasing out of benefits (Atkinson 1998). Finally, the poor may benefit from transfers even when poverty reduction is not the primary policy goal. Transfers such as old age pensions and paternity benefits provide insurance and help smooth consumption over the life cycle and thereby also reduce the risk of poverty during specific periods in a person’s life (Barr 2012). While this research abstracts from such considerations, they are extremely relevant when evaluating specific programs in specific country contexts.

3 Methodology

3.1 Countries, Data and Unit of Analysis

Included are countries with a similar average living standard but different welfare states: Germany (DE), France (FR), Ireland (IE), Netherlands (NL), Sweden (SE) and the United Kingdom (UK). Bukodi and Róbert (2007) classify Germany and France as conservative regimes, Ireland and the UK as liberal regimes and Sweden as a social-democratic regime. The Netherlands is a hybrid between a conservative and social-democratic regime (Kammer et al. 2012). This selection offers a desirable source of cross-national variation in transfer programs (different design and implementation of programs) while it excludes a non-desirable source (different program effects due to (large) differences in fiscal resources and administrative capacities).

The EU-SILC data provides cross-nationally comparable data (Online Appendix 1).³ I use the 2007 wave because I collected detailed qualitative information about the included transfer programs (discussed below). The household is the unit of analysis because a family’s living arrangements are central to the goal and design of family and last resort income transfer programs such as social assistance and housing. Choosing the individual instead does not affect the conclusions of this research (Online Appendix 5). All estimates take account of survey design (Goedemé 2013).

3.2 Poverty Indicators

The poverty indicators are constructed following the EU methodology with the exception that I define the target group as the poorest 20 % of the population, ranked according to pre-transfer wellbeing levels (i.e. Guio 2009; Marlier et al. 2007). Using the EU poverty thresholds would artificially increase the level of disagreement between poverty indicators as the thresholds embed different benchmarks (income: national median income, deprivation: a European deprivation threshold). This paper employs commonly used static simulations to estimate the pre-transfer wellbeing distributions and it is the first to present such estimates for material deprivation. Static simulations do not account for behavioural effects or for the complex interdependencies between individual transfer programs in a social safety net (Marx et al. 2013).

Pre-transfer income includes total disposable income minus social assistance, housing and family allowances (European Commission 2009, pp. 110–111). Thus, a transfer of 1 Euro decreases a household’s pre-transfer income by 1 Euro. As discussed in section two, identification errors arise because the indicator does not take account of debts, savings, subsidized public services, in kind social support systems or special needs and is likely to be measured with error.

Pre-transfer material deprivation is an estimate based on the post-transfer number of deprivation items that a household cannot afford. The post-transfer indicator is constructed from nine binominal variables indicating the ability to afford (1) to pay rent or utility bills, (2) to keep the home adequately warm, (3) to face an unexpected expense of about 800 Euro, (4) to eat meat, fish or a protein equivalent every second day, (5) a week holiday away from home, (6) a car, (7) a washing machine, (8) a colour TV, and (9) a telephone. This indicator simply counts the number of items that the household cannot afford. As discussed in section two, identification errors arise when chronic deprivation lowers aspirations, minority groups prioritize needs differently, and shame leads to underreporting.

The pre-transfer material deprivation indicator is estimated using a negative binomial regression model (Online Appendix 2). This estimator is used when the dependent variable reflects count data (i.e. the number of deprivations). It estimates how much an income transfer has reduced material deprivation controlling for household demographics, debt, ownership dwelling, education level, citizenship and economic status. The average income effect ranges from −0.39 for Ireland to −0.86 for France indicating that a 1 % increase in household income reduces the household’s number of deprivations by 0.39–0.86 %. Berthoud and Bryan (2011) find a similarly sized effect in their cross-sectional analysis. Because the pre-transfer material deprivation distribution is not entirely continuous, additional information is needed to identify the cut-off between the first and second population quintile. Households were additionally ranked relative to their ability to make ends meet (a variable taking values 1—very difficult—to 6—very easy—) and their pre-transfer income.⁴

Table 1 illustrates the lack of agreement on the target group: The indicators agree that 10–11 % of households are belonging to the target group but they disagree on 18 % of households. This disagreement is comparable to that found in the poverty measurement literature (see introduction).

Table 1 Summary statistics

3.3 Income Transfers

I analyze three income transfer categories for which poverty reduction is often an explicit policy goal: social assistance, housing allowances and family allowances. For social assistance and housing, the reference population is the whole sample; for family allowances it is households with children up to age 17. This is an approximation as family allowances are sometimes provided to children above that age while for adults above pension age there is usually a social pension instead of social assistance.

The EU-SILC variables for social assistance (hy060), housing allowances (hy070) and family allowances (hy050) detail the annual amount of transfers a household received in that category (European Commission 2009). Table 2 provides summary statistics. Because each variable typically includes multiple transfer programs and there is no specific information in the official documentation on which programs are included, more information is needed to interpret and assess the validity of observed differences between transfer categories and countries. I therefore contacted the national statistics offices requesting information on which programs are included in the data and documented the key characteristics of the programs by reviewing online program documentation. Table 3 provides a summary; Online Appendix 3 provides detailed information such as program names, maximum transfer and income levels.

Table 2 Summary statistics income transfers (in grey: 95 % confidence interval, +/−)

Table 3 Method used to target transfer (in bold: program is not included in EU-SILC data)

The social assistance variable includes transfers that are provided either through welfare offices or the tax system to households with resources below or modestly above the national minimum. The administrative method to identify the target group, always involves either an income-test or a means-test (income and asset-test). For Germany and France, non-contributory unemployment assistance is not included under social assistance but under unemployment insurance. For the Netherlands and Sweden, the income-tested tax credit programs are not in the EU-SILC’s social assistance variable. Coverage is either low (around 3–5 % for DE, FR, IE and SE) or higher (10–11 % for NL and UK) while transfers vary considerably being lowest in Ireland (€ 800, 4 % of income) and highest in the Netherlands (€ 7700, 51 % of income). Countries cluster as high coverage-high transfer (NL), high coverage-moderate transfer (UK), low coverage-moderate transfer (DE, FR and SE) and low coverage-low transfer (IE). The social assistance variable includes a mix of income and means-tested programs (NL, SE and UK), means-tested programs only (DE), or income-tested programs only (FR and effectively IE because of a lenient asset test).

The housing allowances variable includes either one program or, in the UK, two complementary programs using the same means-test. Housing allowances are means-tested (NL, UK) or income-tested (DE, FR, SE and effectively IE because of a lenient asset test). Coverage is very low in Germany (3 %) but very high in France (25 %) and Ireland (31 %). Most differences between transfers (7–15 % of income) are not statistically significant, except in the UK where they are significantly higher (30 % of income).

The family allowances variable includes many types of transfers: Some transfers are targeted at all families with children (universal), some are only provided to caregivers with a contributions history (contributions-based), while others additionally involve a means- or an income-test. Quite a number of programs are not included in the data or under a different category. Dutch income-tested tax credits are not included; UK maternity transfers are part of family allowances but are classified as sickness benefits for other countries. Coverage is very high for most countries (above 95 %) because each country has a universal transfer program. France’s lower coverage (79 %) likely arises because the universal benefit applies as of the second child. In Sweden (85 %) universal benefits apply for children up to age 16 (or 20 if in school). Transfers are 5 % of income in the Netherlands, 11–13 % for most countries and 18 % in Ireland. The high transfers in Ireland stand out. If income-tax credits were included in the EU-SILC’s family allowances variable, The Netherlands would likely join the other countries in the moderate transfer cluster.

The ‘All transfers’ variable sums up all three types of transfers and thereby illustrates the net performance of a country’s last resort safety net, or some key components thereof. Transfer programs may be designed as complementing or substituting other programs. For instance, UK households receiving social assistance may additionally qualify for housing allowances and child tax credits. In other cases, a transfer may be reduced if a household already receives another. The high coverage (34–71 %) is mainly explained by the prevalence of universal child benefits. Transfers jointly provide 17–26 % of the recipients’ income.

Tables 2 and 3 confirm that the design and generosity of the included transfer categories differs considerably across countries and transfer categories. In the Netherlands and the UK, a combination of income and means-tested social assistance and housing allowances provides relatively generous support while France relies more on housing allowances. In Germany and Sweden, social insurance schemes likely prevent many households from requiring last resort transfers. While all social assistance and housing allowances rely on some form of income or means-testing, each country has at least one universal family allowance program. This variation in transfers across national institutional contexts thus offers an excellent testing ground for assessing whether the choice of poverty indicator matters for evaluating transfer programs, irrespective of their specific design.

3.4 Indicators of Targeting Performance

To assess the targeting performance of income transfers I compare four program indicators between target and non-target populations: a program’s coverage, generosity (two indicators) and distribution of total transfer expenditures. The coverage indicator measures the percentage of households receiving the transfer. One generosity indicator measures the average transfer amount; the other measures a transfer’s average income share using a household’s disposable income. The distribution of total transfer expenditures is measured by the share of total transfers received by target and non-target populations. Following Atkinson (1998) and Coady et al. (2004), a transfer’s targeting performance increases when the target group (in comparison to the non-target population) is more likely to receive the transfer; when they receive larger transfers; and, when they receive a higher share of total benefit expenditures.

4 Results

Two questions guide the analysis of targeting performance of income transfers:

• 1. If only one poverty indicator is used, does it matter if that is income poverty or material deprivation? This perspective tests whether the common practice of using a single poverty indicator is risky in the sense that another indicator could judge a program’s performance differently.

• 2. Does the evaluation change when using both indicators? This perspective tests if there are analytical benefits to combining the information from two imperfect poverty indicators.

The first question is answered by comparing the targeting performance of two target populations namely the 20 % most income poor (Q1_y) and the 20 % most materially deprived (Q1_md) populations. For each target group, the non-target reference group are those who are not in that target group (Q2–5_y, Q2–5_md). The second question is answered by comparing three target populations namely those both part of the income poor and materially deprived target groups (Q1_{y md}), those only part of the income poor group (Q1_{y only}), and those only part of the materially deprived group (Q1_{md only}). For these three groups, the non-target reference group are those who do not belong to any of the poverty groups (NP).

Figure 2 summarizes the results for social assistance transfers in the UK. Tables 4.1 to 4.8 in Online Appendix 4 summarize the results for all transfer categories and countries and include 95 % confidence intervals. In the UK, the most important programs in this category are the income-tested Working Tax Credit and means-tested Income Support (Online Appendix 3). According to the coverage indicator, targeting performance does not differ between target groups Q1_y and Q1_md: in both cases 38 % of households receive social assistance (top chart, left panel). With only 4 % coverage in the non-target reference groups (Q2–5_y, Q2–5_md), social assistance transfers are clearly targeted to the least well off. The assessment changes considerably when combining the information from both indicators into three target groups: coverage among households in target group Q1_{y md} is much higher (58 %); coverage in the other target groups Q1_{y only} and Q1_{md only} is much lower (13 %); coverage in the non-target reference group (NP) is somewhat lower (3 %, top chart, right panel).

Fig. 2

Targeting performance: Social assistance in the UK Notes: 20 % lowest incomes (Q1_y), 20 % most materially deprived (Q1_md), 80 % highest incomes (Q2–5_y), 80 % least materially deprived (Q2–5_md), in Q1_y and Q1_md (Q1_{y md}), only in Q1_y (Q1_{y only}), only in Q1_md (Q1_{md only}), and not in Q1_y or Q1_md (not poor). Source: EU-SILC (2007), author’s calculations

These coverage rates show that it does not matter whether income poverty (Q1_y) and or material deprivation (Q1_md) is used to identify the target group. When both indicators are used however, the targeting performance of social assistance transfers in the UK appears much more effective and efficient when focusing on those population groups for which one is more certain that they do (not) belong to the target group (Q1_{y md} and NP). Figure 2 shows that this pattern also holds for the other targeting indicators in the UK, with the exception that differences in the generosity indicators of target groups are not statistically significant. Households in target groups Q1_y and Q1_md receive on average about 6000 Euro annually, significantly higher than the 4500 in non-target reference groups Q2–5_y and Q2–5_md. Target group Q1_{y md} receives 6138 while non-target reference group NP receives about 4000. While target groups Q1_{y only} and Q1_{md only} receive somewhat lower amounts than target group Q1_{y md}, these differences are not statistically significant. The income share of social assistance transfers does not differ much between targets groups Q1_y (33 %) and Q1_md (31 %) but is significantly lower for non-target reference groups (11–15 %). The income share of target group Q1_{y md} (35 %) is somewhat higher and that for the non-target reference groups is lower (24 and 13 %). Differences between the different target groups are not statistically significant. The share of total transfers indicator shows that target groups Q1_y and Q1_md receive 76 and 75 % of total social assistance transfers while each group comprises only 20 % of the population. Target group Q1_{y md} receives 65 % but only comprises 11 % of the population.⁵ Target groups Q1_{y only} and Q1_{md only} receive only 11 and 10 % and comprise only 9 % of the population. The non-target reference group NP receives 14 % but comprises 71 % of the population. Social assistance transfers in the UK are very much targeted to the least well-off.

The observed differences in targeting performance between different target groups are not specific to social assistance in the UK but apply more generally. Table 4 displays the coverage rates of the different target groups for each transfer category in each country. For target group Q1_y, coverage of social assistance is highest for The Netherlands and the UK (37 and 38 %), in Germany and France 19–23 % of this target group is covered while in Ireland and Sweden only 12–13 % is covered. Despite such large cross-national variation, coverage of target groups Q1_y and Q1_md is very similar while that of target group Q1_{y md} is significantly higher. To facilitate the cross-national comparison, another set of columns (R) expresses coverage as the ratio of target group coverage relative that of the non-target group. For instance, target group Q1_{y md} is 19 (Ireland) to 54 (Sweden) times more likely to receive social assistance than the non-target reference group NP. For social assistance and housing allowances, these coverage ratios are much higher for target group Q1_{y md} than for target groups Q1_y and Q1_md (except Ireland, which is discussed below). The coverage of family allowances does not differ between target groups but this is because each country has a universal transfer program in that category (Table 3).⁶ As most countries have a mix of universal, income-tested and means-tested family allowances, other targeting indicators show that family transfers benefit the target groups more than the non-target reference groups. The generosity and distribution indicators show again a considerably higher targeting performance for target group Q1_{y md} than for any other target group.

Table 4 Targeting performance: coverage (by transfer and target group)

These results suggest that it is advantageous to combine two conceptually distinct poverty indicators in program evaluations. Firstly, income transfers are judged as more efficient and effective in (not) reaching those for whom one is more certain that they (do not) belong to the target group. Secondly, for those groups where the risk of mistaken identification is higher, more analysis is needed to link targeting outcomes to either program failure or poverty measurement. Such analysis reduces the downward bias in program performance estimates caused by poverty identification mistakes.

The Irish case shows, however, that when the targeting of income transfers is relatively lenient the advantages to combining two indicators are lower. In Ireland social assistance and housing allowances are effectively income-tested because the asset component of the means-test is very generous: the value of owned real estate is not included in the test and the claw back of transfers for recipients with assets below 15,000 Euro is minimal (Online Appendix 3). Consequently, Table 4 shows that coverage of social assistance does not differ significantly between Q1_{y md} (17 %) and groups Q1_y and Q1_md (13 %). Coverage of housing allowances does not differ significantly between any of the five target groups (ranging from 44 to 53 %). Because a less strict targeting regime casts a wider net, it is less likely to exclude the least well-off (Notten and Gassmann 2008).

Another advantage to combining poverty indicators is that it can help assess whether a program’s specific design and implementation results in the erroneous exclusion of would-be beneficiaries and/or in (too much) leakage of resources to non-target populations (Coady et al. 2004). While such analysis requires program level data, which are unfortunately not available in the EU-SILC, the results in this paper suggest that such enquiry may be beneficial: both Fig. 3 and Online Appendix 4 show that it is quite common to find statistically significant differences in targeting performance between target groups Q1_{y only} and Q1_{md only}.

Fig. 3

Targeting performance: Share of total transfers (All transfers, by country and target group) Notes: ‘All transfers’ includes social assistance, housing and family allowances. Each bar reflects the share of total transfers received by a target group relative to that group’s population share. 20 % lowest incomes (Q1_y), 20 % most materially deprived (Q1_md), in Q1_y and Q1_md (Q1_{y md}), only in Q1_y (Q1_{y only}), and only in Q1_md (Q1_{md only}). Source: EU-SILC (2007), author’s calculations

Family allowances, for example, regularly distribute relatively fewer resources to those households who only belong to the 20 % most deprived (Q1_{md only}): this group receives lower transfer amounts in France, Sweden, The Netherlands and the UK and it receives a lower share of total benefits in France, the Netherlands and Sweden (Online Appendix 4). Targeted family transfers programs in these countries tend to rely more on income-testing rather than means-testing (Table 3 and Online Appendix 3). Relative to means-testing, income-testing of transfers is cheaper and easier, especially when income tax data can be used. However, a potential drawback of income-testing could be that it does not (sufficiently) account for the extra needs or burdens of households who despite somewhat higher income levels have higher levels of material deprivation. Such households are then less likely to be included or more likely to receive less generous support.

Moreover, depending on the country, target group Q1_{y only} receives a similar or different share of total transfer expenditures compared to target group Q1_{md only} (Fig. 3). In Germany, target group Q1_{y only} receives only 90 % of the transfers (All transfers) in comparison to what that group would receive under a flat rate universal scheme. Target group Q1_{md only}, on the other hand, receives 160 %. In France and Sweden target group Q1_{y only} (FR: 160 %; SE: 190 %) receives proportionately more than target group Q1_{md only} (FR: 110 %; SE: 100 %). In the other three countries both groups receive similar shares (IE: 130 %; NL: 80–90 %; UK: 120 %). Further research with program level data is needed to unravel what aspects of a program’s design and implementation drive such differences and whether they are consistent with the program’s goals.

5 Concluding Discussion

This paper showed that it is advantageous to combine two conceptually distinct poverty indicators, namely income poverty and material deprivation, to evaluate the targeting performance of income transfer programs.

Firstly, income transfers are judged as more efficiently and effectively targeted when focusing on those target and non-target populations on which the poverty indicators agree. Effectiveness increases because fewer members of that target group are excluded. Efficiency increases fewer resources are spent on the non-target group.

Secondly, for those groups where the risk of mistaken identification is higher (those in the target group according to one poverty indicator but not the other), more analysis is needed to link targeting outcomes to either program failure or poverty measurement. Such groups may be part of the target group (because they may have higher needs, debt repayments or other obligations) or not (because they may have alternative resources or access to subsidized public services). Two research strategies seem particularly promising. Firstly, in line with Atkinson’s (1998) argument to favour weak targeting so as not to miss the near poor, the analyst can do a robustness check by expanding the target group (i.e. the poorest 25 % or by setting a more generous poverty threshold). Secondly, information regarding special needs, debts, access to public services or additional resources could be used to identify the target group. The advantage of the first strategy is that it requires no additional information.

Thirdly, combining poverty indicators can help assess whether a program’s specific design and implementation results in the erroneous exclusion of would-be beneficiaries and/or in (too much) leakage of resources to non-target populations. This paper showed that it is quite common to find significant differences in targeting performance between the two target groups on which the poverty indicators disagree (those only part of the income poverty target group and those only part of the material deprivation target group). Further research using program level data can unravel what aspects of a program’s design and implementation drive such differences and whether they are in line with the program’s goals.

In sum, this research questions the dominant practice of using only one poverty indicator, typically income, to evaluate poverty reduction effects of income transfer programs or any other programs that may reduce poverty. The differences in effects are substantial and may influence policy decisions as well as popular and political support for transfers in general. Combining poverty indicators improves the validity of program evaluations because triangulation enables a better separation between (potential) poverty measurement issues and the measurement of a program’s (potential) effects. It is therefore adviseable that, in addition to using a monetary poverty indicator such as income, other information regarding households’ alternative resources, needs or material well-being outcomes is used. Collecting material deprivation information can be a relatively simple and low cost way of doing that.⁷