Introduction

Assessing progress on the SDGs

The Sustainable Development Goals (SDGs) provide an evidence-based framework of targets and indicators to enable integrated national planning and reporting through until 2030. In adopting the SDGs, countries agreed to conduct regular and inclusive reviews of progress at national, regional and global levels (United Nations General Assembly 2015).Footnote 1 Key principles for the review process include that they track progress in implementing the goals and targets and that they are rigorous and based on evidence (United Nations General Assembly 2015).Footnote 2

Central to the national review process are the Voluntary National Reviews (VNRs) submitted annually to the High-Level Political Forum (HLPF) on sustainable development. Implementation of the SDGs commenced in 2016 and many governments have now reported on national progress in their VNRs. However, these official government-led reviews tend to be narrative in style and focus on institutional processes, consultations, and challenges rather than providing an evidence-based quantitative assessment of national progress against the SDG targets and indicators (Allen et al. 2018). While qualitative assessments of progress in the VNRs are important for transparency and sharing of experiences, it is also critical that regular quantitative assessments of progress towards national targets and indicators are undertaken. Early assessments of progress can highlight targets and indicators where a country is currently off-track, support target prioritisation and resource allocation, and enable ongoing adaptive management.

To partially fill this gap, an increasing number of countries are providing access to national data on SDG indicators through national open data portals, with at least 36 platforms in operation by early 2019 (The Center for Open Data Enterprise and Open Data Watch 2019). However, these portals generally focus on enabling access to official data on SDG indicators rather than evaluating national progress towards SDG targets and indicators.

Quantitative assessments of progress on the SDGs and targets have emerged as another complementary mechanism within the review framework for the SDGs. A broad range of assessments have been prepared in recent years at various scales (global, regional, national, sub-national) and by various actors (government and non-government). This includes the annual global assessment of progress on the SDGs led by the UN Statistics Division (United Nations 2018) as well as regional-scale assessments of aggregate SDG performance across groupings of countries (Allen et al. 2017; Eurostat 2017; United Nations Economic and Social Commission for Asia and the Pacific 2018). Independent international rankings on the SDGs are also published annually in the form of the SDG Index and Dashboards which provide an international comparison of country performance (Sachs et al. 2018; Schmidt-Traub et al. 2017; Sachs et al. 2017). National indicator-based assessments of progress towards SDG targets have also emerged as a complementary progress report to the VNRs (Government of Mongolia 2018; Mcarthur and Rasmussen 2017), while a range of sub-national progress assessments have also been published which provide rankings and comparisons across cities (Prakash et al. 2017) and provinces (Government of India 2018).

Despite the growing importance of such indicator-based assessments of progress on the SDGs, there is limited academic literature available addressing and evaluating the approaches and methods applied. It is therefore important that the methodologies and results of such assessments are published and evaluated to contribute to best practice and continuous learning in this important field.

Australia’s performance and the SDGs

Australia is committed to achieving the SDGs and submitted its first VNR to the HLPF in 2018 (Australian Government 2018b). Australia’s first review takes a narrative approach, addressing each of the SDGs with case studies and other analysis. Australia’s VNR does not attempt to systematically assess Australia’s progress towards the SDG targets and indicators, however, it is accompanied by an online SDG data platform which houses available Australian datasets and provides a point of access to data on official SDG indicators (Australian Government 2018a).

At the global level, Australia was one of 156 countries assessed in the 2018 SDG Index undertaken by the Sustainable Development Solutions Network, where it ranked in a modest 37th place with an overall index score of 72.9 (Sachs et al. 2018), slightly lower than its score of 75.9 in 2017 (Sachs et al. 2017). Australia ranked below the regional average score for OECD countries of 76.9 in 2018, highlighting that Australia is lagging behind other advanced nations in terms of progress on the SDGs.

As a member of the United Nations Economic and Social Commission for Asia Pacific (UNESCAP), Australian data was also included at an aggregated regional level in the Asia–Pacific SDG assessment report (United Nations Economic and Social Commission for Asia and the Pacific 2018). Australian data is also included in the annual global SDG report as part of the ‘Australia and New Zealand’ grouping (United Nations 2018).

In this paper, we explore the method and outcomes from an independent, evidence-based assessment of Australia’s national progress towards the SDGs (‘Australia’s assessment’). We first provide a brief review of the expert literature and recent SDG assessments and outline the approach and method adopted to assess Australia’s progress (“Methodology”). We then present a summary of the results from the assessment (“Results”) and explore the results through sensitivity analysis and comparison against Australia’s results from the SDG Index in 2017 and 2018 (Sachs et al. 2017, 2018). Through a discussion of these results (“Discussion”), we draw out some important lessons learned to inform future national and global assessments of the SDGs, as well as identify remaining gaps and priorities for further analysis and implementation of the SDGs in Australia.

Context for the research

The national assessment of Australia’s progress on the SDGs was undertaken as a component of the SDG Transforming Australia Project which is overseen by an independent cross-sectoral expert advisory group—the National Sustainable Development Council (NSDC). The NSDC includes senior representatives from business, civil society and academia in Australia.Footnote 3 The assessment responded to the outcome from the Australian SDG Summit held in 2016, which identified the need for a national baseline assessment of Australia’s progress on the SDGs as a top priority action (ACFID et al. 2016). The aim was to build upon previous and existing assessments and reporting initiatives, including the Sustainable Australia Report (National Sustainability Council 2013) and Measures of Australia’s Progress (Australian Bureau of Statistics 2014), to provide a current, comprehensive, relevant and expert assessment of Australia’s baseline performance against each of the SDGs.

Methodology

SDG assessment approaches and methodologies

Indicator-based assessments apply a data-driven analysis to assess trends and progress and interpret results in a format that provides clear messages for policy-makers, stakeholders and the public. They have been used extensively over the past several decades for reporting on a range of indicator frameworks and composite indices of relevance for sustainable development, many of which are reviewed elsewhere (Stiglitz et al. 2010; Böhringer and Jochem 2007; Singh et al. 2009; Mori and Christodoulou 2012; Managi 2016; Eurostat 2014; United Nations Economic Commission for Europe 2014; Dasgupta et al. 2015; Mayer 2008; Dahl 2012; United Nations Department of Economic and Social Affairs 2007; Hák et al. 2016; Wilson et al. 2007; Diaz-Sarachaga et al. 2018). International experience in the use of indicator-based assessments was also greatly influenced and expanded through national, regional and global reporting on the Millennium Development Goals (United Nations 2015). Monitoring sustainable development through indicators is an efficient way to condense complex data and dynamics into a manageable set of meaningful information which can more effectively inform decisions and direct actions (Astleithner et al. 2004; United Nations Development Group 2017).

The adoption of the SDGs has reinvigorated interest in indicator-based assessments of sustainable development, with a suite of new assessments evaluating progress at a range of scales (Sachs et al. 2018; Mcarthur and Rasmussen 2017; Prakash et al. 2017; Eurostat 2017). An official set of 232 indicators was adopted by the UN General Assembly in 2017 (United Nations General Assembly 2017) to assist with national reporting, providing a consistent framework to guide assessments. The Inter-Agency and Expert Group on SDG Indicators responsible for developing the official set used three tiers to classify indicators based on their level of methodological development and the availability of data at the global level (Inter-Agency and Expert Group on Sustainable Development Goal Indicators 2016). Less than half of the official indicators are currently classified as ‘Tier I’, meaning that they had an internationally agreed methodology and data regularly produced for at least 50 per cent of countries (United Nations 2019). As such, many indicators still lack methodologies and timeseries data. As a globally-applicable set, the indicators will also vary in relevance from county to country based on national circumstances and priorities (e.g. in terms of national poverty lines, educational or health standards, and geography). To address gaps and ensure relevance, assessments have included a combination of official SDG indicators and other international and national indicators.

The selection of indicators and assessment methodologies can be problematic, and often relies upon pragmatic and subjective choices. For example, indicators are often selected based on political prioritisation, perceived importance and/or data availability (Shen et al. 2011; Lehtonen et al. 2016; Zinkernagel et al. 2018). To address potential bias and subjectivity, it is useful to adopt selection criteria and involve multiple stakeholders and experts in the process (Bell and Morse 2013; Rowe and Frewer 2000). Criteria for selecting sustainable development indicators include that they are policy-relevant, reliable, measurable/clearly defined, simple/easily communicated, broad in scope, and limited in number (Wilson et al. 2007; Böhringer and Jochem 2007; Niemeijer and de Groot 2008; Sustainable Development Solutions Network 2015; Allen et al. 2017; Rametsteiner et al. 2011; De Olde et al. 2017). An emphasis on an effective process including transparent and well-defined procedures can ensure relevance and validity of an assessment (De Olde et al. 2017).

A range of different approaches and methods can also be applied to assess trends and progress (Eurostat 2014). In practice, there is no ‘best’ method as each has its strengths and limitations. Ultimately, the approach and methods applied will depend upon a range of factors, including the availability of clear targets and associated datasets and timeseries data, the purpose and scale of the analysis, the target audience, the reporting channels available and associated costs, and the overall size and complexity of the indicator set.

At a rudimentary level, assessments compare the observed evolution or baseline value of an indicator against its desired evolution or value, as determined by a numerical target or relevant benchmark. This comparison enables an interpretation of progress, which is often categorised into classes using pre-determined thresholds. By presenting the assessment results in a dashboard format using intuitive symbols (e.g. coloured arrows, traffic lights, weather symbols) or by aggregating the results at the goal or national level, the information contained in the results can be easily communicated and readily accessed (Allen et al. 2017).

Selection of targets and indicators for Australia’s assessment

The selection of targets and indicators for Australia’s assessment was undertaken by the NSDCFootnote 4 in consultation with thematic experts and contributorsFootnote 5 for each of the 17 goals. Prioritisation and selection of targets and indicators was based on consideration of a set of criteria, namely: importance for Australia’s development; relevance to the global SDG framework; and data quality, timeliness and reliability (official statistics, robust and statistically reliable, recent baseline values, adequate historic timeseries). The process was expert-based and iterative, and was facilitated through five meetings of the NSDC, consultations with thematic experts on each of the 17 goals, and two meetings held with project partnersFootnote 6 undertaken during the period from October 2017 to July 2018. A final proposed set of SDG targets and indicators was also submitted to the multi-stakeholder ‘Australian SDGs Summit’ held in March 2018 for broader feedback, which resulted in some minor revisions.

The SDG targets and indicators are inextricably linked, and the selection of targets ultimately influences the selection of relevant indicators. For this study, the preference was to first identify priority SDG targets of relevance and importance for Australia’s long-term sustainable development, and then explore potential indicator options drawing from the official SDG indicators as well as other national indicators. Figure 1 provides a decision tree which outlines the process adopted.

Fig. 1
figure 1

Decision tree for selecting targets and indicators for the assessment

Full size image

All targets used in the assessment were drawn directly from the 169 official targets of the SDGs. To identify potential gaps of importance for Australia, the official targets and indicators were cross-referenced against national indicators that were reported in the comprehensive Sustainable Australia Report (National Sustainability Council 2013). This enabled the NSDC to identify additional priority national indicators that corresponded to priority targets but were not well-represented in the official set of SDG indicators. All additional indicators were aligned directly with a priority SDG target to ensure consistency and maintain the integrity of the global framework.

Clear target values are also important for assessing and benchmarking national progress on the SDGs, however, in many instances the SDG targets use generic language and do not include specific, numerical target values. In such cases, it is anticipated that each country will set its own national targets guided by the global level of ambition and national priorities and circumstances (United Nations General Assembly 2015).Footnote 7 In practice, very few countries have yet attempted to set national SDG target values, and this remains an important gap in Australia and elsewhere. To complete the assessment, where possible, the NSDC and/or the project team set a numerical target value based on an existing Australian target value, international guidelines (e.g. WHO guidelines for air pollution), expert analysis of historic trend data and rates of improvement, or available proxy values including the target and threshold values adopted in the global SDG Index (Sachs et al. 2018) or data on top-performing OECD countries (e.g. average of top-five performing OECD countries).

In total, 144 indicators encompassing 86 priority SDG targets and spanning all 17 goals were selected for the assessment (Table A, Supplementary Information). While all indicators used in the assessment are linked to a specific SDG target and goal, differentiation is made between official SDG and national indicators. Three types of indicators are reported (Fig. 2):

Fig. 2
figure 2

Distribution of indicators by type

Full size image

  • Original SDG indicators (‘Original’): as per the official list of 232 indicators adopted by the UN General Assembly in 2017 to assist with national reporting. These comprised approximately 41% of indicators used in the assessment.

  • Alternative SDG indicators (‘ALT’): minor variations on the original SDG indicator where it was necessary to adapt the indicator to align with Australian datasets and reporting. These comprised 27% of indicators used in the assessment.

  • Complementary national indicators (‘NEW’): new national indicators selected to complement the set of global SDG indicators due to their high importance and relevance for reporting on a particular SDG target in the Australian context. These comprised approximately 32% of indicators used in the assessment.

Overall, the majority of indicators assessed (approximately 70%) were official SDG indicators or similar alternative indicators (Fig. 2). At the goal level, the number of indicators was not distributed evenly across the goals (Fig. 3), with a larger proportion of indicators for goal 3 (health), goal 8 (economy), and goal 16 (governance). This was due to several factors, including the availability of data, the perceived importance of the goal in the Australian context, and the uneven distribution of official SDG indicators for each goal. A comparison of the distribution of indicators used for the Australian assessment against the official list of SDG indicators shows some clear similarities (Fig. 3).

Fig. 3
figure 3

Distribution of indicators by goal: Australia’s assessment (left) and the official set of SDG indicators (right). Percentages in this figure represent the proportion of the total number of indicators corresponding to each goal. For example, in Australia’s assessment, Goal 8 included 9% of the total number of indicators (left figure), while in the official set of SDG indicators the corresponding number was 7% (right figure). Totals for both charts are 100%; small discrepancies are due to rounding

Full size image

Assessment approach and method

Review of recent experience with SDG assessments

A review of the expert literature as well as recent SDG and sustainability assessments was undertaken to formulate the methodology for the assessment, while taking into account the specificities for Australia’s assessment (Schmidt-Traub et al. 2017; Allen et al. 2017; Eurostat 2014, 2017; Australian Bureau of Statistics 2014; National Sustainability Council 2013; Mcarthur and Rasmussen 2017; United Nations Economic and Social Commission for Asia and the Pacific 2018).

The literature describes a range of methods which vary depending upon the availability of target values, international benchmarks and/or trend data. Different types of linear and non-linear assessment methods can be used to analyse trends (most commonly ‘compound annual growth rates’ or ‘CAGRs’), and normalised scales are also used to assess progress and compare results across goals and countries.

The level of sophistication of the method used in recent assessments is greatly influenced by the availability of numerical target values and long-term trend data. However, interpreting clear target values in the SDG context is complicated by the broad scope of the targets, differing scales (national, regional, global), coverage of multiple issues in a single target, and ambiguous language (Allen et al. 2019).

To address the dearth of targets, previous studies have adopted different approaches. For example, an independent SDG assessment for Canada used a combination of SDG target values (where available), several existing national target values, and a majority of proxy target values which were set by the study authors (generally a 50% improvement on a baseline level) (Mcarthur and Rasmussen 2017). Other global and regional studies have used ‘top-performers’ and/or expert judgment to set target levels to assess progress (United Nations Economic and Social Commission for Asia and the Pacific 2018; Schmidt-Traub et al. 2017). Other assessments used methodological approaches to assess progress without target values, for example by relying solely upon an assessment of long- or short-term trends and the scale and pace of change over time (Eurostat 2014; Australian Bureau of Statistics 2014; Eurostat 2017). In other cases, international benchmarks (e.g. global averages) are combined with trend data to provide a more meaningful analysis of progress in the absence of target values (Allen et al. 2017, 2019). There is limited available literature assessing the strengths and weaknesses of different assessments and their methodologies (Wackernagel et al. 2017; Diaz-Sarachaga et al. 2018; Eurostat 2017).

Long-term trend data is important for assessing historical performance and determining a country’s current trajectory, and a period of 15 years of timeseries data is generally deemed adequate (Eurostat 2014). While an assessment of progress is possible without trend data by comparing a single data point against a target value (e.g. on a normalized scale), the assessment is more robust when trend data is also used.

Assessments can be undertaken at different levels of aggregation, ranging from individual indicators with no aggregation, to groups indicators (e.g. aggregated at the target or goal level), to composite indices and aggregated dashboards (e.g. combining results across all goals at the national level). The large number of goals, targets and indicators in the SDGs means that it can be difficult to make sense of an assessment and communicate the results without some form of aggregation—either at the goal level across a selection of targets and indicators, or at the national level across all goals, targets and indicators. Both composite indices (Schmidt-Traub et al. 2017) and simple aggregation (Eurostat 2017) methods have been used in recent SDG assessments to help synthesize the large number of individual scores. Such approaches can greatly assist with communication of results and in stimulating public interest and debate.

Approach and method for Australia’s national assessment dashboard

For the purposes of Australia’s assessment, four different methods were applied to cater for differences in the availability of target values, trend data and benchmarks (Table 1). In line with international practice, the period of the analysis was set from 2000 (or first available year after 2000) to the latest year available (most commonly 2015 or 2016). In general, a minimum length of 10 years and a minimum number of three data points were required to complete the analysis. However, for indicators with only a recent timeseries, a short-term (< 10 year) trend was calculated based on a minimum of three consecutive data points.

Table 1 Methods applied for assessing trends and progress
Full size table

A dashboard approach was used to present the results from the assessment, adopting traffic light symbols for easy interpretation. In the dashboard, each indicator was assessed using one of the four methods described in Table 1. A set of rules associated with each method are used to assess Australia’s progress and allocate traffic light symbols: ‘On Track’ ( ), ‘Needs Improvement’ ( ), ‘Breakthrough Needed’ ( ), ‘Off Track’ ( ), or ‘Not Assessed’ ( ). The dashboard included a range of additional information to assist with interpretation of Australia’s progress and provide transparency to the analysis (Table A, Supplementary Information). Overall, a total of 133 indicators were assessed in the dashboard out of the full set of 144 indicators (i.e. 11 indicators were ‘Not Assessed’ due to limited data or where an assessment was not feasible or appropriate).

The methods in Table 1 adopt similar approaches to recent SDG or sustainable development baseline assessments (Mcarthur and Rasmussen 2017; Eurostat 20142017), combining them in a novel way to provide greater flexibility to the assessment. Of the four methods, the preference was for the use of both time series data as well as a clear target value and target year to assess progress (Method A). In such instances, the actual rate of change of the indicator (based on the observed compound annual growth rate or CAGR) (1) was compared against the desired rate of change that would be required to meet the specified target value in the target year. This comparison calculated a ratio (R) by dividing the observed rate (CAGRO) by the desired rate (CAGRD) (2).

$${\text{CAGR}} = \left( {\frac{{X_{\text{t}} }}{{X_{\text{t0}} }}} \right)^{{\frac{1}{{{\text{t}} - {\text{t}}0}}}} - 1$$
(1)
$$R\; = \; \frac{{{\text{CAGR}}_{ 0} }}{{{\text{CAGR}}_{\text{D}} }}$$
(2)

Australia was considered ‘On Track’ for an indicator when the observed CAGR was greater than 90% of the desired rate needed to achieve a target by 2030. Similarly, Australia was considered ‘Off Track’ where an indicator value was below a target value and the observed CAGR was moving in the wrong direction. However, target values and time series data were not always available for each indicator, with only around 40% of indicators falling into this category (Fig. 4). In such cases, an alternate method was applied to complete the assessment.

Fig. 4
figure 4

Proportion of indicators assessed with each assessment method

Full size image

Where an indicator had available timeseries data but lacked a clear target, the assessment used international benchmark values (e.g. OECD averages) to complete the analysis (Method B). In such cases, Australia was considered ‘On Track’ where an indicator value was better than an available benchmark value and the trend was moving in the desired direction. A further 19% of indicators fell into this category (Fig. 4).

Where a time series was available but there was no available target or benchmark, trend data alone was used to complete the analysis, again using the observed CAGR (Method C). A rate of change of more than 1% per annum was considered a significant change, and the desired direction of the trend was used to interpret progress. Approximately 31% of indicators were assessed based solely on timeseries data (Fig. 4).

In cases where no trend was available, the assessment relied solely on a comparison of the latest data point against a target or benchmark value (Method D). This method was applied to the remaining 10% of indicators (Fig. 4).

In addition to the dashboard assessment, complementary assessment outputs were produced in the form of aggregated progress charts, timeseries data charts as well as written narratives that highlighted interlinkages across goals. This content was built into an online portal to enable easy access to the assessment results (National Sustainable Development Council, 2018). The charts also provide access to disaggregated datasets based on attributes such as indigenous status, gender, age cohort, socioeconomic status, and location. Each goal included a ‘leave on-one behind’ narrative that summarised disaggregated data for different disadvantaged groups.

Aggregation of results

Once the dashboard assessment had been completed at the target and indicator-level, a simple aggregation method was then used to synthesise and communicate Australia’s progress at the goal level for each SDG. Again, the method drew upon recent literature and assessments. The SDG Index developed a normalized scale (0–100) for each indicator using global data, with upper and lower bounds based on best and worst performing countries (Sachs et al. 2017). A simple average of these scores was then calculated at the goal level and across all goals to provide aggregated scores. United Nations Economic and Social Commission for Asia and the Pacific (2018) generates a similar normalized index for each indicator, this time using reference values for the year 2000 and target values for 2030 as the bounds for the scale. Simple averages are again used to aggregate scores for each indicator at the goal level.

It’s important to note that clear target or threshold values are needed to generate such indices, which was not an option for Australia’s assessment. Instead the approach to aggregation relied upon the allocation of scores on a linear scale and the calculation of simple averages as per Eurostat (2017). Pre-determined scores were allocated to each of the traffic light assessment categories, and then an average score (out of 10) was calculated for each goal based on the dashboard results.Footnote 8 While simple, the aggregation enabled clear and consistent presentation of Australia’s performance at the goal level, and comparison between goals. However, there are several limitations including that equal weight is given to each indicator which may not be realistic, and that the results can be subject to misinterpretation.

Comparison of results and sensitivity analysis

Finally, an additional comparative analysis was conducted to explore the sensitivity of the results to both the selection of indicators and the assessment methods applied. This was done by comparing results across different sets of indicators within our assessment (i.e. official SDG indicators versus alternative or new indicators), as well as by comparing our results against the results for Australia in the 2018 and 2017 SDG Index (Sachs et al. 2017, 2018).

Results

Dashboard assessment results

The complete results of the analysis are presented in a final assessment dashboard (Table A, Supplementary Materials) and web platform with accompanying charts and narrative analysis (National Sustainable Development Council 2018). Figure 5 provides a brief summary of the dashboard results. Overall, the assessment highlights mixed progress for Australia on the SDGs, with only around 35% of SDG indicators assessed as being ‘On Track’.Footnote 9

Fig. 5
figure 5

Summary of results from Australia’s SDG assessment. Coloured dots represent the assessment outcome for each individual indicator: ‘On Track’ ( ), ‘Needs Improvement’ ( ), ‘Breakthrough Needed’ ( ), ‘Off Track’ ( ), or ‘Not Assessed’ ( )

Full size image

At the indicator-level, this equates 32 individual indicators assessed as ‘Off Track’ (Table 2), representing areas where Australia has made little progress and has been moving in the wrong direction (on average) over the past decade or more. They can be considered as areas requiring urgent attention. A further 24 indicators were assessed as ‘Breakthrough Needed’ (Table 2). These represent areas where Australia is heading in the right direction, but with little possibility of reaching a target level or international benchmark based on current trends. These are additional areas where Australia would need to focus its efforts to rapidly accelerate progress over the next 10 years. When these indicators are combined, it can be seen that Australia has considerable work to do across 56 indicators, representing 42% of the 133 indicators that were assessed in the dashboard.

Table 2 SDG indicators assessed as ‘Off Track’ or ‘Breakthrough Needed’
Full size table

Aggregated assessment results

The aggregated results (Fig. 6) show that at the goal-level, Australia performs particularly well on goals relating to good health (Goal 3), quality education (Goal 4) and, to a lesser degree, life below water (Goal 14). The worst performing goals relate to inequality (Goal 10) and climate action (Goal 13). Other goals that could be considered lagging behind (i.e. with an average score < 6) are poverty (Goal 1) and energy (Goal 7). Across all 17 goals, the average score for Australia was 6.5/10.

Fig. 6
figure 6

Aggregated summary of assessment results, by goal. Numbers in the circle represent average progress scores out of a maximum of 10 for each goal across all of their targets/indicators, while the ‘average’ number in the centre represents the average score across all goals (i.e. 6.5/10)

Full size image

Results from sensitivity analysis

Comparison of official SDG indicators versus national indicators

Figure 7 compares the aggregated results of the assessment across all 133 indicators assessed in the dashboard (left), for the 57 official SDG indicators included in the dashboard assessment (middle), and the 76 alternative and new indicators (right). This highlights that when assessed based on only the official SDG indicators, Australia’s performance is considerably better on goals relating to governance (Goal 16), cities and communities (Goal 11), life below water (Goal 14), energy (Goal 7), infrastructure and innovation (Goal 9), gender equality (goal 5), and to a lesser degree, inequality (Goal 10), and food and agriculture (Goal 2).

Fig. 7
figure 7

Comparison of aggregated results with different indicator sets. Left diagram: results from all 133 indicators assessed; middle diagram: results from 57 official SDG indicators; right diagram: results from 76 ‘ALT’ and ‘NEW’ indicators

Full size image

As a result, Australia performs better overall on the official SDG indicators (average score of 7.2) than the alternative and new national indicators selected for the assessment (average score of 6.1). Australia’s stronger performance on the global indicators is perhaps to be expected from an advanced economy when compared against its developing counterparts. Nevertheless, the comparison serves to highlight that assessment results will vary depending on the indicators selected. A combination of global and national SDG indicators can provide a more balanced assessment for an advanced country, provided that the indicators are sufficiently ambitious.

Comparison of national results against the global SDG Index

While it can be difficult to compare results across different assessments, it can prove illustrative in terms of highlighting the influence of indicator selection and methods on the assessment results. Australia’s baseline assessment included 144 indicators in total, of which around 98 indicators (or ~ 70%) are official SDG indicators or slight variations on these (59 original and 39 similar). In contrast, the 2018 SDG Index assesses Australia’s performance across 111 indicators, of which 65 (~ 60%) were reported as being SDG indicators or closely aligned (37 original and 28 similar) (Sachs et al. 2018) (Fig. 8). The two assessments share around 42 indicators in common (Table 3), which represents less than one-third of Australia’s assessment. Based on this comparison, Australia’s national assessment has a larger number and proportion of official SDG indicators and greater coverage of the global SDG framework as well as national indicators of high relevance and importance for Australia.

Fig. 8
figure 8

Proportion of official SDG indicators included in Australia’s national assessment (inner ring) and the SDG Index (outer ring)

Full size image
Table 3 Comparison of assessment results at the indicator level: Australia’s national assessment versus the SDG Index 2017 (Sachs et al. 2017) and SDG Index 2018 (Sachs et al. 2018)
Full size table

The methods used also differ between the assessments. Australia’s national baseline assessment relies upon the use of long-term trend data combined with 2030 target values or international benchmarks (where available) to assess progress. In contrast, the SDG Index (Sachs et al. 2018) assesses Australia’s performance for each indicator based on a comparison of the most recent baseline value against a desired threshold value on a normalized scale. Thresholds were set by expert judgement and analysis of top-performing countries. Both studies report results in a dashboard format using a similar four-scale traffic light system as well as a radial chart of aggregated scores at the goal and national level, however, aggregation methods are considerably different. The 2018 SDG Index (Sachs et al. 2018) also analyses short-term trend data, however, this is assessed and reported on separately (i.e. it does not influence the allocation of traffic light symbols in the dashboard). These differences in methods will also influence results, but perhaps not as much as would be expected.

Table 3 provides a comparison of the dashboard assessment results for 42 common indicators used in both Australia’s national assessment and the SDG Index in 2017 and 2018 (Sachs et al. 2017, 2018). Overall, there is a high degree of consistency at the indicator level between the assessment results. This is perhaps surprising given the considerable differences in the methods applied. Small differences between the assessment results for some indicators are evident. However, these differences can be explained in most cases by the inclusion of trend data in Australia’s assessment. For example, Australia’s assessment of PISA scores (Goal 4) is yellow compared with green in the SDG Index (Sachs et al. 2018), which reflects the declining trend in scores in recent years—i.e. Australia’s PISA results are comparatively high, but they are moving in the wrong direction. Conversely, Australia’s assessment of government health and education spending is green compared with yellow in the SDG Index (Sachs et al. 2018) which reflects the improving long-term trend in Australia—i.e. if this trend continues the indicator is expected to meet the target value by 2030. Similarly, our assessment of the gender wage gap as orange instead of red in the 2017 SDG Index (Sachs et al. 2017) is a reflection of the improving trend in this indicator over time—i.e. Australia is heading in the right direction, but a breakthrough will be needed to achieve gender equality. These differences highlight the more nuanced analysis that is possible with the inclusion of trend data in the analysis, however, this timeseries data may not always be available, particularly for developing countries.

Comparing the aggregated results from both assessments also highlights several similarities and differences (Fig. 9). For both assessments, Australia performs particularly well on health (Goal 3) and education (Goal 4), while results are also similar for goals relating to food and nutrition (Goal 2), gender (Goal 5), water (Goal 6), and partnerships (Goal 17). However, Australia performs significantly worse in the national baseline assessment for goals relating to poverty (Goal 1) and inequality (Goal 10), and to a lesser degree, infrastructure and innovation (Goal 9), energy (Goal 7), economy and decent work (Goal 8), and cities (Goal 11). As a result, Australia performs better overall on the SDG Index (72.9 out of 100) (Sachs et al. 2018) compared with the Australian national assessment (6.5 out of 10). There are several reasons for these differences which are explored further in the discussion (“Useful ingredients for a robust national SDG assessment”), but primarily relate to differences in both the selection of indicators and methods applied for assessing progress. However, it’s important to note that there is no universally agreed methodology for an SDG assessment, and critiques of assessment methods are limited. Different studies will give different outcomes depending on the indicators and methods applied.

Fig. 9
figure 9

Comparison of the aggregated results for Australia’s national assessment (left) against the SDG Index assessment for Australia (right) (Sachs et al. 2018)

Full size image

It’s also important to note that the aggregated scores have limited tangible meaning. For example, an average goal score in our assessment of 6.5 out of 10 does not mean that Australia is 65% of the way to achieving the SDGs. Similarly, Australia’s score of 72.9 out of 100 on the SDG Index (Sachs et al. 2018) should not be interpreted in this way either given the limited coverage of indicators considered of high importance for assessing Australia’s progress on the SDGs. These aggregated scores are indicative only but are useful as they enable a comparison of performance across goals (in the case of our assessment) or across countries (in the case of the SDG Index). While they have their limitations, they provide a simple and powerful means to rapidly communicate progress and results to a broad range of audiences. However, they should be viewed with caution and interpreted in combination with the underlying data and assessment outcomes at the indicator level.

Discussion

The results from Australia’s SDG assessment and subsequent evaluation highlight a range of issues and lessons learned that are important and relevant for research and reporting efforts on the SDGs, both in Australia and globally. We discuss these briefly here.

SDG assessments: an important initial step in national SDG implementation

Our assessment highlights how the SDGs can provide a powerful tool for evaluating Australia’s progress across a comprehensive range of objectives and benchmarks that are important for a modern, sustainable society. The assessment results (Table A, Supplementary Materials) highlight that Australia has a healthy and well-educated population, and that life expectancy is among the highest in the world (Goals 3 and 4). The Australian economy has experienced sustained growth and a high rate of employment, and productivity and personal incomes have been rising (Goal 8). However, these results could be expected from an advanced nation such as Australia. They represent headline indicators that are often reported by politicians and the media.

Australia’s SDG assessment clearly highlights that strong progress in some areas is undermined by poor progress in others. Overall, Australia’s performance on goals relating to inequality (Goal 10), poverty (Goal 1), and decent work and economic growth (Goal 8) were lower than the SDG Index (Sachs et al. 2018) (Fig. 9). However, each of these goals was assessed with a majority of national indicators (rather than global indicators), and Australia generally performed worse on such indicators (Fig. 7). The inclusion of these national indicators was important to highlight areas of concern that would not have been adequately assessed through the global framework. For example, that low-income households reliant upon government allowances are falling below the national poverty line (Goal 1), underemployment is on the rise (Goal 8), and inequality has increased across several metrics, with a greater share of income and wealth going to the top 10% of earners (Goal 10) (Table A, Supplementary Materials).

Australia’s performance on environmental goals such as life on land (Goal 15), life under water (Goal 14), and sustainable consumption and production (Goal 12) was better on Australia’s assessment compared to the SDG Index. There are several possible reasons for this, including differences in indicators used as well as the assessment methodologies employed. A key challenge to reporting on biodiversity goals at the national level for a large country such as Australia relates to the broad spatial variability. Further disaggregated analysis is needed in these areas to better explore progress and challenges.

Finally, it’s important to note that while indicator-based assessments of the SDGs are an important step in initial SDG implementation, they are not sufficient in themselves as decision support instruments because they are generally unable to provide adequate insights and information on how multiple interactive forces have led to specific outcomes (Allen et al. 2018; Schmalzbauer and Visbeck 2016). Such assessments need to be complemented with analytical frameworks, tools and analysis that review policy and institutional settings, and enable interlinkages between targets and indicators to be assessed (Weitz et al. 2017; Nilsson et al. 2016). Recent studies are showing how SDG assessments can be combined with systems analysis of interlinkages and policy gap analysis to provide a robust prioritisation of national SDG targets (Allen et al. 2019). More advanced integrated modelling and quantitative analysis can then be used to select ambitious yet feasible target values, and explore policy and investment pathways towards achieving these targets under different future scenarios (Pedercini et al. 2018; Gao and Bryan 2017; Collste et al. 2017). Such research represents a logical next step for Australia.

Useful ingredients for a robust national SDG assessment

The results highlight that both long-term time series data for each indicator and clear numerical target values are useful ingredients for a robust SDG assessment. The use of both target values and timeseries data has clear advantages in terms of a more meaningful assessment of progress and gaps, as well as enabling the application of a single, consistent methodology across all indicators. While we faced challenges in both areas, the main gap for Australia related to the absence of target values rather than a lack of timeseries data. We were able to compile timeseries data for around 90% of indicators that were assessed, however target values were available for less than 40% of indicators.

Most of the SDG targets fail to set a clear target value for achievement by 2030 (United Nations General Assembly 2015), and Australia generally lacks national target values to fill this gap. Target values were therefore drawn from other sources, including international guidelines and thresholds used in the SDG Index (Sachs et al. 2018) (see “Selection of targets and indicators for Australia’s assessment”). Where a clear target value could not be readily identified, Australia’s assessment did not set generic target values (e.g. based on a proportional improvement by 2030, or top performers) as has been used in other studies (Mcarthur and Rasmussen 2017; United Nations Economic and Social Commission for Asia and the Pacific 2018). It is important that target values are both ambitious and achievable, and further research and broader consultation with experts and multiple stakeholders would be the best way to proceed. There remains considerable scope for further national target formulation and this should be a top priority for Australia to move ahead with SDG implementation.

This gap is not limited to Australia and is a common challenge for indicator-based assessments globally. Few countries have yet attempted to set a broad range of national targets corresponding to the 17 SDGs (Allen et al. 2018). This is partly because governments can be reluctant to adopt clear and measurable targets due to implications associated with transparent reporting, and the potential for political backlash if their implementation efforts fall short.

The 2030 timeframe of the SDGs encourages a shift to long-term thinking and visioning, based around a set of quantifiable, measurable, timebound national targets and indicators. Such a framework could be embedded into a national vision or long-term strategy which is currently lacking in Australia. The 86 priority targets identified in this study provide a critical resource and valid starting point for a national target-setting exercise, led by the government in consultation with experts and stakeholders.

In lieu of target values, we developed some novel methodological solutions to address gaps. However, each of the additional methods used has its limitations. For example, benchmarks provide a means to highlight areas where Australia is underperforming compared to its peers, but they do not assess the gap to a 2030 target value. Similarly, an assessment based solely on trend data does not take into account a country’s current performance on an indicator—i.e. Australia may be performing very poorly on an indicator despite an improvement in recent years. It is therefore difficult to assess the adequacy of progress without target values.

Assessment sensitivities: maintaining the integrity of the SDG framework and ensuring national relevance

With numerous assessments emerging at different scales (Sachs et al. 2018; Schmidt-Traub et al. 2017; United Nations Economic and Social Commission for Asia and the Pacific 2018; Allen et al. 2017; Eurostat 2017; Mcarthur and Rasmussen 2017), each with its own set of indicators and methods, there is a risk of inconsistent messaging and confusion. While this should not prohibit the emergence of different global and national SDG assessments, it is critical that some degree of comparability across assessments is maintained. This can be achieved by ensuring clear linkages to the official SDG framework, and by managing sensitivities associated with indicator selection and adoption of methods.

The method adopted for Australia’s assessment was developed following a review of the peer-reviewed literature as well as recent SDG assessment reports and guidelines (see “Methodology”). This is important to provide a sound methodological basis for an assessment and can build consistency and comparability across assessment results.

The comparison of our assessment dashboard results with those of the SDG Index (Sachs et al. 2017, 2018) (Table 3) demonstrated that different assessment methods generate some variation in results, however, this was lower than might be expected. This suggests that indicator selection plays a greater role in shaping the outcomes from an assessment than the assessment method applied. As such, the differences between the aggregated results from our assessment and the SDG Index (Sachs et al. 2018) (Fig. 9) may best be explained by the considerable variation in indicators used in each assessment.

The comparison of results across different types of indicators within our dataset also supports this conclusion. Variation in the assessment results between the official SDG indicators (United Nations General Assembly 2017) and other indicators clearly showed how indicator selection can significantly impact upon results (Fig. 7). These sensitivities need to be acknowledged and adequately addressed during the process of indicator selection, by maintaining a clear link to the official SDG framework and ensuring a transparent, expert-driven, and consultative process.

At the same time, it’s important that national SDG assessments also consider national relevance and priorities. It became evident during our assessment that not all official indicators are relevant to Australia, and that the official indicator framework did not capture all issues that are important for Australia’s long-term sustainable development. For example, Goal 13 on climate action fails to include any targets or indicators relating to the root cause of climate change (i.e. reducing greenhouse gas emissions); Goal 3 on health fails to include life expectancy and psychological distress; Goal 4 on education focuses on basic literacy and numeracy rather than more advanced capabilities in science, mathematics, and collaborative problem solving; and Goal 8 on the economy fails to capture underemployment and government and household debt. Further, a range of key indicators relating to inequality, cost of living, and affordability of housing and services are not well-represented across any of the goals.

These limitations to the official set of indicators could be partially explained by the extremely broad scope of the SDGs as well the technocratic (and politicized) process taken to develop them. There is a great deal of diversity in national priorities of different countries, and it is extremely challenging to capture this variability in a single, manageable global indicator framework.

Any SDG assessment will need to grapple with these issues and find a balance between national relevance and global comparability. All assessments that we reviewed had relied upon some combination of official SDG indicators and additional indicators. Given that the choice of indicators can greatly influence the results, it is important that a transparent, consistent, consultative and robust process is undertaken for indicator selection, drawing upon the advice of experts in each goal area. Without such a process, governments and analysts may choose to report solely on indicators for which they are performing well. Such cherry-picking would undermine the intent and transformative potential of the SDGs.

Some assessments that we reviewed did not maintain an explicit link to the specific targets and indicators in the official SDG framework when presenting the results (Eurostat 2017; Sachs et al. 2017). We maintained this link in our assessment through the consistent use of the official target and indicator numbering and coding, clearly identifying where we diverged from the framework. We feel that this is important to enable greater comparability across assessments and to be able to evaluate the degree of alignment of each assessment with the SDGs.

While global and regional SDG assessments provide a powerful tool for comparing progress across countries, the broad scope of these assessments means that national priorities and specificities can be overlooked. We would encourage all countries to undertake their own national-level assessment to complement the outputs of global assessments.

Finally, as this was an independent assessment of national progress on the SDGs (as opposed to an official government assessment), there was more flexibility in terms of expanding on the official set of indicators as well as the assessment methods applied. Governments may be hesitant to set clear targets and benchmarks and undertake transparent assessments of progress due to the potential for highlighting weaknesses or shortcomings in performance.

Disaggregation of assessment results: indigenous lens

The assessment of progress and scores presented in the results can best be interpreted as national averages of Australia’s performance on the SDGs, meaning that the dashboard results do not adequately capture disadvantaged groups within Australia that are lagging behind the national average. It’s important to note that the official set of SDG indicators calls for disaggregation of datasets at a range of levels, including by location (spatial), age cohort, indigenous status, gender, socioeconomic status, disability, etc. While disaggregated data is available for Australia for many of the indicators assessed, it is difficult to capture and communicate this disaggregation in the dashboard results and scores from the assessment. As such, the national dashboard results can fail to shed light on pockets of disadvantage. This is an important gap that needs to be reconciled.

To partially address this critical issue, the dashboard assessment was accompanied by over 220 individual charts, many of which provide a degree of disaggregation across a range of indicators.Footnote 10 The narrative summaries accompanying the assessment also included a ‘leave no-one behind’ assessment for each goal. However, it would also be useful to pass a ‘disaggregated lens’ across the results of the assessment to highlight these variabilities for select groups. For example, disaggregated data in our assessment highlights that the indigenous population lags significantly behind the national average for 11 out of 12 indicators reviewed.

It is important that these problems are not ‘skimmed over’ in national assessments of progress on the SDGs. An important next step for Australia would be to further disaggregate the results of the assessment for different disadvantaged groups in Australia as well as spatially (e.g. by State and Territory, city, and Local Government Area) to explore the distribution of progress across Australia. The increased granularity of this data would prove extremely useful for long-term integrated spatial planning in Australia. For example, a subsequent assessment of Australia’s progress towards the SDGs for each city and Local Government Area would provide a useful complement to the national-scale assessment, and could adopt similar assessment methods to ensure consistency. To do so, considerable efforts will first be needed to map existing spatial datasets to the SDGs framework.

Data quality and gaps

During the assessment we noted some inconsistencies between national data reported by official government sources, and data reported in international databases of the World Bank and United Nations. In all cases where these inconsistencies arose, we selected national data. It is important that analysts are aware of these inconsistencies and the need to improve the data quality and timeseries in international databases. The emergence of alternative data sources, big data, and real-time data provide many opportunities for expanding on traditional data sets, however, additional attention will need to be paid to data quality and comparability across spatial and temporal scales.

In terms of national data availability, Australia is a data rich country and in general the availability of data was adequate to undertake a comprehensive assessment. This was particularly the case for economic and social data. However, due to the large volume of data and the many datasets and providers, it was often very time consuming to locate and compile the necessary data. The development of a dedicated data portal housing SDG data by the Australian Government could greatly assist in future assessments of Australia’s progress, provided that it is populated with datasets and regularly updated.

Overall, the availability and quality of environmental data for Australia was not comparable to economic and social data. National datasets that corresponded to the SDGs were difficult to locate, there were many different national and international data providers, and time series were often dated. Improving the quality and accessibility of data on environmental SDG indicators should be a priority for government and research institutions moving forward, to ensure that future SDG assessments provide a balanced report card on the economic, social and environmental dimensions of sustainable development.

Conclusions

The SDGs have reinvigorated interest in indicator-based assessments of sustainable development, and it is important that the process, methods and results of such assessments are published and evaluated to contribute to best practice. Our study has shown that the results of such assessments are highly sensitive to both indicator selection as well as the methods applied. These sensitivities need to be acknowledged and managed by maintaining a clear link to the official SDG framework and ensuring a transparent, expert-driven, and consultative process. Comparison of results across different assessments can also help to explore the reasons for variations in results and address inconsistent messaging.

The results for Australia highlight that both long-term time series data for each indicator and clear numerical target values are fundamental ingredients for a robust SDG assessment. In Australia’s case, there remains a key gap in the availability of clear, agreed national target values and filling this gap should be a priority moving forward. The 86 priority targets identified in this study provide a valid starting point for a national target-setting exercise, led by government in consultation with experts and stakeholders. This could lead to a national vision or long-term 2030 roadmap for the SDGs which could provide a coherent framework for coordinating efforts, and which is currently lacking in Australia.

Finally, while global and regional SDG assessments provide a powerful tool for comparing progress across countries, we would encourage each country to undertake their own national-level assessment which can be tailored to national priorities and specificities. In Australia’s case, further disaggregation of results will also be needed over time to ensure that pockets of disadvantage in the community are not skimmed over and to ensure that no one is left behind.