Introduction

Organizations regularly face challenging resource allocation decisions in an effort to remain competitive and profitable. With limited time and competing demands on scarce funds, managers are apt to focus on those occupational health and safety (OHS) issues that are required by law, or ones that have been identified as clearly having an impact on the bottom line. Consequently, the notion that health and safety pays is an important message in getting some organizations and their managers to adopt OHS interventions once they have been proven effective. Not all organizations focus exclusively on financial concerns, and safety initiatives may not always bring a financial return to an organization. Nonetheless, complete information on the costs and consequences of an OHS intervention can be an invaluable input into the decision of whether or not to undertake a particular OHS initiative.

In many cases, managers are simply not aware of the financial burden of work-related injuries and illnesses or the financial returns from instituting workplace-based interventions focussed on improving OHS performance [35]. Consequently, the burden of avoidable work-related injuries and illnesses remains quite large [7, 28, 46]. In particular, musculoskeletal injuries comprise a substantial fraction of this burden [26]. Understandably, Leigh et al. [28], amongst others, concluded that resources devoted to primary and secondary prevention of occupational injuries and illnesses are insufficient compared to the burden they constitute in terms of health care costs in the U.S.

When managers and others do focus on OHS issues, it is often in an effort to reduce workers’ compensation costs. Yet workers’ compensation costs are a poor proxy measure of the total cost of work-related injuries and illnesses borne by the various stakeholders. Indirect costs such as accommodation costs, recruitment and training costs incurred for replacing injured workers, and lost home production are all costs not captured by workers’ compensation. These too are not a complete list of costs borne by organizations, workers, and society (examples of other costs include home care provided by family members, and the direct value of health lost by an individual). Also at issue with an exclusive focus on workers’ compensation claims is that only a fraction of work-related injuries and illnesses are reported [43]. Essentially, a broader perspective is needed such that a more complete list of costs is taken into consideration in an economic evaluation.

Lack of knowledge on the financial merits of OHS interventions is largely due to the fact that few studies of such interventions undertake an economic evaluation. In turn, this is due in part to the lack of competency of OHS researchers in the methodology of economic evaluation. These professionals tend to focus on the evaluation of the health-effectiveness of interventions rather than the cost-effectiveness, since their training generally does not include rigorous instruction in economic methods or financial analysis techniques. Furthermore, formulaic methods books, often designed for use in clinical settings, are difficult to adapt to workplace contexts. Undertaking economic evaluations in this arena can be difficult for a number of other reasons: the policy arena of OHS and labour legislation is complex, with multiple stakeholders and sometimes conflicting incentives and priorities; there are substantial differences in the perceptions of health risks associated with work experiences amongst workplace parties and policymakers; the burden of costs and consequences may be borne by different stakeholders in the system; there are multiple providers of indemnity and medical care coverage such that no one measure accurately captures the full cost of work-related injury and illness; industry-specific human resources practices (e.g., hiring temporary workers and self-employed contractors, farming out non-core activities) can make it difficult to identify all work-related injuries and illnesses; and in general the dearth of data available from organizations on costs and consequences can make it challenging and expensive to obtain good measures.

The above list of reasons might also explain why the quality of economic evaluations in the few OHS studies that do attempt to undertake one, is usually poor. Indeed, a common complaint in the assessments of the research literature on the economic evaluation of workplace interventions is that “well-designed and conducted evaluations of programme costs and benefits were nearly impossible to find” [35]. What is needed in the field is a collaborative effort between OHS researchers and health economists. This might allow for a more prominent role for economic evaluations in studies evaluating the merits of a workplace-based OHS intervention.

In this paper we review economic analyses in studies of workplace-based OHS interventions in order to report on evidence of their financial merits, assess the quality of application of economic evaluation methodologies in these studies, and provide practical advice for researchers on how to best improve on the application of these methodologies. Our initial intent was to synthesize the evidence on the economic evaluation of workplace-based interventions focussed on primary and secondary prevention of musculoskeletal injuries in office settings. Unfortunately, we found that there are few workplace-based OHS intervention studies that undertake such analysis, and even fewer still in the subset of this literature focussed on musculoskeletal injuries in office settings. Furthermore, evidence suggests that study design in some of this literature is weak [36] and, as noted, the level of sophistication of cost-benefit and cost-effectiveness analyses can be low [35]. Consequently, we have modified our objectives to some degree. First, we have expanded the range of OHS interventions included in our review to encompass workplace-based return-to-work programs and participatory ergonomics programs in any workplace setting that have undertaken an economic evaluation. Further, we have included studies that consider only the consequences in monetary terms, i.e., studies that have not undertaken a full economic evaluation but begin to consider the “bottom line” or the “business case.” Lastly, we have focussed our attention on reviewing the quality of the application of economic evaluation methodologies in workplace-based OHS intervention studies rather than on synthesising the evidence. We discuss the barriers and issues often encountered in undertaking an evaluation of the costs and consequences of an intervention, and propose practical recommendations to overcoming such barriers.

In the following section we describe, in greater detail, the search strategies employed to identify workplace-based OHS intervention studies that undertake an economic evaluation. The subsequent section reports on the findings of our literature search. This is followed by a discussion of the key issues we identified in our review. Lastly, we conclude with some suggestions on how to further develop the application of economic evaluation methodologies in such studies.

Literature search methods and sources

This literature review might best be described as an environmental scan, since we draw on several sources to identify studies of workplace-based OHS interventions. It is not a systematic review, though we do draw on several systematic reviews of workplace-based interventions. Specifically, we draw on systematic reviews underway or completed at the Institute for Work & Health. In developing our recommendations, we also reflect on our own experience with undertaking economic evaluations of participatory ergonomics interventions. By no means do we purport to have all the solutions, and in fact, there is much work to be done in the development of sound and practical practice guidelines for undertaking economic evaluations of workplace-based OHS interventions.Footnote 1

We identified relevant studies through five sources: (1) a structured MEDLINE search (encompassing literature from 1990 to 2005) undertaken as part of a systematic review of economic evaluation of workplace-based interventions focussed on primary and secondary prevention; (2) a systematic review of workplace-based return-to-work programs [16]; (3) a systematic review of participatory ergonomics interventions [40]; (4) a systematic review of computer-related office interventions to improve musculoskeletal and visual health [6]; and 5) a summary table of studies on office ergonomics [18]. The consideration of studies from sources two and three (return-to-work and participatory ergonomics interventions) were included under the premise that they are general enough to be applicable to many settings. Additionally, the dearth of intervention studies undertaking an economic evaluation lead us to broaden our scope.

The structured MEDLINE search consisted of four categories of keywords: (1) the type of study (e.g., cohort); (2) the setting (e.g., factory, office); (3) the outcome measure (e.g., musculoskeletal), and (4) the type of economic analysis (e.g. cost analysis, cost-benefit analysis). Category three consisted of terms for health outcomes related to musculoskeletal and/or upper extremity injuries and illnesses. At least one keyword from each of the four categories had to be identified in the title, abstract or MESH heading. The titles and abstracts review was completed by one reviewer, with periodic quality checks performed by a second reviewer. If a given reference’s title and abstract was deemed relevant (met the four criteria required for inclusion), the article was retrieved and reviewed in its entirety for inclusion. In addition, we reviewed the bibliographies of studies included in the review to identify incremental eligible studies.

The criteria for inclusion were that a study be about a workplace-based OHS intervention directed at primary or secondary prevention and have undertaken either a full economic evaluation or a partial evaluation. We use the term ‘full economic evaluation’ or ‘full evaluation’ to describe studies that consider both costs and consequences concurrently. We use the term ‘partial evaluation’ or ‘costing/outcome study’ to describe studies that considered only the consequences of the intervention in monetary terms. Lastly, we flagged studies that were in an office setting for the purpose of this review.

Details on the search and inclusion criteria of systematic reviews listed in sources two through four above can be found in the original studies. The review of workplace-based return-to-work programs included a review of the evidence on the economic evaluation of studies that undertook such an evaluation. Therefore, we did not need to review the full set of studies in order to identify those that undertook an economic evaluation, though we reviewed them to identify others that might contain costing studies. The other systematic reviews did not flag or synthesize evidence on the economic evaluation of the studies, so we reviewed their entire list of titles and abstracts for inclusion based on the criteria listed above.

Goggins [18] collected a series of case studies as part of a larger project to develop cost-benefit calculators for ergonomics interventions. The case studies have been organised in a table that identifies key characteristics of the studies, including whether they have investigated costs and/or consequences. We retrieved all studies that were available and reviewed them according to the criteria listed above for the purposes of inclusion in this review.

Findings

We found few workplace-based intervention studies that had undertaken a full economic evaluation, and fewer still that undertake a good quality one. Through our MEDLINE literature search, we identified 11 studies with full or partial economic evaluations, and two that were in an office setting. The two studies are: Lewis et al. [29] and Richardson [39]. Neither undertook a full economic evaluation.

The systematic review of workplace-based return-to-work programs [16] identified five studies out of 35 that undertook an economic analysis [2, 4, 24, 32, 47]. Three of these studies undertook a full evaluation. The systematic review of participatory ergonomics interventions [40] had seven studies out of 23 with economic analyses [3, 15, 23, 25, 27, 33, 34]. Three of these undertook a full evaluation. No studies of the 103 in the systematic review of musculoskeletal and visual health interventions [6] were identified as having economic analyses. Lastly, the table by Goggins [18] contained four studies with economic analyses [5, 11, 44, 45]. Three of these also had full evaluations. In total, we identified 12 additional studies from these sources (in addition to those identified through our MEDLINE search).

In total we found 11 full economic evaluations and 12 partial economic evaluations. Of the 23 studies, nine were undertaken in an office setting (five of the nine focussed primarily on upper extremities and there was one hospital based study focussed on upper extremities), 16 were on ergonomics interventions (eight on participatory ergonomics), and six on return-to-work programs (see Table 1 for details).

Table 1 Studies identified in the literature search
Full size table

The fact that so few economic analyses were undertaken amongst such a large number of workplace-based interventions magnifies the concern that economic analysis is rarely regarded as a critical component of an intervention study. Furthermore, more than half of the 23 studies considered only the consequences (costs savings) associated with a given intervention. The results of our search are in accord with the findings of other researchers who have reviewed this aspect of the OHS intervention literature [12, 21, 35].

In reviewing the studies, we identified a number of methodological shortcomings that warranted discussion. We did find several studies that were exemplary in their efforts to confront the challenges of undertaking a quality economic evaluation (for example, [11, 27, 32]). In the following section we review these and other studies in the context of the methodological issues listed below. As noted, the intent of highlighting these methodological issues is to provide insight into how to take the application of economic evaluation methodology further in future studies. Following is a summary of our methodological issues:

Study design and related issues

Study design

Studies are predominantly before-after studies with no concurrent controls, randomization or adjustment for confounders;

Study perspective

In most cases the perspective is not expressly stated, though that of the firm is often implied. Few studies considered a societal perspective or multiple perspectives;

Measurement time frame and sustainability

In many cases the measurement time frame is not sufficiently long, and in most cases there is no assessment of the sustainability of the program.

Measurement and analytic issues

Consideration of all important costs and consequences

The few studies that undertook an economic evaluation generally considered only a subset of costs and consequences;

Valuation of costs and consequences

In valuing costs and consequences, studies generally took measures at face value, without questioning whether their “sticker price” reflected their true value.

Analytical time frame and future costs and consequences

Future costs and consequences were rarely considered even though most interventions were ongoing.

Adjustment for inflation and time preference

In most cases there was no clear indication that monies from different years were adjusted for inflation or discounted for time preference;

Use of assumptions and treatment of uncertainty

Assumptions were often made with little justification or sensitivity analysis, particularly with regard to the effectiveness of the intervention and its implied savings.

Computational and reporting issues

Choice of summary measure

Most studies undertaking a full evaluation employ cost-benefit analysis. Few studies consider cost-effectiveness analysis and no studies undertook a cost-utility analysis;

Reporting issues

In many cases clear reporting of context, measures, and computations was seriously lacking, making it difficult to evaluate the quality of the analysis and generalizability of the findings.

A key message from our review is that an economic evaluation cannot be treated as a formulaic recipe. Rather, it requires a number of skills such as a sound understanding of the distinction between accounting and economic costs. This suggests a need for economic expertise in the multidisciplinary research team evaluating a workplace-based OHS intervention, since rote application from cookbook-style methodological text can lead to oversight of critical issues.

Discussion

Study design and related issues

Study design

Study design is closely tied to the evaluation of the effectiveness of an intervention, but it also bears on the economic evaluation, particularly with regard to capturing the magnitude of consequences. The merits of different epidemiological designs are well documented [31, 41]. Most would claim that randomized controlled trials are the design of choice, but non-randomized controlled studies and simple before-after studies can be well executed and provide invaluable information. Many of the intervention studies we identified that had economic analyses were based on a simple before-after study design without randomization or concurrent controls (see, for example, [3, 5, 29, 44, 45]). The main flaw we found with many of these studies was the attribution of all changes in the outcome of interest to the intervention, without accounting for confounding factors. An extreme example is provided by the simple before-after study by Thompson [45]. The author attributed observed productivity gains in an office environment entirely to a workplace-based exercise break program being implemented, even though it was noted that an incentive pay program designed to raise worker productivity was introduced shortly after the intervention began. Particularly with a before-after design, but also with other study designs, known or conjectured confounders that are not controlled for through randomization ought to be controlled for in some other way, for example, through multivariate modelling techniques. This is the approach taken by Lanoie and Tavenas [27]; the researchers controlled for several individual demographic, employment, and other contextual characteristics through regression modelling in order to assess the effectiveness of a participatory ergonomics intervention in reducing work-related back injuries amongst workers at a warehouse distribution centre. With some interventions, individual micro data may not be available, but similar techniques can also be used with departmental, plant or firm-level data.

Many macro-level factors merit careful consideration but are often ignored in intervention studies. In particular, the declining injury rates observed in jurisdictions in North America and other continents over the last decade or so may be an important factor to consider in the analysis, as the trend-level improvement in the outcome measure of interest can be easily, yet inappropriately, attributed to the intervention under consideration. Baxter and Harrison [3], for example, identified a downward trend in the percentage of operators reporting tendonitis over a 7-year period, four years of which were prior to the intervention, but they did not adjust for the impact of this downward trend on the outcome of interest.

Even with randomization and concurrent controls, there may still be pre-existing (pre-intervention) differences that bear on the outcomes of interest. As noted, controlling for pre-existing conditions and other confounders can be undertaken through regression modelling techniques. This was the approach taken by DeRango et al. [11], even though the subjects were partly randomized to control and intervention arms and concurrent controls were used. Specifically, the researchers took a three-step approach to ensure that they had correctly identified the effects of the intervention on the outcome: first, they had ‘partial’ randomizationFootnote 2 of subjects between the two intervention groups and a control group; second, they employed a quasi-experimental study design using a concurrent control group; and third, they used regression modelling to assess the consequences side of the intervention, controlling for a variety of contextual factors. In addition, the study design included data collection on dependent and independent variables prior to the implementation of the two interventions in order to control for pre-existing differences between treatment and control groups at baseline [11].

Study perspective

Most economic evaluations of workplace interventions we identified were conducted from the perspective of the firm. One exception is Loisel et al. [32] which took the perspective of the provincial workers’ compensation insurance provider.Footnote 3 In many studies the perspective was not explicitly stated, but simply implied to be that of the company, and in fact, we noted in some studies an inconsistent treatment of costs and consequences relative to their implied perspective (details are described in topic Valuation of costs and consequences).

There is a strong normative argument to justify the consideration and adoption of a broader, societal perspective. Specifically, the fact that there are multiple stakeholders affected by OHS issues (firms, workers and their families, unions, health care providers, insurer(s), society), suggests that costs borne by all the stakeholders ought to be considered in the analysis. This is the norm in other economic evaluation contexts (e.g., environmental impact assessment) where there are multiple stakeholders. Clearly, the introduction of workplace interventions can result in reallocation of resources, the consequences of which can reach far beyond the firm’s bottom line. A broad perspective does not preclude providing information on other perspectives. In fact, a disaggregation of costs and consequences would be invaluable, as it would provide insight into the distribution and burden of costs and consequences. A full accounting of the social cost of work injury and illness provides a better sense of the costs and consequences that are external to the firm. However, the relevant (as opposed to the ideal) perspective depends largely on the audience of the evaluation study (e.g., a firm or insurer).

Measurement time frame and sustainability

The purpose of most, if not all, OHS interventions is to have a long-lasting impact on the OHS performance of a firm. Yet, few studies have the luxury of a measurement time frame that is long enough to assess sustainability. A related issue is that OHS performance can vary dramatically from year to year in ways that are unrelated to the intervention, and a longer measurement time frame in conjunction with regression modelling techniques might help separate the noise from the true impact.Footnote 4 In some studies the measurement time frame was not even sufficient to observe a measurable and statistically significant impact of the intervention. This is particularly be an issue with long-onset conditions such as some musculoskeletal injuries. In such cases, the full magnitude of health impacts can be drawn from other information sources (e.g., published industry statistics, firm historical data—see for example [3]). Modelling of future consequences that may be significant but which extend beyond the measurement time period might also be considered. Some studies also extrapolated costs and consequences into the future, based on underlying assumptions about the longevity of the intervention (see for example [27]). If one takes this approach, assumptions should be well grounded, and their impact on the results should be through sensitivity analysis. Extrapolation of costs and consequences into the future is a highly debated issue, which we discuss under topic Analytical time frame and future consequences.

Measurement and analytic issues

Consideration of all important costs and consequences

As noted earlier, few studies undertook economic analyses, and those that did often undertook only a partial analysis. Specifically, we identified and included several studies that considered consequences in monetary terms, but did not consider any intervention costs (e.g., 29; 45, amongst others). These are not full economic evaluations since they stop one step short of a full evaluation.

Amongst the studies that undertook a full economic evaluation, many considered only a subset of costs and consequences [5, 39, 44]. In an evaluation of a workstation redesign and education program, Bradley [5] measured the costs of engineering evaluation and workstation redesign, but not employee time for training. In an office ergonomics intervention, Richardson [39] considered the cost of an ergonomics assessment and purchase of new equipment, but not the costs of time devoted to installation and training. Similarly, in an evaluation of an ergonomic workstation redesign and training program, Tadano [44] included the costs of therapist time, but not the costs of workstation redesign, training, and time for mini-breaks. Two studies that are quite thorough in their consideration of costs and consequences are Lanoie and Tavenas [27] and Loisel et al. [32].

Most studies that undertook economic analyses focussed on workers’ compensation claim costs and medical care costs [5, 29, 39, 44, 3]. However, using workers’ compensation claims as the sole or primary health outcome is generally inadequate. Workers’ compensation claims do not reflect the full extent of work-related injuries and illnesses. Many compensable injuries and illnesses go unreported [43], and others are not compensable. A related issue is that workers’ compensation premiums are simply transfers and do not reflect the full set of costs and consequences borne by different stakeholders (a more detailed discussion of transfers can be found in topic Valuation of costs and consequences). Studies might consider other measures of health and their associated costs, either through primary data collection or exploitation of other administrative data sources (e.g., first aid reports, modified duty, private indemnity claims, and short-term absenteeism).

Though some studies considered proxies for productivity such as absenteeism, only a few studies considered explicit measures of productivity or product and service quality. Identifying measures of productivity can be a challenge, though some studies were quite creative in exploiting administrative data sources within the firm (see for example [11, 34]).

Valuation of costs and consequences

Accurate valuation of costs and consequences can present some critical challenges that are not apparent at first blush. Three issues in particular stand out: identifying prices that correctly reflect the value of resources embodied in the costs and consequences under consideration, identifying prices that are consistent with the perspective taken; and measuring incremental costs and consequences attributable to an intervention, rather than the total costs incurred and consequences realized.

Regarding the first issue, prices should be based on the true worth of a good or service, i.e., the opportunity or resource costs embodied in it. If a well-functioning market exists for a good or service, then the “sticker price” (nominal market price) will accurately reflect this worth. In some cases a well-functioning market does not exist, and therefore the “sticker price” may not be a good measure. In such cases, economists attempt to estimate a shadow price to better reflect the opportunity or resource cost. A good example of this issue is presented in a study by DeRango et al. [11]. The study is of a new, highly adjustable chair introduced to a state governmental agency that collects sales taxes. The incremental taxes collected by workers who were provided a new chair and training were estimated to be $25,398 per worker per year. But taxes are simply transfers, and transfers often net out to zero at the societal level. In some cases transfers might be reasonable proxy measures for costs or consequences. For example, workers’ compensation costs might be a good measure for some of the productivity and health costs of work injury and illness.Footnote 5 In the DeRango et al. [11] study, the shadow price of the incremental increase in productivity of the tax collectors attributable to the intervention is likely much lower than the incremental taxes collected by them.Footnote 6 The investigators proxied this value with the percentage of increased taxes collected multiplied by the wage rate (based on the notion that the wage rate reflects the marginal product of a worker).

The issue of incorrectly using transfers to value consequences also arises in a study by Thompson [45]. The researcher undertook an evaluation of a paid exercise break program designed to reduce musculoskeletal strain in the data-entry office of a large California public utility. One of the functions performed by the data-entry staff was processing customer cheques. The more expedient processing of cheques resulted in substantial increases in interest income earned by the utility ($130,992 per annum) due to earlier withdrawal of funds from customers’ accounts. Once again, for reasons stated earlier, this measure may not reflect the true value of the productivity consequences of the exercise program.

The issue of inappropriate pricing might also arise when prices are generated within a firm for accounting purposes to allow for transfers and billing between departments. If these prices do not reflect true market prices, one can look to the market to identify costs for similar products or services. In some cases market prices can also be inappropriate when a competitive market does not exist, and a premium is paid for a good or service (e.g., specialty clinician services, new pharmaceutical products). In this case, adjustment needs to be made to the market price, possibly by using the prices of comparable products and services.Footnote 7

The second key issue, identifying prices that are consistent with the perspective taken, can arise when costs or consequences are shared by several stakeholders. This is particularly relevant for insurance expenses covered by third-party payers, e.g., workers’ compensation, medical care, and private indemnity insurance. Since many firms are not fully experience rated, the full cost of a claim will not be billed to the firm. As noted, workers’ compensation cost savings were one of the most frequently considered consequences in the studies we identified, yet few studies adjusted the wage replacement and health care costs borne by the insurer to reflect the fraction of these costs that would be billed back to the company in the form of higher premiums. One exception is Lanoie and Tavenas [27]. In this study the researchers consulted the firm’s financial analyst for an estimate of the costs to the firm of a day of workers’ compensation paid absence. If a societal perspective is taken, then the cost to an insurer will underestimate the true costs, since workers often share in the burden through co-payments, and both the worker and the employer incur expenses that are not covered by an insurer.

The third key issue is the need to consider incremental (marginal) costs attributable to an intervention rather than the total costs. At issue is the fact that some of the costs incurred during the intervention period would be incurred even if the status quo or comparator option were chosen. Since two or more programs are being compared in an economic evaluation, one is interested in identifying the incremental costs and incremental consequences attributable to the intervention of interest. For example, the total cost of the purchase of a workstation and related equipment may not be fully attributable to an ergonomics intervention if such purchases were already planned. Only the incremental cost of purchasing more expensive, ergonomically-designed workstation and equipment would be relevant. Most studies we identified attributed the total costs incurred during the intervention period to the intervention under investigation, though in a few studies researchers made the effort to identify the incremental costs. For example, in Lanoie and Tavenas [27] the researchers consulted the company’s equipment purchase policy to identify the incremental costs of equipment associated with the participatory ergonomics program. In Loisel et al. [32] the researchers made efforts to distinguish between standard expenses incurred by the insurer for treatment of back injuries from incremental expenses attributable to the particular back pain management intervention under study.

Analytical time frame and future costs and consequences

In many cases substantial costs and consequences may occur after the measurement time frame. The projection of costs and consequences beyond the period of measurement is a highly debated issue [14] and the validity of such projections rests on the quality of the data that it is drawn from. Clearly, there is a risk of overstating the benefits, particularly if there is the possibility of decreased effectiveness of the intervention over time. In such cases, a sensitivity analysis should be undertaken to assess the robustness of projections to the assumptions underlying them. Most of the studies we identified took the conservative “stop and drop” approach in which no costs or consequences are assumed after the measurement time period. One study, Lanoie and Tavenas [27], considered both the “stop and drop” scenario and a scenario in which consequences are realized for five years into the future. The researchers found that the participatory ergonomics intervention evaluated in the study was financially rewarding for the firm only in the second scenario. They did not test their extrapolation assumptions by considering different scenarios for the future, though a sensitivity analysis was performed on the discount rate.

Adjustment for inflation and time preference

Adjustment for time preference through discounting is critical if the costs and/or consequences of a study are realized over more than one year. If prices are expressed in nominal dollars from different calendar years, one should also adjust for inflation. Discounting is required for both costs and consequences, even if consequences are not measured in dollars. Many jurisdictions stipulate the discount rate at which public sector investments are to be discounted. For the private sector, firms may have their own specific rate used for project investments. The real discount rate recommended by several researchers for discounting constant dollars to a particular calendar year is 3% [19, 21, 22], but because a number of studies in the past used 5% it is recommended to consider both rates in an analysis. Undertaking a sensitivity analysis using a range of rates from 0% to 10% is also recommended [22]. In general, it is not a recommended practice to adjust the discount rate to allow for uncertainty or inflation. It is better to address these issues directly, through a sensitivity analysis in which different assumptions for the estimates are considered, along with a presentation of the reasons for the specific estimates employed in the sensitivity analysis.

Many of the studies we identified did not appear to adjust for inflation and time preference before aggregating monies from different time periods. For example, in Bradley [5] the researcher described an ergonomics program introduced to address cumulative trauma disorders (CTDs) in an office environment that was evaluated over a time frame that spanned several years. Prior to the program (1991--1992), health care costs for 5 cases of CTD totalled $63,628.98 (or $12,725.80 per case), whereas after the introduction of the program there were 35 cases over the 1993--1994 period with health care, engineering evaluation, and changes to workstations costs totalling $2,886.25 (or $82.46 per case). The researcher made no mention of adjusting health care and other costs for inflation, or discounting costs to a particular calendar year. In another office ergonomics study [29], the researchers evaluated the program over a period from 1990--1998. The key outcome variable was workers’ compensation medical treatment costs for VDT-related musculoskeletal claims, which were adjusted for inflation using the Consumer Price Index (CPI) for Medical Care to make the costs comparable across years. It appears that workers’ compensation wage replacement costs were also adjusted using the same CPI measure, which is a less appropriate adjustment factor for wage replacement benefits. A good example of adjusting for inflation is a study of a participatory ergonomics program in a warehouse distribution centre by Lanoie and Tavenas [27]. Direct and indirect costs from 1991 through 1998 were discounted to the 1989 calendar year with a nominal discount rate of 11.5% that incorporated both inflation and time preference. Analysis of the sensitivity of results to the discount factor was tested by recalculating the net present value (NPV) using 5%, 10% and 15% rates.

Use of assumptions and treatment of uncertainty

Economic analyses are often conducted in the context of incomplete information or uncertainty. This necessitates the use of proxy measures, information from alternative data sources, and invariably, the use of assumptions. Assumptions need to be well-grounded and justified. Sensitivity analysis is usually undertaken to test the robustness of results to variations in key assumptions. There are various approaches to dealing with sensitivity analysis across multiple dimensions (e.g., multi-way sensitivity analysis, threshold analysis, scenario analysis, probabilistic sensitivity analysis [8, 14]).

The above principles were not always followed in the studies identified. For example, in Baxter and Harrison [3] the researchers used data from published sources to identify an average duration of tendonitis-related lost-time claims in an office environment. A value of 30 days was used based on the average maximum days published by Reed [37]. Little justification was given for this choice, and no sensitivity analysis was undertaken. In Bradley [5], the researcher concluded that the intervention was effective based on the reduction in the severity (not incidence) of CTDs after the introduction of the intervention. The number of CTDs post-intervention, however, had actually increased seven fold (from 5 to 35 CTDs). The inherent assumption is that the 35 CTDs would have occurred without the intervention and each of them would have been as severe as the 5 pre-intervention CTDs, if not for the intervention. Once again, no sensitivity analysis was undertaken. In Lanoie and Tavenas [27], the researchers estimated the number of prevented work-related back injuries to be 6,11, and 15 during the three years of the active intervention phase based on regression modelling results. In extrapolating into the future, the researchers assumed that the number of prevented back injuries from the last year forward would remain at 15 per year. Their argument was that it is not possible for the number of avoided injuries to continue to increase at the same rate, since the total number of injuries would quickly fall to zero (which is not realistic). The positive net present value identified from the ten-year analytic time frame appears highly contingent upon this assumption, yet no sensitivity analysis was undertaken around it. The researchers did consider various discount rates in their analysis.

Computational and reporting issues

Choice of summary measure

Choosing between one of the three key types of economic evaluation (cost-benefit, cost-effectiveness, and cost-utility analysis) for a particular study should be based on the objective of the intervention and the question being addressed by the study. These in turn are influenced by the nature of the key outcome variable and the perspective being taken. For example, if the perspective is that of a private sector firm, cost-benefit analysis might be preferred if the key objective of an intervention is to reduce workers’ compensation costs through improved OHS performance. The three types of evaluation differ primarily in the measurement of the outcome (monetary metric for cost-benefit analysis, natural units for cost-effectiveness analysis, and utility metric for cost-utility analysis), and each has its strengths and weaknesses (see 13 or 22 for details). Cost-benefit analysis is by no means a gold standard for economic evaluation, since it can fail to adequately account for many aspects of the health consequences realized by an intervention. There are a number of issues that arise when placing a monetary value on health and life (see 12, 13; 22 for details). In fact, there is merit in considering more than one type of analysis in a particular study in order to provide insight into the different facets of an intervention. For example, one could undertake a cost-effectiveness analysis to better capture the health outcomes that are not readily translatable into a monetary metric (e.g., pain reduction, musculoskeletal disorders averted), and a cost-benefit analysis in which health outcomes are proxied through some monetary measure (e.g., workers’ compensation wage replacement costs, the wage cost of absences, reduced productivity, health care costs). This is the approach taken by Loisel et al. [32]. Most of the studies we identified with a full evaluation undertook a cost-benefit analysis, which is consistent with the literature on workplace-based interventions. A few studies undertook cost-effectiveness analysis and no studies undertook a cost-utility analysis.

Reporting issues

In order for a study to be useful to decision makers, the study must provide new and relevant information on issues they are confronting. Transferability of an intervention to other settings requires sufficient information to assess its applicability or generalizability. Information that assists with this process includes details about the context, the nature of the intervention, and the timing, magnitude, and the type, amount and variability of costs and consequences. At another level, clear and complete reporting facilitates a reader’s ability to assess the quality of a study, or simply understand what assumptions and calculations were made to arrive at a final value.

In several of the studies we reviewed, we found the reporting of context, measures, computational formulas, and assumptions were inadequate, making it difficult to determine how values were derived, evaluate the quality of the analysis, and assess the generalizability of the findings. Missing were items such as specifics about the nature of the intervention [29], sample size [3], details on how critical measures were calculated [25, 39], and whether a discount rate was used in aggregating monies from different calendar years [29]. There were three studies that provided quite substantial detail on many aspects of the analyses, specifically DeRango et al. [11], Lanoie and Tavenas [27], and Loisel et al. [32], though even these studies had important information missing. We discuss the findings of studies in greater detail below.

Summary of evidence

The studies we identified with full economic evaluations used different approaches to the computation and analysis of costs and consequences, making it difficult to compare results across studies. This suggests the need for a standard approach to computations, i.e., a reference case or gold standard. As noted, most studies used the cost-benefit approach. But even within this specific type of evaluation, methodology texts and studies present an array of computational possibilities. In cost-benefit analysis, the costs and consequences of alternatives are best expressed as the net benefit (loss) of one alternative over another. Other options include presenting the results as a cost-benefit or benefit-cost ratio (monetary costs divided by monetary benefits for the cost-benefit ratio and the reverse for the benefit-cost ratio), though sometimes there are issues regarding what values to put in the numerator versus the denominator [13, 22]. Other computational options are return on investment and pay-back period calculations. Each computational approach may have advantages and disadvantages, but a standard is needed if studies are to be comparable. This does not have to preclude the use of more than one approach. We recommend the NPV approach, which is the standard recommended by most methodology texts ([13, 22], amongst others). The final value is best supplemented with clear reporting of absolute costs and benefits and their breakdown by category and time period, as well as other relevant factors that give insight into the scope of an intervention (e.g., number of employees reached by the program, number of injuries).

Several of the studies we identified calculated a yearly per person/claimant benefit or cost-benefit/benefit-cost ratio [11; 29; 32]. At face value, this appears to be a simple way to factor out the scale aspect of a project and provides valuable information, but should be treated as an add-on to standard computational practices of computing the NPV of the intervention. For example, in Lewis et al. [29] described above, the researchers reported that per-claim costs decreased from $15,141 to $1,553 between the pre-intervention period of 1990--1994 and the post-intervention period of 1995--1998. Although the number of claims is provided, a NPV is not calculated and it is not possible to calculate it with the information provided.

The studies we identified with cost-benefit analyses found that the interventions under study were profitable (positive NPV or benefit-cost ratio greater than one) within the measurement or analytic time frames. However, these findings should be interpreted with a healthy dose of scepticism, given the methodological limitations and generally poor quality of these studies. Table 2 provides details on the design and findings of the studies that undertake economic analyses of interventions in office settings.

Table 2 Details of Office Intervention Studies
Full size table

Three studies stand out as exemplary in terms of their methodology [11, 27, 32]. These studies are not directly comparable,Footnote 8 but we elaborate on their findings for expositional purposes. In an evaluation of a return-to-work program for lower back pain [32], the researchers found that within the measurement time frame of 6.4 years mean follow up, the incremental NPV for the key experimental arm (a combined clinical occupation treatment which was compared to standard care) was $18,585 per claim (1991 Canadian dollars). The study was undertaken from the perspective of the public insurer. In Lanoie and Tavenas [27], the researchers found that a participatory ergonomics program implemented at a warehouse distribution centre of the Quebec liquor board resulted in an estimated 107 back-related injuries being prevented over a 9-year analytic time period. The intervention targeted roughly 90 workers, and the total intervention costs, including future projections, were $227,318.31, while the NPV (including future projections) was $187,700.79 (1989 Canadian dollars). The perspective in this study was that of the distributor. In DeRango et al. [11], the researchers reported on an ergonomics intervention directed at improving health and productivity outcomes at a State Department of Revenue. The intervention consisted of an ergonomically designed chair in conjunction with ergonomics training. The researchers found that the intervention was effective in reducing pain (a proxy measure was used for ‘freedom from pain’). The per-person intervention cost was estimated to be $1,032 (U.S. dollars), with an upper-bound estimate of the benefit-cost ratio of 24.61 (benefits based on incremental taxes collected) and a lower-bound estimate of 2.13 (benefits based on percentage increase in productivity times the wage rate). The perspective in this study was that of the Department of Revenue.

Conclusions

The two main findings from this review are that: (1) few workplace-based intervention studies undertake economic analyses, and (2) the intervention studies that do undertake economic analyses present a mixed bag of methodological approaches and quality. A focus on office-based interventions directed at preventing work-related upper-extremity injuries further limits the number and quality of analyses available for synthesis. These findings are not unique to the subject matter at hand; researchers that have reviewed other subsets of the OHS literature on workplace interventions have had similar findings (35, 21, amongst others). In our review, we have attempted to summarize the key methodological shortcomings, using examples from the literature identified, in an effort to provide insights and prescriptions on how to advance applications in this field. Some of the issues identified may be due to limited expertise in economic evaluation methodologies, others may be associated with the low priority given to economic analysis by evaluators, and yet others relate to practical limitations of the workplace context that can prove to be a challenging hurdle. Below we discuss practical considerations drawn from our own experiences.

An economic evaluation, like an effectiveness evaluation, rests on the quality and appropriateness of the data employed in the analysis. To ensure that appropriate data are collected, an economic evaluation ought to be built into an intervention study at the planning and development stage. Economic evaluation should be seen as an integral component of a workplace-based intervention study, rather than an add-on. First, an economic evaluation should build on the effectiveness evaluation, since data needs will vary depending on the key question being addressed and the nature of the study design. Second, with the economic evaluation planned for at the outset, the adequacy of the data availability through secondary and administrative sources can be assessed, and appropriate primary data collection strategies can be designed to fill the gaps. A multifaceted intervention evaluation exercise of this sort calls for a multi-disciplinary research team effort, an integral part of which is a researcher with expertise in economic evaluation.

We have experienced first hand the challenges of attempting to begin an economic evaluation of an intervention after the measurement time frame has passed, and can attest to the almost insurmountable barriers that this approach can impose. Most notable are the loss of contact and commitment with key workplace parties, and the sobering experience of fumbling through a jumble of data collected with other purposes in mind. Even basic intervention costing information, such as worker time commitments to training and other intervention activities, if not collected on a timely basis will require resorting to crude measurement exercises based on recall or averages drawn from the literature.

Given the reality of financial constraints faced by workplaces in a competitive, global market, an economic evaluation can reinforce the merits of OHS investments. Not all organizations focus exclusively on financial concerns, and not all effective OHS interventions bring positive financial returns. Nonetheless, an understanding of the costs and consequences of the set of intervention options provides invaluable information when one cannot do everything that has been proven to be effective.