Introduction

Immunization of populations against harmful diseases by the use of vaccines has been one of the most beneficial endeavors of modern medicine and public health. The economic returns and health gains associated with vaccination are substantial (Whitney et al. 2014). In the early 2000s, the worldwide vaccine market was estimated to account for approximately $6 billion (Gréco 2002; Whitehead and Pasternak 2002). To maintain the effectiveness and value of these vaccines, public health researchers have focused on ways to reduce risk to vaccines due to temperature fluctuations in the vaccine distribution system (Chojnacky et al. 2009; Kristensen et al. 2011) and in storage at vaccine provider locations (Bell et al. 2001; McColloster and Vallbona 2011). The most important risk to vaccines from being subjected to inappropriate temperatures is the loss of effectiveness, which can be due to exposure to either heat or freezing temperatures, depending on the type of vaccine (Kumru et al. 2014; Matthias et al. 2007).

Events where vaccines are exposed to temperatures outside the recommended range are often referred to as temperature excursions. Temperature excursions are identified as a potential cause of vaccine wastage in a variety of settings (Mukherjee et al. 2004; Setia et al. 2002). In response to the risk posed to vaccine supplies from temperature excursions, a number of interventions have emerged to promote and maintain temperatures in vaccine distribution, which may also be referred to as maintaining the cold chain. Research efforts have focused on manufacturing more thermally stable vaccines (Schlehuber et al. 2011), improving transportation and distribution of vaccines (East and Smale 2008), and applying different methods to pack and arrange vaccines within storage units (Chojnacky et al. 2009). There have also been investigations into the effects of ambient environmental factors such as ambient air temperature or power outages (Chojnacky et al. 2009) and into improving temperature monitoring in vaccine storage units (Fisun et al. 2016; Kartoğlu et al. 2010; McColloster and Vallbona 2011).

Continuous temperature monitoring (CTM) consists of using technology to record temperatures in storage units at regular intervals. These temperatures are recorded and saved electronically in a digital data logger (DDL). The CDC currently recommends the use of a continuous temperature monitoring device (Centers for Disease Control and Prevention 2018), and began requiring these devices for storage units containing federally funded Vaccines For Children vaccines in January 2018. The World Health Organization (WHO) also recommends the use of CTMs (World Health Organization 2015). However, in specific situations CTM devices may not be feasible. As an example, in Haiti, several challenges were encountered after DDLs were implemented, including spans of time when the DDL did not collect data or communicate data to the central data collection hub (Cavallaro et al. 2017). In addition to the use of CTM, both CDC and WHO also recommend daily manual temperature checks, which we refer to as non-continuous temperature monitoring practices. Non-continuous temperature monitoring helps to actively monitor temperatures and this serves as a backup or supplement to any data collected with DDLs.

The focus of this study is to assess the potential benefits of different types of non-continuous temperature monitoring practices. Findings from this study would be relevant to vaccine provider locations where non-continuous temperature monitoring practices are being used as a complement to CTM, but would be especially important for vaccine provider locations where CTM is not yet feasible. Using a simulation methodology, we evaluated the ability of different non-continuous temperature monitoring practices to detect the occurrence of temperature excursions. To our knowledge, no thorough investigation has been conducted that quantifies the relative strengths of these temperature monitoring practices for their ability to detect and record temperature excursions. Our study aims to fill this knowledge gap by providing an assessment of several temperature monitoring practices with a simulation model.

Background on vaccine storage and handling practices

In spite of substantial efforts to improve the vaccine supply chain across the world, ubiquitous and effective vaccine storage and handling practices remain elusive in many parts of the world (Ashok et al. 2017; Hanson et al. 2017). For example, improper storage and handling practices were identified, among other violations of proper vaccination procedures, at a workplace influenza vaccination clinic in New Jersey (Taylor et al. 2015). Freezing temperatures in vaccine storage units have been linked to pertussis incidence in Texas (McColloster and Vallbona 2011). Specific deliveries of live attenuated influenza vaccine for the 2013–2014 influenza season appear to have been subjected to excessive temperatures, and these excessive temperatures were found to be associated with low vaccine effectiveness for particular influenza strains (Caspard et al. 2016). More globally, low vaccine effectiveness, which may have been related to cold-chain inadequacies or waning immunity, was acknowledged as a contributing factor in a 2014 measles outbreak in the Federated States of Micronesia (Hales et al. 2016). A literature review looking at vaccine cold-chain studies found that in lower income countries, as many as 19% of vaccine shipments and 37% of vaccine storage units experienced temperatures below recommended ranges (Hanson et al. 2017). Another review of cold-chain effectiveness in developing countries identified the “lack of the latest technology or optimal equipment” as a key issue that has been limiting cold-chain performance (Ashok et al. 2017). In more specific geographic regions, vaccine cold-chain management has been studied by public health researchers in the context of Cameroon (Yakum et al. 2015), Pakistan (Buledi et al. 2017), Tunisia (Lloyd et al. 2015), and Haiti (Cavallaro et al. 2017).

Non-continuous practices consist of vaccine provider staff checking storage unit temperatures from a temperature monitoring device at one or more times during the workday to ensure the storage unit temperature is within the recommended range. By contrast, CTM utilizes DDLs, which are small electronic devices, typically weighing less than a pound and about the size of a scientific calculator, connected to a temperature probe that is usually suspended in buffered material such as a glycol solution. Temperature data is collected by DDL at small time intervals, such as 5, 10, or 15 min intervals. The CDC Vaccine Storage and Handling Toolkit recommends these intervals be no longer than 30 min (Centers for Disease Control and Prevention 2018). DDL data is either transmitted or uploaded to a computer where past temperatures of a storage unit may be inspected, analyzed, or stored for later review. Many DDLs also feature an alarm system that can indicate in real time if a temperature excursion is occurring. As an illustrative example, data from a DDL is presented in Fig. 1. This data exhibits several temperature excursions, where recorded temperatures fall below the lower bound of the normal range, along with a corrective action, after which the recorded temperatures remain within the normal range.

Fig. 1
figure 1

Illustrative example of temperature excursions with evidence of a corrective action in a household-grade combination refrigeration vaccine storage unit. In this illustrative example, the corrective action takes place around 22 h, several hours after the first temperature excursion that would be detected by continuous temperature monitoring. In an ideal scenario, if staff are present and available to address the excursion, the corrective action would occur as soon as the first temperature excursion is detected.

Full size image

Vaccine providers can have storage units that are either refrigerators or freezers, depending on the types of vaccines distributed by the provider. Storage units may be household-grade combination units (i.e., refrigerator and freezer combined into a single appliance), household-grade stand-alone units, purpose-built stand-alone units, and purpose-built combination units. These storage units may vary in their ability to sustain a constant temperature. Household-grade combination units can be problematic for vaccine storage due to relatively dramatic fluctuations in temperature, frequently caused by de-frost cycling within the freezer section. For this reason, CDC does not recommend using the freezer section (Centers for Disease Control and Prevention 2018).

Methods

We developed a Monte Carlo simulation model that represents a household-grade refrigeration unit. As a matter of scope, this study focuses on refrigerated storage units only, not freezer units. For vaccine storage in refrigerators, we define the normal temperature range as between 2 °C and 8 °C, which is the range stated in all manufacturer package inserts for all routinely recommended vaccines (Centers for Disease Control and Prevention 2018). Monte Carlo is a type of simulation methodology that uses repeated, random sampling from a distribution. Assumptions in the simulations can be adjusted so that the simulations represent a storage unit that is relatively reliable and has a fewer number of temperature excursions. Alternatively, assumptions in the simulations can be adjusted so that the simulations represent a storage unit that is functioning less reliably. To investigate the simulation model across a range of reliability assumptions, we put simulations into four groups (numbered I, II, III, and IV in the results) that represent different levels of storage unit reliability. Within each group, we ran the simulation 500 times, or for 500 iterations. Each iteration constructs a hypothetical new, randomly generated storage unit with a unique distribution of temperature excursions. For each iteration, we recorded the total number of temperature excursions, the total temperature excursion runtime, the temperature excursions as a portion of total runtime, and the number of temperature excursions per day. Results from the simulation models are summarized as mean number of excursions detected and mean portion of excursions detected from the 500 iterations for each of the four simulation groups.

Pilot project data

The simulations in this study are based on data collected during a larger project to pilot (DDL Pilot Project) and distribute DDLs to US vaccine providers, which ran from 2012 to 2016 (Stevenson et al. 2014). Our study utilizes the temperature and excursion data collected by DDLs from a convenience sample of 243 household-grade, combination refrigeration units that had received DDLs in the DDL Pilot Project (Stevenson et al. 2014). Across the 243 units, temperature data was collected for an average of 4305 h, or 179 days. Summary statistics are reported for these data in Table 1. Other aspects of the DDL Pilot Project have been described and summarized elsewhere (Stevenson et al. 2014). The DDL Pilot Project was a Prevention and Public Health Fund awardee grant, funded by CDC.

Table 1 Summary data for recorded temperature excursions in household-grade storage units used in the simulation model.
Full size table

Brief description of the simulation model

The four simulation groups are characterized by the amount of possible excursion time that can occur in a simulation iteration. Simulation groups I, II, III, and IV allow excursions to occur between 0 to 0.001, 0 to 0.005, 0 to 0.022, and 0 to 0.900 of total runtime respectively. The ranges for simulation groups I, II, and III are based on the DDL Pilot Project data, where 0.001, 0.005, and 0.022 were the median, 70th, and 90th percentiles of the total excursion time in the DDL Pilot Project data, among our sample of household-grade combination units. Simulation group IV allows for excursion time to occur up to 90% of the total runtime, which provides a worst-case scenario that is the least reliable in terms of maintaining a stable temperature within the normal range.

In each iteration, the model randomly generates simulated data that represents 90 days of runtime in a household-grade refrigeration storage unit. In particular, the simulated data represents temperature values recorded at 5-min intervals for 90 days. We simulated data at 5-min intervals to capture temperature excursions at a high resolution, and we limited the time horizon to 90 days to keep the size of simulated data files manageable. These simulated temperatures are either within the normal range (between 2 °C and 8 °C) or outside the normal range. The simulation uses two key inputs based on data from the DDL Pilot Project: (1) the portion of time outside the normal range for a given storage unit, and (2) the extreme temperatures and the length of time associated with excursions. Additional technical details on the simulations are presented in the Appendix.

Non-continuous temperature monitoring practices

In this model, temperature readings can occur once per day at 9 am or twice per day at 9 am and 4 pm. At each of these times, the current temperature and the min/max temperatures can be recorded. The current temperature readings record the current temperature in the storage unit. The min/max temperature reading records both the minimum and maximum temperature that was detected since the last time the min/max temperature was checked.Footnote 1

The times of temperature readings occurring in the model are based on temperature monitoring activities in provider offices, during the business days of a 5-day workweek. We assumed no readings occur on the weekends so that for every 7 days in the simulation, 2 of the days have no readings, no matter which monitoring practice is being evaluated. As an example, consider twice daily min/max readings. The last min/max reading of any given week in the simulation occurs at 4 pm on Friday. The next min/max reading occurs the following Monday at 9 am. The Monday 9 am reading documents the minimum and maximum temperatures experienced by the storage unit since Friday at 4 pm. For simplicity, we assumed no holidays occur during the 90-day simulation. We also assumed that the temperature readings are never skipped and temperatures are recorded with perfect accuracy. In practice, holidays, sick days, and occasional human error would likely result in skipped or inaccurate readings.

Analysis

In the primary analysis, three temperature monitoring practices were evaluated: (1) twice-daily current temperature readings without min/max readings, (2) twice-daily current temperature readings with daily min/max readings, and (3) twice-daily current temperature readings with twice-daily min/max readings. The outcomes of interest were the number of excursions and portion of total excursions that were detected by a given temperature monitoring practice. Outcomes for all three practices were reported as averages across the 500 simulation iterations for each of the four simulation groups. The list of temperature monitoring practices evaluated in the primary analysis centered on three common practices and was not exhaustive. For this reason, we implemented sensitivity analyses, where a more complete list of potential temperature monitoring practices could be evaluated. The monitoring practices evaluated in the sensitivity analysis include all possible combinations of once-daily and twice-daily current temperature readings, and min/max temperature readings. For these sensitivity analyses, we also assumed our widest range (0 to 0.9) for the possible portion of cumulative excursion time.

Results

Table 1 presents summary statistics for storage units and excursion events from the DDL pilot data. The portion of time that the household-grade combination refrigerator units from the DDL Pilot Project operated in the normal range was 0.989. Of the 4972 excursions, the portion that were lower excursions (i.e., temperature below the bottom of the acceptable range, putting the vaccine at risk of freezing) was 0.57 (2832) and the portion that were high excursions was 0.43 (2140). The average time of a temperature excursion from the DDL pilot was 1.749 h, with high excursions lasting longer (2.888 h per excursion) on average than low excursions (0.889 h per excursion). Figure 2 presents the distribution of each excursion with respect to the length of time each excursion lasted and the most extreme recorded temperature in each excursion. The majority of excursions appear to be clustered in areas relatively near the normal temperature range and in areas that indicate relatively short-duration excursions.

Fig. 2
figure 2

Duration and temperatures associated with excursions from household-grade combination refrigeration units with a grey box indicating the normal range for temperatures. Each point in the figure represents the duration (x-axis) and the most extreme temperature (y-axis) of a temperature excursion from the DDL Pilot Project data. As an example, points just above the area labeled the normal temperature range constitute relatively long excursions which had a maximum temperature that was only slightly warmer than the recommended temperature range. By comparison, points in the bottom left region constitute relatively brief temperature excursions with a low minimum temperature recorded.

Full size image

The results of the primary analysis from the simulation model can be found in Table 2. The average portion of temperature excursions detected by twice-daily current temperature readings with no min/max was between 0.047 and 0.068 of all temperature excursions. Including a once-daily min/max temperature reading increased the portion of detected temperature excursions to a range of 0.275 to 0.963, depending on simulation group. Twice-daily current temperature readings and twice-daily min/max temperature readings yielded a range of the portion of detected temperature excursions from 0.347 to 0.965. The number of excursions detected by any of the three monitoring practices increased in the simulation groups with greater numbers of temperature excursions.

Table 2 Results from four groups of 90-day simulations of storage unit temperature excursions, evaluating three temperature monitoring practices
Full size table

Results from the sensitivity analyses are summarized in Fig. 3, which plots the portion of temperature excursions detected as a function of the portion of cumulative excursion time in a simulation iteration, stratified by different non-continuous temperature monitoring practices in each panel. The negative relationship between cumulative excursion time of a simulation and the portion of temperature excursions detected by monitoring practices with at least one-daily min/max reading is apparent in the sensitivity analysis (Fig. 3c-h). This analysis also supports the main finding from Table 1, which is that monitoring practices that have no min/max reading (Fig. 3a and b) do not detect as many excursions as monitoring practices with at least one min/max reading (Fig. 3c-h).

Fig. 3
figure 3

Portion of detected excursions verses total excursion time in a sensitivity analyses, with each panel representing different temperature monitoring practices and 500 simulation iterations. Each point in the figure represents the outcome of a single iteration of the simulation model. For example, a point in the bottom left of panel a, represents a simulation iteration that experienced relatively few temperature excursions, and a relatively small portion of these excursions that was detected by once daily current temperature readings. For a comparison, consider a point in the top left of panel d. A point in this region represents a simulation iteration that experienced a relatively small number of temperature excursions but the majority of which were detected by the twice daily min/max temperature readings

Full size image

Discussion

Proper storage and handling along the vaccine supply chain is a necessary component of a well-functioning immunization system. Temperature excursions can compromise vaccine effectiveness and contribute to vaccine wastage. Failure to maintain the cold chain may have played a role in past outbreaks of vaccine-preventable diseases (Paunio et al. 1998; Yeung et al. 2005). The objective of this study was to use a simulation model to assess the potential benefits of different types of non-continuous temperature monitoring practices. In terms of detecting temperature excursions, we found min/max temperature readings to add substantial value, relative to relying on current temperature readings alone, particularly when a storage unit experiences relatively few temperature excursions. However, none of the non-continuous temperature monitoring practices that were assessed in this study were able to reliably detect 100% of temperature excursions, which underscores the additional value of using DDLs for CTM.

Even though CTM devices are broadly recommended, non-continuous temperature-monitoring practices may continue to be an important part of vaccine storage and handling procedures for at least two reasons. First, for some geographic regions across the world, particularly those where information technology infrastructure remains underdeveloped, CTM may not be technically feasible. Secondly, even when CTM is in practice, non-continuous temperature monitoring provides an additional, documented record of temperature stability in the event that certain functional issues with the DDL or data management system occur. Our study demonstrates, in both of these cases, that the use of min/max temperature readings may probably provide greater awareness of any temperature excursions when compared to non-continuous practices that only utilize current temperature readings.

Findings from this study support and underscore the benefits of CTM. While non-continuous temperature-monitoring practices that utilized min/max readings outperformed those that only utilized current temperature readings, in the majority of simulation iterations there was no non-continuous temperature-monitoring practice that detected all of the temperature excursions that occurred. For this reason, CTM and the use of DDLs offer the current, best available means of capturing the most complete assessment of temperature stability in vaccine storage units. Temperature excursions can have a varied effect on the effectiveness of stored vaccines, which may depend on several factors including: the temperatures experienced during an excursion, the duration of any out-of-range temperatures, and the types of vaccines being stored. Currently, the CDC recommends the use of a continuous temperature monitoring device (Centers for Disease Control and Prevention 2018), and began requiring these devices for storage units containing federally funded Vaccines For Children vaccines in January 2018. At least one state vaccine program adopted the requirement of CTMs before January 2018 (Oregon Health Authority 2015).

Our model and results should be interpreted in consideration of some important limitations and context. First, human behavior is not modeled in these simulations. In practice, recording of temperatures by staff will likely be subject to skipped observations or imprecisely documented temperature readings. Second, storage units modeled were household-grade, combination-style units. So some caution may be warranted when considering the generalization of this study’s results to other types of storage units, such as stand-alone refrigerator units, purpose-built or pharmaceutical-grade storage units, or even freezers. Third, this study models temperature excursions in vaccine units as intermittent, non-permanent events. One likely consequence of a permanent temperature excursion would be the replacement of a vaccine storage unit. We did not model the replacement of storage units. Finally, the data used as inputs into the simulation model in this study did not come from a sample that was known to be representative of storage units or vaccine providers in the United States or any other jurisdiction. As such, generalizations of these results to a nation-wide sample may not be appropriate.

In light of these limitations, we can conclude that the simulations presented in this study provide a useful, evidence-based comparison of several non-continuous temperature-monitoring practices using data generated in a simulation. The results of this study highlight the benefits of min/max as a non-continuous temperature-monitoring practice. These results also provide support for the transition to CTM as a way to ensure vaccination effectiveness, which can improve immunity, lead to healthier communities, and confer greater protection from vaccine-preventable diseases.