FormalPara Key Points

Most vaccines in early childhood are administered on the same day as other vaccines, making it challenging to pinpoint which of the vaccines may be causally related to an observed increase in risk of an adverse event.

This paper outlines a systematic process for determining which vaccine(s) are the most likely to have caused an observed increase in risk of an adverse event and whether there is excess risk from co-administration.

This methodology is important because if substantial excess risk is detected with same-day vaccination, vaccine schedule recommendations can be modified.

1 Introduction

Childhood immunization schedules involve multiple vaccinations per visit to target preventable diseases [1]. The Centers for Disease Control (CDC)-recommended child and adolescent immunization schedule includes 15 vaccines, nine of which have doses recommended in the second year of life for all children [2]. Most pre- and post-licensure studies of vaccine safety evaluate a single vaccine in relation to an outcome of interest. While there is some literature on the safety of same-day vaccinations, there are numerous challenges [3,4,5,6,7,8]. Randomized pre-licensure studies tend to have insufficient sample size to evaluate rare adverse events. In contrast, while non-interventional studies conducted in large administrative or clinical healthcare databases have larger sample sizes, the recommended immunization schedule results in frequent (non-random) administration of specific combinations of vaccines on the same day. Because of the high prevalence of same-day vaccinations, when there is an observed increase in risk of an adverse event in a defined risk window after a healthcare encounter with numerous vaccines administered, it can be difficult to ascertain which of the vaccines triggered the adverse event.

This methods paper outlines a systematic process to determine which vaccine(s) out of a simultaneously administered group of vaccines are the most likely to have caused an observed increase in risk of an adverse event. We focus on an example where an increased risk of seizure has been observed on the day after vaccine administration in the second year of life, but frequent simultaneous vaccination has made it difficult to discern which vaccine(s) may have triggered the observed excess risk [7, 9].

2 Objective

The aim was to describe a process for evaluating and determining which vaccine(s) may increase the risk of an adverse event when same-day vaccine administration is common.

3 Methods

We used vaccination and electronic medical record data from Kaiser Permanente Northern California and Colorado (1995–2015). We included vaccination dates that occurred after at least 56 days without any prior vaccination, which we defined as an “incident vaccination,” to avoid contamination from effects of other vaccines. We focused on vaccines administered in the second year of life; therefore, we restricted the record search to vaccination dates for children 11–23 months of age and categorized vaccines based on the Immunization Information Systems (IIS) HL7 standard CVX code set [10]. The categories we evaluated included diphtheria–tetanus–acellular pertussis (DTaP), inactivated influenza, Haemophilus influenzae type B (HiB), hepatitis A, hepatitis B, measles–mumps–rubella (MMR), pneumococcal conjugate (PCV), varicella, and inactivated poliovirus. The measles–mumps–rubella–varicella (MMRV) vaccine was categorized as both MMR and varicella administered on the same day. Every vaccine administered on an incident vaccination date was included as a same-day vaccination. We identified the adverse outcome seizure using International Classification of Diseases 9th revision (ICD9) codes (780.3* or 345*) in inpatient, emergency room, and urgent care settings, which has previously been shown to have a high positive predictive value for true incident seizure events [11]. Incident outcomes were defined as relevant ICD9 codes that occurred after at least 56 days of enrollment in the health system, during which there were no recorded seizures. Incident outcomes were included if they occurred within 56 days after an incident vaccination date. Children were allowed to contribute more than one incident vaccination episode with an incident outcome to the analysis.

We used the observed data from Kaiser Permanente to generate simulated data with known true relationships between vaccine combination and seizure. We simulated five scenarios, which varied the independent and interactive effects of DTaP and PCV on seizure. These two vaccines were selected for the simulation because of prior research suggesting potential relationships with seizure the day after vaccination [7, 9]. Independent effects and interactions were generated for the simulation on the multiplicative scale. We selected five plausible scenarios for independent and joint effects of vaccinations, including (1) independent effects of both vaccines with no interaction; (2) no independent effects, with joint effects due to interaction; (3) independent effects of both vaccines and negative interaction; (4) independent effect of one vaccine, with no interactions; and (5) independent effect of one vaccine and positive interaction with one other vaccine (Table 1).

Table 1 Incidence rate ratios in simulated scenarios
Full size table

For our simulation, we randomly assigned observed seizures to risk or reference windows while retaining the observed simultaneous vaccination patterns and number of seizures observed in the empirical data. The “true” relationships with seizure were generated in simulated data by identifying four mutually exclusive strata (exposed to neither DTaP nor PCV, exposed to DTaP, but not PCV, exposed to PCV but not DTaP, or exposed to both). Within each stratum, we randomly assigned a proportion of the outcomes to occur during the risk window and the remainder to the reference window, with the proportions selected to generate the desired relative risk in each stratum (see the Electronic Supplementary Material, “Electronic Supplemental Materials A”). Because we do not directly alter the relationships between other childhood vaccines and seizures, associations between other vaccines and seizure occur in the simulation through their association with DTaP and PCV (e.g., simultaneous administration). We generated 1000 datasets for each simulation scenario and report the mean incidence rate ratio (IRR), mean 95% upper and lower confidence intervals (CIs) from the model-based standard error, mean p value, and proportion of p values that were below 0.05.

We followed a four-step process to identify the most likely vaccines associated with increased risk of seizure for each of the simulation scenarios as well as the observed data: (1) univariate analyses, (2) multivariable analyses with and without interactions, (3) stratified analyses for most likely vaccines, and (4) attributable risk (AR) for same-day and separate-day vaccination (age-standardized). Each step used a self-controlled risk interval design where day 0 was the day of vaccine administration, the risk window was day 1 after vaccination, and the reference window was days 15–56 (Fig. 1). In our analysis, we chose a reference window that excluded days 7–10 post vaccination because of the known risk conferred by MMR within that time frame. We chose not to count seizures that occurred on day 0 (the date of vaccination) because of the inability to ascertain in the data whether seizure preceded vaccination on the same day. Although it was unlikely that children would be vaccinated on the same day after being seen for febrile seizure, restricting our analysis to counts of outcomes starting the day after vaccination ensured temporality of exposure and outcome. We used SAS 9.4 PROC LOGISTIC event/trials syntax to fit logistic regression models that accounted for the length of follow-up in risk and control windows. This is equivalent to a conditional Poisson model that has an offset term equal to log(days) [12]. These models were conditioned on the unique vaccination episode to make within-episode comparisons (example code in “Electronic Supplemental Materials B”). For the observed data, we also evaluated the potential for age to modify vaccine-related risk of seizure and estimated age-standardized AR for same-day and separate-day vaccination.

Fig. 1
figure 1

Self-controlled risk-interval design

Full size image

3.1 Process

3.1.1 Step 1: Univariate Analyses

  1. a.

    Independent variable was exposure to any vaccine.

  2. b.

    Independent variable was the number of vaccines administered (either a continuous or categorical variable).

  3. c.

    Independent variable was each vaccine (in separate models, naïvely ignoring simultaneous vaccines).

  4. d.

    Independent variable was each vaccine (in separate models) after restricting to vaccination episodes where only a single vaccine was administered.

  5. e.

    Independent variable was each vaccine (in separate models) after excluding vaccine episodes with suggestive evidence in earlier exploration in step 1c or 1d.

3.1.2 Step 2: Multivariable Analyses with and Without Interactions

  1. a.

    Bivariate with one vaccine of interest and “other” vaccines grouped together

  2. b.

    Multivariable with all vaccines included as independent variables

  3. c.

    Multivariable with all vaccines included as independent variables as well as pairwise interactions between vaccines with suggestive evidence from steps 1c and/or 1d or 2a and/or 2b.

3.1.3 Step 3: Stratified Analyses for Most Likely Vaccines

We did stratified analyses to estimate the IRR for seizure for pairwise patterns of simultaneous administration, e.g., no vaccines, one vaccine, or both vaccines on the same day. We further explored within strata the potential for age at time of vaccination to modify vaccine-related risk of seizure 1 day post vaccination.

3.1.4 Step 4: Attributable Risk for Same-Day and Separate-Day Vaccination (Age-Standardized)

The AR was defined as the excess number of seizures (observed − expected) during the risk window (Eq. 1). We age standardized the AR to account for the different age distributions for children with different patterns of same-day vaccination (Eq. 2). The expected number of seizures was estimated using the number of seizures in the reference window and accounting for differential follow-up time (Eq. 3).

$$ {\text{AR}} = \mathop \sum \limits_{i}^{n} {\text{AR}}_{i} = \mathop \sum \limits_{i}^{n} ({\text{Observed}}_{i} - {\text{Expected}}_{i} )/\# {\text{ ofeligible vaccinations}}_{i} $$
(1)
$$ {\text{AR}}_{{\text{age standardized}}} = \mathop \sum \limits_{{{\text{age}} = i}}^{n} w_{i} \left( {{\text{AR}}_{i} } \right) $$
(2)
$$ {\text{Expected}} = \# {\text{ seizures in reference window }}\left( {\frac{{\# {\text{ days in risk window}}}}{{\# {\text{ days in reference window}}}}} \right) $$
(3)

4 Results

We identified 2324 episodes of vaccination followed by seizures in the risk or reference windows in the empirical data. Of these, 1736 (75%) involved simultaneous vaccination. The same number of exposed patients and outcomes were included in each simulation scenario; however, the timing of the outcome was varied as described in the “Methods” section to generate the desired independent and joint effects of vaccination.

4.1 Step 1: Univariate Analyses

4.1.1 Independent Variable was Exposure to Any Vaccine.

The average IRR for seizure 1 day after any incident vaccination relative to days 14–56 for the five simulation scenarios and empirical data ranged between 1.3 and 1.9 (E-Table 1a, see the Electronic Supplementary Material). This reflected the average risk of seizure from any same-day vaccination pattern.

Conclusion This provides evidence that there is at least one vaccine associated with an increase in seizure risk 1 day after vaccination.

4.1.2 Independent Variable was the Number of Vaccines Administered (Either a Continuous or Categorical Variable)

When modeling risk of seizure 1 day after vaccination with number of simultaneously administered vaccines as a continuous variable, each of the simulation scenarios as well as the empirical data had an IRR of 1.2 per additional vaccine (p < 0.001) (E-Table 1b, see the Electronic Supplementary Material). Modeling the number of simultaneously administered vaccines as binary or categorical variables also suggested that having more vaccines administered on the same day was associated with greater risk of seizure in each of the simulation scenarios and the observed data.

Conclusion Although simulation scenarios were generated such that at most two vaccines increased risk of seizure, children with more same-day vaccinations were also more likely to be exposed to the relevant vaccine combination, resulting in the association between number of same-day vaccines and seizure risk.

4.1.3 Independent Variable was Each Vaccine (in Separate Models, Naïvely Ignoring Simultaneous Vaccines)

In naïve univariate analyses, which do not account for simultaneously administered vaccines, many vaccines in the schedule appeared to be associated with an increased risk of seizure the day after vaccine administration. For each of the simulation scenarios and in empirical data, DTaP, HiB, and PCV had the largest effect sizes coupled with smallest p values (E-Table 1c, see the Electronic Supplementary Material).

Conclusion From this analysis, we cannot conclude that any particular vaccine increases the risk of seizure, but there is suggestive evidence for DTaP, HiB, and PCV.

4.1.4 Independent Variable was Each Vaccine (in Separate Models) After Restricting to Vaccination Episodes Where Only a Single Vaccine was Administered

In univariate analyses restricted to vaccination dates where only a single vaccine was administered, as indicated by wide CIs, the sample size was generally insufficient to provide evidence regarding the relationship with seizure across simulation scenarios as well as empirical data (E-Table 1d, see the Electronic Supplementary Material).

Conclusion Simultaneous administration of childhood vaccinations is the norm, making restriction to patients with single vaccines administered infeasible for understanding which vaccine in a schedule may increase risk of adverse events.

4.1.5 Independent Variable was Each Vaccine (in Separate Models) After Excluding Vaccine Episodes with Suggestive Evidence in Earlier Exploration in Step 1c or 1d

Simulation In univariate analyses restricted to episodes where DTaP, HiB, or PCV was not administered (E-Table 1e, see the Electronic Supplementary Material), we observed similar results for scenarios 1 and 3, where DTaP and PCV had simulated true independent effects (with and without departure from multiplicity). When vaccination episodes where DTaP was administered were excluded, univariate analysis of PCV suggested an association with seizure with magnitude of the IRR approximating 2.0. Similarly, univariate analysis of DTaP after excluding vaccination episodes with PCV suggested a twofold association between vaccination and seizure. After excluding vaccination episodes with HiB, univariate analysis of DTaP as well as PCV provided evidence suggesting a roughly twofold increase in risk of seizure in the day after vaccination. In the table, we only provide results after excluding the top three vaccines selected based on p value from naïve univariate analyses. For scenario 2, no vaccine had strong associations with seizure after excluding episodes with DTaP, PCV or HiB, respectively. In scenarios 4 and 5, where the truth was that PCV had an independent effect, but DTaP did not, there was evidence supporting an association between PCV and seizure when vaccination episodes with administration of either DTaP or HiB were excluded. When vaccination episodes where PCV was administered were excluded, there was no evidence supporting a relationship between other vaccines and seizure in the risk window.

Observed data After excluding vaccination episodes with DTaP, univariate analysis of both HiB and PCV suggested a roughly 2.0 increase in risk of seizure the day after vaccination. If episodes with HiB were excluded, there were strong positive associations observed for DTaP and polio. When episodes with PCV were excluded, we observed strong positive associations with seizure for DTaP, HiB, hepatitis B, and polio.

Conclusion These results suggest that the elevated risk of seizure 1 day after vaccination is related to more than one vaccine and add to the evidence for effects of DTaP, HiB, and PCV.

4.2 Step 2: Multivariable Analyses with and Without Interactions

4.2.1 Vaccine of Interest and “Other” Vaccines

Simulation Generally, the simulations provided strong evidence in bivariable models for an elevated IRR for PCV and/or DTaP in scenarios where a true effect of those vaccines was simulated, with little evidence of an association for “other” vaccines. There was also often weak evidence supporting an association between HiB and seizure in the risk window, 1-day post vaccination. For all other vaccines, there was little evidence of an association with risk of seizure. In bivariable models containing an indicator for the vaccine of interest and an indicator for all other vaccines, the effect was concentrated in “other” vaccines.

Observed data In bivariable models, there was evidence that DTaP and some other vaccine, HiB and some other vaccine, as well as PCV and some other vaccine were associated with elevated risk of seizure one day post vaccination.

Conclusion These results provide further evidence that each of three vaccines, DTaP, HiB, and PCV are associated with increased risk of seizure 1 day after vaccine administration.

4.2.2 No Interactions

Simulation In scenario 1, where DTaP and PCV have a multiplicative effect on seizure risk, the model produces IRR of 2.0 for both, and all other vaccines have null associations (E-Table 2b, see the Electronic Supplementary Material). In scenario 2, DTaP and PCV do not have independent effects on seizure, yet a multivariable model without interactions finds strong associations for both due to the frequent co-administration of these two vaccines. In scenario 3, the model without interactions produces an IRR for PCV and seizure risk that is less than 2.0 and a p value of 0.179. This is due to the negative multiplicative relationship between DTaP and PCV in this scenario, where joint administration confers an elevation in risk less than the risk from DTaP multiplied by the risk from PCV. In scenario 4, only PCV has an effect on risk of seizure. The estimated IRR is 2.0 for PCV and null for all other vaccines. In scenario 5, the truth is that PCV has an independent effect, DTaP has no effect independently, but when administered together, the risk is magnified. The multivariable model suggests almost a threefold increase in risk associated with PCV and a 50% increase in risk with DTaP.

Observed data When all vaccines in the schedule were included as independent variables, with no interactions, most had point estimates of about 1.5. None had an effect estimate with a p value < 0.05.

Conclusion There may be departure from a multiplicative effect of simultaneous vaccinations on risk of seizure 1 day after vaccine administration.

4.2.3 Interactions Between Vaccines with Suggestive Evidence in Earlier Exploration

Simulation When multivariable analysis included interactions between the two most likely vaccines from the prior univariate and multivariable analyses without interactions, there was no evidence of departure from multiplicity for scenario 1 and 4 (p > 0.70), but evidence of a joint effect of DTaP and PCV without independent effects for scenario 2 (p < 0.01), a negative multiplicative interaction for scenario 3 (p = 0.13), and a positive multiplicative interaction for scenario 5 (p = 0.12) (E-Table 2c, see the Electronic Supplementary Material).

Observed data When an interaction between DTaP and PCV was included in the model that included every other vaccine in the schedule as independent variables, the coefficients for other vaccines in the schedule shrank toward the null and there was suggestive evidence of negative multiplicative interaction (p value = 0.22). Including interactions between DTaP and HiB or HiB and PCV likewise shrank coefficients toward the null; interaction coefficients had large p values (p > 0.40) (E-Table 2c and E-Table 5, see the Electronic Supplementary Material).

Conclusion The departure from multiplicity with the three most likely vaccines related to increased risk of seizure 1 day after vaccination is negative, indicating that the risk with co-administration is lower than the multiplied risk from administration on separate days.

4.3 Step 3: Stratified Analyses for Most Likely Vaccines

Simulation Stratified analyses within each pairwise combination of DTaP and PCV (neither, DTaP without PCV, PCV without DTaP, or both), without adjustment for other vaccines, produced estimates consistent with the simulated truth for scenarios 1–5 (E-Table 3, see the Electronic Supplementary Material).

Observed data Stratified analysis within pairwise combinations of DTaP and PCV, DTaP and HiB, or HiB and PCV most closely mirrored results from simulation scenario 3 (independent effects with negative multiplicative interaction).

After adjusting for the vaccine not included in the pairwise combination (e.g., including HiB as a covariate in strata formed by pairwise combinations for DTaP and PCV), there was weak evidence of an independent risk conferred by HiB or PCV; however, the IRR for joint exposure to HiB and PCV after adjusting for DTaP co-administration was 2.3 (95% CI 1.4–4.0).

There was little evidence that age modified risk in models with DTaP and PCV or PCV and HiB interactions. However, there was some support for age as a modifier of the relationship between DTaP and risk of seizure 1 day after administration in models with DTaP and HiB interaction (E-Table 4, see the Electronic Supplementary Material).

The IRR associated with joint exposure to DTaP, HiB, and PCV on the same day was 2.5 (95% CI 1.5–4.2). Children administered other vaccine combinations without simultaneous DTaP, HiB, or PCV did not have higher risk of seizure 1 day after vaccination, IRR 1.0 (95% CI 0.6–1.6).

Conclusion These results also support negative multiplicative interaction with co-administration of any combination of DTaP, HiB, and PCV.

4.4 Step 4: Attributable Risk for Same-Day and Separate-Day Vaccination (Age-Standardized) in Empirical Data

The AR for seizure the day after vaccine administration after standardizing age distributions to that of children who were administered DTaP, PCV, and HiB on the same day was between one and five per 100,000 vaccinated children (Table 2). The AR was highest for children administered all three on the same day (five per 100,000), followed by three per 100,000 for children administered DTaP only or PCV only, two per 100,000 for children administered HiB only, and one per 100,000 for children administered two of the three.

Table 2 Attributable risk based on empirical data
Full size table

5 Discussion

In this paper, we illustrate a process for determining which vaccine(s) may increase risk of an adverse event in the context of immunization schedules that make same-day vaccine administration common. This is important because if substantial excess risk is detected with same-day vaccination, vaccine schedule recommendations can be modified. We used a simulation strategy that retained the observed same-day vaccination patterns and generated scenarios with different “true” relationships between vaccine combinations and seizure 1 day after vaccination. These simulations demonstrate the complexity of identifying the vaccine(s) that may trigger a transient period of elevated risk and the importance of understanding how simultaneous administration patterns may influence results. For example, in each of the simulated scenarios, only two vaccines had “true” impact on risk of seizure. However, models that evaluated the effect of number of vaccines on risk of seizure consistently suggested that the higher the number of vaccines administered on the same day, the greater the risk of seizure. This naïve interpretation might be used to bolster anti-vaccination fears of “vaccine overload” [13] when in truth, the two vaccines were more likely to be co-administered with several other vaccines than by themselves, resulting in the apparent relationship between number of vaccines administered and seizure. Following the process for determining risk associated with vaccine combinations in each of the five simulated scenarios, we were able to narrow down the field to the vaccines in the schedule for which we had simulated “true” effects on risk of seizure.

We implemented the same process to identify vaccine(s) associated with increased risk 1 day after vaccination in observed data where the truth was unknown. The relationships between vaccines in the schedule and risk of seizure 1 day after vaccination in the observed data were more complex than in the scenarios we simulated. However, the results most closely paralleled the simulated scenario with negative multiplicative interaction—where risk from same-day administration was lower than the multiplied risks from separate-day administration. We narrowed the association with seizure 1 day after vaccination from all vaccines in the schedule to DTaP, PCV, and/or HiB, but were not able to further disentangle these relationships in our data. These relationships could be further elucidated with a larger sample size with sufficient variation in simultaneous vaccine combinations. When none of these three vaccines were administered, no elevation in risk was observed.

There are several limitations of the process outlined in this paper. First, the process involves running many models to better understand the relationships between vaccine combinations and risk of an adverse event in a defined risk window. Although some may raise concerns about multiple testing, in this process, we are not conducting formal hypothesis tests. Instead, the models are designed to explore and inform selection of promising candidates to evaluate in later analyses. Second, when narrowing the selection of candidate vaccines, semi-arbitrary decision rules will be necessary to determine which vaccines in the schedule seem more promising than others. In our analyses, we focused primarily on p values as a criterion for suggestive evidence and secondarily on effect size. Recognizing that the p value conflates sample size and effect size, we used it not as a statistical test, rather as a filter. The sample sizes for specific vaccine combinations are not large. If certain vaccine combinations occur less commonly due to adherence to the recommended immunization schedule, there may be greater utility in pursuing evaluation of other combinations that have similar effect sizes but greater sample size and lower p values. Third, as with any self-controlled analysis, it is important to select appropriate risk and reference windows. How the risk window is defined, including decisions about whether or not to count outcomes on day 0, will influence measures of AR in an actual safety study. Fourth, our simulations generated “true” exposure effects based on only variations of multiplicative effects and not additive effects. While additive measures may have more public health significance [14], multiplicative relationships may correspond more naturally with biological mechanisms [15]. Fifth, variation in vaccination patterns could be driven by vaccine hesitancy and parental concerns about vaccine safety. Bias stemming from non-random selection of families that adhere to the vaccination schedule will affect the proposed process in a similar way as any non-randomized study. The magnitude of bias will depend on the strength of the association between adherence and risk factors for the adverse event of interest. Risk factors for febrile seizure in childhood include age, family history, and infections causing fever [16]. Sixth, vaccine formulations can change over the study period. Shifts in formulation could reduce ability to detect true effects on risk of seizure.

6 Conclusion

Most vaccines in early childhood are co-administered with other vaccines, making it challenging to pinpoint which are associated with adverse events in observational studies. The process outlined in this paper could provide valuable information on the magnitude of potential risk from individual and simultaneous vaccinations. Candidate vaccines identified in initial analyses should be further investigated with independent data as well as biologically based, statistically independent hypotheses [17].