The use of models to forecast disease outcomes and the effects of interventions seems to be exploding, perhaps prompted by the prominent use of models during the coronavirus disease 2019 (COVID-19) pandemic. A cursory literature search for ‘modeling’ and ‘cost-effectiveness’, and limiting this to health care journals, reveals a steady rise from less than 40 papers per year at the turn of the century to nearly 300 in 2020 and 2021. In 2022, the number is approaching 500, with 20% of those related to COVID-19. With the growing number of models published in any given disease area, it is important to ensure that the models have been validated and that they undergo systematic critical review. To foment this activity, Pharmacoeconomics is launching a series of papers reporting on critical review of modeling approaches in specific disease areas. In this short introduction to the series, I lay out what the review papers should cover.

1 Comprehensiveness

Each review should strive to cover all models in a disease area that seek to inform decisions about the use of health technologies, regardless of the language or year of publication. The focus of the review should be on papers that report on the methods used to conceptualize and implement the model, no matter what specific technologies are assessed or what analyses are reported. Supplementary online materials, and any additional documentation posted on websites should also be sought and reviewed. How papers were identified should be described following applicable guidelines, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [1]. If the model itself is available, it would be ideal if the review authors examined it, but it is recognized that this may go too far.

Publications reporting results with only cursory descriptions of the model should be used to trace back to the paper that provides the required details. If none is found, it would be most helpful if an attempt is made to contact the authors to request the details. When this also fails to turn up sufficient information, the model should still be listed, noting that it could not be critically reviewed and the reason for this.

2 Focus on Methods

Although the quality of the reporting will undoubtedly affect the ability to critically review the model, the review should be about the methods not about how they are reported. If some methodological aspect is left out or is unclear, this should be noted, but there is no need to score each paper on its reporting quality. Results of particular analyses are not of interest, except where they reflect on methodological choices. If so, the review should address what the results indicate about those choices. Similarly, the specific values used as inputs (e.g., the discount rate) are not relevant unless they form part of the model structure (e.g., the cycle length in a Markov model).

The review should be structured using a methodological best practice guidance (e.g., Caro et al. [2]). Generally, it should cover how the model was conceptualized and its intended uses; the type of framework selected; details specific to that type of framework; sources for the structural assumptions; how uncertainty was considered; and how transparent the model is and how it was validated. Most of these details can be tabulated for each model, with the text reserved for the reviewers’ assessment of their adequacy.

3 Conceptualization

In this section, the review should consider what the objectives were for the model; who is the intended audience(s); what problems it was meant to address and its intended uses. Is it meant for a single or multiple application (‘whole disease model’ [3])? The scope of the model (e.g., is it limited to a particular severity of disease); what perspective(s) are enabled for analyses; details of the disease types covered; which populations are targeted; which interventions are modeled and how feasible it is to add new ones; what outcomes are considered and how they are measured; and what formal process was followed in designing the model should be covered. The reviewers should not just list the details but rather critically assess these and note any gaps.

The type of model selected, and its justification, should be reported. A simple classification can be used: is the model deterministic or stochastic? If it is stochastic, does it consider what happens periodically or does it contemplate the time until each event occurs? Decision trees, partitioned survival, cohort state-transition (‘Markov’) and static SEIR models are deterministic frameworks. Stochastic structures can be individual-level Markov (‘microsimulation’), discrete-event simulation (usually unconstrained), and agent-based simulation. Other types (e.g., dynamic transmission models, systems dynamics, general equilibrium) are much less common in our field. Whatever type was selected, the review should also assess whether it is adequate for the stated purpose(s) of the model.

4 Structure

The review should provide details of the structure according to the model type. For example, for a cohort Markov model, it should be specified if it is a chain (i.e., constant transition probabilities); what states were defined, together with which transitions are allowed and whether these adequately represent the problem; what sources were used to derive transition probabilities and how projections were made; how heterogeneity in determinants of the transition probabilities was handled (e.g., were there separate states for males and females or was a proportion used in a single state); what cycle length was chosen, whether it was short enough to adequately reflect the frequency of transitions; if it can vary over time and was there a half-cycle correction; which outcomes are accrued and how utilities are applied.

In stochastic models, the review should describe the possible trajectories; what patient profiles are considered, on what basis they were defined and whether the same individuals are modeled for each intervention to reduce nuisance variability; the handling of continuous disease parameters (e.g., are they regularly updated); for time-to-event models what event time distributions are used, how they were determined, how times are drawn and redrawn; how stochastic uncertainty is handled and the availability of stability analyses.

Other model types may require description of additional or alternative aspects. While the choice of input values is not generally relevant for the review, some inputs may affect the structure. For example, how a determinant of patient trajectories is handled, especially if it is projected into the future, would be germane. For all aspects, appraisal of the structural choices must be made.

5 Uncertainty

The results of specific uncertainty analyses are not of much interest—they are entirely dependent on the purpose of that particular study. Instead, the review should address the types of uncertainty analyses the model enables.

Of greatest importance is structural uncertainty that arises from the assumptions made and methodological choices. Does the model facilitate analysis with different structural assumptions? For example, is a structure approach that facilitates scenarios mentioned? If time-to-event distributions are used (e.g., for mortality), can the user select a different distribution? Are alternative structures available with toggles for easy activation? Has structural uncertainty been parameterized to facilitate testing?

Also of importance is parameter uncertainty. Most models today allow for one-way deterministic analyses across a range of input values and also for probabilistic analyses that draw input value sets according to distributions that describe their uncertainty. The review should address the extent to which the model enables these analyses. Can the user readily modify how the uncertainty is characterized? Were any inputs derived via calibration and can their uncertainty be examined? Are the inputs that control the analysis (e.g., the number of replications to be run) easily changed?

6 Validation

The first step in validation is to appraise the face validity of the model concept, its structure, and the evidence used in its design. Modelers should have independent experts evaluate face validity and document what questions were raised, or, at a minimum, how they were resolved. It is also helpful if the model has been submitted for review by an agency or other external organization.

Errors in implementation of models are common and thus it is important to subject a model to formal, rigorous verification that it is correctly specified and works as intended. This verification should be documented and its results should be available to anyone interested in that model.

Assessing the extent to which a model’s forecasts are accurate is perhaps the most important aspect to document in a review. As the reviewer cannot be expected to independently verify this, it behooves the modelers to compare their model’s forecasts to actual observations. Ideally, these external data were not used in the building of the model and thus an independent validation was performed. However, at a minimum, modelers should have compared the model predictions with what was obtained in the studies used as sources for the model. These dependent validations are not as strong but are more practical to do.

For all three types of validation, the review should note whether the modelers have indicated that it was done and the process that was followed. Published validation checklists can be leveraged in this regard [4, 5]. For face validity, they should note who was involved and their degree of independence from the project and funders. The review should also list known limitations of the model specified by the authors and the extent to which external validation was performed. While this is rarely included in a paper reporting on a model, the availability of report(s) detailing what was done, and any resulting modifications, should be noted.

7 Transparency

All models should be accompanied by non-technical documentation that describes the model in terms that any reader can follow. The review should note whether this documentation is available freely, perhaps in the model itself, or via supplementary materials or posted on a website. Models should also have full technical documentation that provides all the details that would enable an interested person with the proper skills to rebuild the model. The review should address whether this documentation exists and whether it is available freely or under some non-disclosure agreement. Apart from documentation, the review should note whether the model is available for others to review and use, and whether openly or under licensing. The software used to implement the model should also be given. Funding sources and other potential conflicts of interest should also be listed.

8 Conclusion

Modeling review papers submitted to Pharmacoeconomics will undergo peer review guided by the criteria set out here. We encourage researchers to contribute to this Series—those having already completed such a review some time ago are welcome to update it, ensuring that it meets the criteria, and submit it for consideration. We hope that this Series will not only be a useful resource for researchers in the area but will also provide guidance on best practice for future modeling efforts. Hopefully, it will also help elevate the standards these models must meet.