Introduction

The idea that stents placed in arteries bearing an aneurysm could reduce blood flow in the sac to induce aneurismal thrombosis is at least 20 years old [1, 2], but flexible devices capable of safely reaching intracranial arteries have only recently been developed and approved for clinical use in 2007 [3] and 2011 [4]. The clinical introduction of new devices, especially those that change treatment paradigms (for flow diversion (FD), from occluding the aneurysm sac to repairing the parent vessel), is typically preceded by positive animal studies. FD is a promising and innovative treatment strategy, but unexpected complications and failures have occurred in early clinical series [5–7]. There is little agreement regarding indications, and the exact role FD should play in practice remains unclear [8]. A re-appraisal of the animal studies performed to date may be warranted in order to better understand possible reasons for failures or complications. Multiple preclinical studies, featuring different flow diverters with various aneurysm models have now been published and results vary widely (references available in online supplement). We sought to systematically review the preclinical animal studies to summarize the results and attempt to identify reasons for discrepancies. A second goal was to assess whether preclinical studies realistically anticipated clinical results. Finally, we aimed to assess the quality of reporting using a checklist derived from the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines [9].

Materials and methods

Search strategy

The systematic review was performed according to the guidelines [10–12]. Search strategies were developed using terms from controlled vocabularies (MeSH for MEDLINE, EMTREE for EMBASE, and CINAHL Headings). We used words from “all fields” for Cochrane Central and from “title” and “abstract” to identify studies in MEDLINE, EMBASE, and CINAHL. The search interfaces were PubMed for MEDLINE, Ovid for MEDLINE, EMBASE and Cochrane, and EBSCO for CINAHL. Details of the search strategy are provided in the Appendix. We also hand-searched grey literature sources, including: System for Information on Grey Literature in Europe (Open Grey), National Guideline Clearing House, National Institutes for Health and Clinical Excellence (NICE), The Grey Literature Report (NYAM), Google, and Google Scholar.

We included all articles that met the following selection criteria: (1) an animal study; (2) at least one flow diverter used; (3) peer-reviewed; (4) original research (review articles, abstracts, editorials, and letters excluded); (5) English or French language.

Data extraction

The data extraction form is available in the supplemental material (online). Extracted data included: (i) study characteristics: (year of publication, laboratory, funding source (industry, public, both or unclear), and number of citations; (ii) animal model: (species, type of aneurysm(s), aneurysm dimensions, and presence of jailed branches; (iii) type of flow diverter; (iv) treatment (single or multiple devices, anti-platelet medications); (v) results: method of angiography, timing of follow-up studies, aneurysm occlusion rates, incidence of parent or jailed branch vessel stenosis or occlusion; and pathology. Successful FD was predefined as complete or near-complete occlusion of the aneurysm.

To assess the quality of reporting, we used a checklist derived from the 38 ARRIVE guidelines [9]. Three items were added for a total of 41 items: (i) the use of a control group, (ii) the use of an explicit definition of aneurysm occlusion, and (iii) the mention of the method of selection of the size of the device. Data extraction was performed by two independent authors (RF and CD), and discrepancies were reviewed through consensus. Differences in experimental methods, animal models, species, devices, outcome measures, follow-up periods, and definitions of angiographic occlusion were considered in the judgment of whether numerical results could be combined into a meta-analysis.

Results

The initial title search yielded 3544 publications (Fig. 1). After viewing the abstracts, 3462 articles were excluded, retaining 82 articles for full text evaluation; 40 articles were excluded because they did not meet the selection criteria, leaving 42 articles for final analyses (references available in online supplement).

Fig. 1
figure 1

Flow-chart

Full size image

The first paper on FD in laboratory animals was published in 2003; 11 were published between 2003 and 2009; 31 (or 74 %) between 2010 and 2015. Studies were conducted in 9 laboratories, 3 in North America (27 publications), 3 in Asia (10 publications), and 3 in Europe (5 publications).

Eighteen articles (40.9 %) were published in two journals (AJNR and Neuroradiology). The median (interquartile range (IQR)) number of citations (according to [http://scholar.google.ca]—search performed on August 11, 2015) was 12 (5–18). Three articles presenting the first preclinical results of FD in the rabbit elastase-induced model had more than 100 citations.

Twenty-eight (67 %) studies were at least partially funded by public institutions; seven studies (17 %) were solely funded by industry. Funding sources could not clearly be identified in seven articles (17 %).

Quality of reporting

Overall, articles complied with 20.8 ± 3.9 ARRIVE items, with a range of 12–28/41. Five articles included more than 25 of 41 items; 21 articles complied with 20–25 items; 16 articles included less than 20 items. Main results are presented in Fig. 2 (details can be found in Fig. 3). The median number of items was 23 (22–24) in canine studies; 20 (17–22) in rabbit studies. The median number of items reported in the seven industry-funded studies was 20 (18.5–20.5) versus 22 (18–23.5) in the remaining 35 studies.

Fig. 2
figure 2

Compliance with selected ARRIVE guidelines

Full size image
Fig. 3
figure 3

Compliance with ARRIVE guidelines

Full size image

The items that were most commonly reported were provision of a scientific background, details of how the procedure was carried out, and interpretation of results. Other items, such as mode of treatment allocation (7/42, 16.7 %) and details of adverse events (13/42, 30.9 %) were less commonly reported.

A justification for the choice of the size of the device was mentioned in 14/42 (33.3 %) articles. None of the 42 articles mentioned the ARRIVE guidelines for reporting animal studies.

Aneurysm models

Two species were used: rabbits (29) and canines (13 articles). None of the articles tested FD in both species.

There were nine different aneurysm models. The rabbit-elastase model was used in 25 articles (440 aneurysms), by eight of nine laboratories. The surgical sidewall carotid model was the next most frequently used, with two laboratories, six articles, 67 aneurysms in canines, and one laboratory, one article, 22 aneurysms in rabbits. Six complex canine models were reported in seven articles, including a curved sidewall (two articles, nine aneurysms), an end-wall bifurcation (two articles, 23 aneurysms), a Y-bifurcation (one article, 20 aneurysms), and a giant fusiform aneurysm model (one article, six aneurysms). Mean aneurysm dimensions for the various models are presented in Table 1.

Table 1 Aneurysm models
Full size table

Articles reported a mean of 17.7 ± 8.5 rabbits per study (elastase model); studies on canine models reported 11.2 ± 6.2 animals per article. Published operative mortality rates were 1.4 % (95 %CI [0.6–2.9 %]) in the rabbit studies and 3.4 % (95 %CI [0.1–6.7 %]) in the canine studies. Most aneurysms were reported to be patent at the time of treatment (462/462 rabbit aneurysms; 165/168 canine aneurysms).

Flow diverters

The PED was most frequently studied, but only in rabbits (153 devices in 153 rabbit aneurysms). The FRED device was mostly used in canine aneurysms (137 devices; 115 devices in 85 canine aneurysms). Twelve different devices were tested in the rabbit-elastase model. Most studies (39/42, 92.8 %) used a single flow diverter (in 476 aneurysms), while multiple telescoping devices were studied in 3 articles (65 devices to treat 26 aneurysms). Two articles assessed intra-saccular FD (36 devices in 36 rabbit elastase-induced aneurysms). Details about the number of articles assessing each device are available in Table 2.

Table 2 Details of FDs used in each species
Full size table

Results of flow diversion

Aneurysm occlusion was studied in 39/42 articles (92.8 %); the three remaining articles focused on the patency of jailed branches. An explicit definition of aneurysm occlusion was reported in 21/39 (53.8 %) articles, with the scale of Kamran et al. [13] most commonly used (eight articles). Eight other articles (19 %) used a 3-point occlusion scale (complete/near-complete/incomplete occlusion).

Intra-arterial catheter angiography was used to assess results of FD in 32/39 articles (82 %). Intravenous angiography was used in 4/39 articles (10.2 %). The occlusion of aneurysms was only verified at pathology in 4/39 articles (10.2 %).

A comparator or control group was reported in 13/39 (33.3 %) articles (nine canine studies and four rabbit studies): 7/39 (17.9 %) studies included aneurysms treated with non-FD devices; 3/39 (7.7 %) included untreated control aneurysms, and 3/39 articles (7.7 %) involved both types of controls.

The various experimental methods and models, species, devices, outcome measures, follow-up periods, and definitions of angiographic occlusion were too heterogeneous to allow meta-analysis.

Aneurysm occlusion was most commonly assessed at 3 months (20/39 articles [51.3 %]). Results are summarized in Table 3. Details are provided in Table 4.

Table 3 Summary of aneurysm occlusion rates at 3 months
Full size table
Table 4 Summary of aneurysm occlusion rates
Full size table

Three months after FD, rabbit elastase-induced aneurysms were completely or near-completely occluded in 50/68 cases (73.5 %, 95 %CI [61.9–82.6 %]); surgical sidewall aneurysms were completely and near-completely occluded in 9/9 cases (100 %; [65.5–100 %]). Canine complex aneurysms were completely and near-completely occluded in 12/78 cases (15.4 %, [8.9–25.1 %]).

All seven industry-funded studies involved rabbits: five used the rabbit elastase-induced model, one used the rabbit surgical sidewall model, and one study focused on the patency of jailed branches after device implantation in the abdominal aorta. The overall rate of (near)-complete occlusion of the industry-funded studies was 98/105 aneurysms (93.3 %, [86.6–97.0 %]).

Anti-platelet regimen

Details regarding the anti-platelet regimen were available in 35/42 (83.3 %) of articles. ASA and clopidogrel were used in 25 articles, a single anti-platelet agent in five articles. Anti-platelet agents were not administered in four studies.

Complications

Five articles reported seven parent artery occlusions visible on follow-up angiograms. In two articles (four parent artery occlusions), the authors proposed that complications were due to a failure to fully deploy the devices.

There was a single reported branch occlusion (out of 18 articles) in which retrograde flow in a jailed right vertebral artery was noted.

Because none of the articles reported a denominator (the number of arteries or branches that could have become occluded after flow diversion), a rate of arterial or branch occlusion could not be extracted in this review.

Parent artery stenosis was assessed in 22/42 articles (52.4 %) (eight canine and 14 rabbit studies). Among the 22 studies, at least one parent vessel stenosis was noted in 5/8 canine studies (62.5 %) and 3/14 rabbit studies (21.4 %). Because of suboptimal reporting and heterogeneity, a rate of parent artery stenosis could not be extracted from this data.

No hemorrhagic complications from FD were reported in any article.

Pathological analyses

Pathological results were reported in 31/42 (73.8 %) of articles. Most common findings were that treated aneurysms were partially or completely filled with organized connective tissue (18 articles; seven canine and 11 rabbit studies), and partial or complete neointimal coverage of the device overlying the aneurysm ostium was found (19 articles; six canine and 13 rabbit studies).

Discussion

The main findings of this review are (1) almost all laboratories used a standardized rabbit-elastase model, featuring small aneurysms with a narrow neck, to test 12 different devices. Occlusion rates were high at all time points with most devices; (2) fewer laboratories used surgical sidewall aneurysms; only three devices were tested in rabbits and canines, with high occlusion rates; (3) more complex canine models were used to test FD in contexts of treatment failure (large or giant wide-necked aneurysms at bifurcations for example), with modest occlusion rates (approximately 15 %); (4) branch or parent vessel occlusion has rarely been reported, and none of the hemorrhagic complications reported in humans treated with FD have been described in animals; (5) reporting of animal studies could benefit from compliance with the ARRIVE guidelines.

The capacity for animal studies to predict the clinical efficacy of innovative treatments has been questioned, particularly after the disappointing results of clinical trials on neuroprotective agents shown effective in preclinical studies [14–16]. There are many reasons for the discrepancies between animal and clinical results. One surmountable problem is the lack of transparent reporting of study design and implementation [14, 17]. Recent studies have shown that 80 % of reported animal studies do not describe key aspects of study design, such as randomization or method of blinding [17, 18]. In one survey, important adverse events in animals were reported in less than 10 % of publications [19]. Inadequate reporting may reduce the credibility of preclinical studies and act as barriers to successful translation of research findings. The ARRIVE guidelines were developed to promote consistency in reporting of animal studies [9]. They have been endorsed by the US National Research Council Institute for Laboratory Animal Research [20, 21]. The present review suggests that the design and reporting of animal studies on FD could be improved if investigators complied with available guidelines. Items that can substantially influence results, such as randomization, blinding, or justification of sample size, were infrequently used or reported.

Another barrier to the reliability of preclinical reviews is the presence of publication bias [22, 23]. The predominance of positive studies for various devices in terms of aneurysm occlusion and safety, and the low mortality and spontaneous aneurysm occlusion rates suggest the present review may be affected by publication bias.

Most devices were studied in a generally practiced, apparently standardized rabbit elastase aneurysm model. The severity of the testing accomplished with this model may be questioned when one considers the variety of treatment strategies (including coiling or second-generation coiling [24, 25], and high-porosity stenting [26, 27]) previously shown successful and the high occlusion rates reported in the present review. Yet, the overall occlusion rates obtained with FD in the rabbit model is in line with human results reported in systematic reviews of clinical case series [28].

Similar considerations apply to the ability of preclinical testing to anticipate clinical complications. While the parenchymal hematomas and the delayed aneurysmal ruptures that occasionally follow clinical applications of FD [5, 7, 29] were never reported in animal models (except in a small explanatory study that did not specifically used flow diverters [30]), the reassuring paucity of thrombotic complications seems to have correctly anticipated the relatively small number of ischemic complications encountered in human applications [6, 28]. It is important to remember that few studies looking at arterial patency rates in a small number of animals at fixed time points (which could miss occluded arteries that subsequently recanalized) should not be expected to anticipate clinical complications that could follow even a transient occlusion of a human perforating artery.

Results of FD were heterogeneous. The wide discrepancies between occlusions reported in rabbit models (and sidewall canine aneurysms) and failures reported in other canine models can readily be explained by differences in the configuration of the parent vessel-aneurysm complex (such as surgically created bifurcations), parent artery sizes and flow rates, aneurysm dimensions, in addition to biological differences between species. Thus, a meta-analysis of all results obtained with the various models combined to estimate the overall success of FD, in an attempt to anticipate results in human applications, would not make sense. The more severe testing that can be achieved with some surgical canine models was not designed to match a clinical occlusion rate, but to investigate potential causes for failures which can occur when FD is used in challenging clinical contexts [31].

For the same reasons, comparing occlusion rates for various devices in different studies is inappropriate. One must ensure that the same aneurysm model, aneurysm and neck sizes, methodology, time points, and occlusion scales have been used. Ideally, valid comparisons would necessitate a specific experimental design, with blinded adjudication of angiographic outcomes, comparing animals randomly allocated to one device or another.

Various animal models may play different roles. The most frequently used rabbit elastase-induced aneurysm model features small aneurysms that do not severely test the device under examination. It is a standardized model, available in multiple laboratories, that seems to be a good starting point. Failures observed in this relatively easy model may signify that the device being tested needs further improvements.

Some surgical constructions in canines have the capacity to place FD into circumstances where aneurysm occlusion can be a challenge. However, these models have not been replicated and standardized, and they may require a surgical expertise that is not readily available in all laboratories.

Limitations

We included only studies published in English or French, and some publications may have been missed. Positive studies are more likely to be published in English, and the review may suffer from language bias [32]. We did not perform a meta-analysis because of the substantial heterogeneity in methods and results. One important source of heterogeneity, whose impact on treatment results and complications remains unclear, is the differences in biology and response to anti-platelet agents between animals of the same species as well as between different species.

Conclusion

Preclinical studies on FD have shown various results. Occlusion of elastase-induced aneurysms was common after FD. The model is not challenging but standardized in many laboratories. Failures of FD can be reproduced in less standardized but more challenging surgical canine constructions. Quality of reporting could be improved.