How Biomedical Research Can Inform Both Clinicians and the General Public

Abstract

This study involved the collection of clinical practice guidelines ( s) on five noncommunicable disease (NCD) areas from 21 European countries, and extraction of their evidence base in the form of papers in journals processed on the Web of Science ( ). We analyzed these cited papers to see how their geographical provenance compared with European research in the respective subjects and found that European research (and that from the USA, Australia, and New Zealand) was over-cited compared with that from East Asia. In cancer, surgery and radiotherapy research made important contributions to the CPGs.

We also collected medical research stories from 30 newspapers from 22 European countries and the WoS papers that they cited. There was a heavy emphasis on cancer, particularly breast cancer, and its epidemiology, genetics, and prognosis, but new treatment methods were seldomly reported, particularly surgery and radiotherapy. Some of the stories quoted commentators, with those from the two UK newspapers often mentioning medical research charities, which thereby gained much free publicity.

Both sets of cited research papers showed a marked tendency to be over-cited by documents from their countrymen; the ratio was higher the smaller the country's contribution to research in the subject area.

1 Study Objectives

The main purpose of biomedical research is to improve healthcare [22.1], both by the better treatment of patients and by the prevention of illness [22.2]. The second of these goals is often given lower priority by national healthcare systems, because the immediate need to treat patients claims more attention [22.2]. This is, of course, a common problem in policy-making, summed up neatly in the phrase, we were fighting off the alligators, but the real need was to drain the swamp. In effect, longer-term problems that could provide great benefit at a modest cost are neglected in favor of short-term problems that are crying out for a solution [22.3]. The question of what treatment to provide should be answered with reference to the best available science [22.4, 22.5, 22.6], but instead it is often based on the personal experience of clinicians [22.7, 22.8], and the lobbying of special interests [22.9], particularly in healthcare by pharmaceutical companies [22.10, 22.11].

1.1 Importance of Study

We considered that it was important to examine two ways in which biomedical research could influence these two goals. The first is to examine the evidence underlying clinical practice guidelines (CPGs), which are increasingly being used to determine patient treatment [22.12]. The second is to look at the stories in the mass media, which are the main means whereby research is brought to the attention of the public [22.13]. The public includes a wide range of people, from politicians who decide healthcare policy, their expert advisers, clinicians and other healthcare personnel, other researchers, and of course the general public. Nowadays, we are being encouraged to take a more active role in the protection of our own health, assisted by public health legislation [22.14, 22.15]. The latter depends to a large extent on public consensus, and good timing, so that it will readily be put into practice [22.16]. Examples of evidence-based policy [22.17] are the mandatory use of car seat belts [22.18], and the prohibitions on smoking in enclosed spaces such as offices and restaurants [22.19]. Others currently being considered, but having difficulty making headway against determined industrial lobbying, are minimum prices for alcoholic beverages and restrictions on sugary soft drinks [22.20].

The work to be described formed part of a major European Union (EU)-funded project on the mapping of European research on five noncommunicable diseases over 12 years, 2002–2013. Europe or EUR31 was defined as the 28 member states of the EU, plus Iceland, Norway, and Switzerland. The five NCDs were cardiovascular disease including stroke (cerebrovascular disease), designated as ; diabetes or ; mental disorders or ; cancer or ONCOL; and respiratory diseases or RESPI. This was undertaken in 2014–2015 by King's College London (KCL) in association with six partners: the London School of Economics (LSE, the coordinator); in Estonia, the Estonian Research Council, Tartu; in France, Université Paris Est Créteil, Paris; in Germany, Technische Universität Berlin; in Italy, Università Commerciale Luigi Bocconi, Milan; and in Spain, Escuela Andaluza de Salud Pública, Granada. These partners assisted us with the updating and calibration of the five NCD filters that were used to identify research papers in the Web of Science (WoS, © Clarivate Analytics, formerly part of Thomson Reuters) and with the work described in this chapter. We also recruited KCL graduate students from most of the other EU Member States with the necessary research and language skills to read European CPGs and newspaper stories citing research.

1.2 The Development of Clinical Practice Guidelines in Europe

Clinical practice guidelines started to be described in the literature in 1971 [22.21], and the first papers were all from the USA. There was rather little notice taken of them in the 1970s and 1980s, but interest really began in the 1990s. After something of a lull in the 2000s, interest picked up in the 2010s (Fig. 22.1). There was also a shift in the countries that were involved. In the 1970s it was only the USA, but the UK started publishing in the 1980s, and was gradually overtaken by the other EUR31 countries, and the rest of the world (Fig. 22.2).

Fig. 22.1

Increase of numbers of articles in the Web of Science (WoS) with clinical guidelines in their title from 1971 to the present, 3-year running means

Fig. 22.2

Percentages of total set of clinical guidelines articles in the WoS in each of five decades from the USA, the UK, the rest of Europe (EUR30), and the rest of the world (), integer counts. Note: percentages sum to more than ${\mathrm{100}}\%$ because of international collaboration. In the 1970s, some papers had no recorded addresses

Figure 22.2 makes clear that the European Union (plus the two European Economic Area ( ) states and Switzerland) has been increasingly dominant. In parallel with the growing numbers of research papers, there has been a similar growth in the numbers of CPGs published in European countries, see Fig. 22.3 which shows the numbers that concern diabetes.

Fig. 22.3

Numbers of clinical practice guidelines in diabetes from EUR31 countries, 2002–2013, 3-year running means

Although our search for these CPGs was not exhaustive, we did identify ones from 21 countries out of the 31. In the UK, there are two organizations that publish CPGs: the Scottish Intercollegiate Guidelines Network (SIGN), which began operations in 1993 [22.22], six years before the one for England and Wales, the National Institute for Clinical and Care Excellence (NICE) [22.23]. In France, Guides Parcours de Soins are published by the Haute Autorité de Santé [22.24]. In Germany, a large number of organizations, some federal and some private nonprofit, come together to prepare and publish CPGs, Nationale VersorgungsLeitlinen [22.25] as the collection of logos on a diabetes guideline shows (Fig. 22.4). In Italy, some CPGs are developed by the Istituto Superiore di Sanità in partnership with its parent department, the Ministero della Salute [22.26]; others are produced by nonprofit societies (Figs. 22.5 and 22.6). This is also common in Spain, where the societies form a group and the CPGs are published in an academic journal, but other guidelines are sponsored by national and regional ministries [22.27] (Figs. 22.7 and 22.8). So there is a wide variety of publishers of European CPGs, and both governments and nonprofit organizations are involved. A more comprehensive list of European CPG providers is provided in the Appendix, Table 22.B17.

Fig. 22.4

The German organizations that were associated with a diabetes guideline

Fig. 22.5

The Italian organizations that were associated with a diabetes guideline

Fig. 22.6

A larger group of Italian nonprofit organizations involved in a diabetes guideline

Fig. 22.7

Sponsors of Spanish CPG in diabetes, including Catalonia

Fig. 22.8

Another example of Spanish diabetes CPG sponsors, including the Basque Country

1.3 Previous Work on the References on Clinical Practice Guidelines

Although clinical practice guidelines started to be described in the literature in the 1970s, it was almost two decades before their impact on medical practice was evaluated [22.28] or their evidence, in the form of the cited references, began to be considered as a means to evaluate biomedical research [22.29, 22.30]. Grant's conclusions are still valid, namely that the papers cited on CPGs published by the NICE are clinical rather than basic; that they are fairly recent; and that they tend to over-cite research by own-country authors. This last conclusion strictly only applied to British guidelines as there does not appear to be any comparable study based on those of other countries. Subsequently, the scope of these studies was extended to a wider selection of CPGs [22.31, 22.32, 22.33], including those from the SIGN and the British Medical Association's handbook, Clinical Evidence. Subsequently, Kryl et al [22.34] showed that the references on two CPGs from the NICE, on dementia and chronic obstructive pulmonary disease (), could provide a useful tool to evaluate medical research, particularly if the cited papers contained data on their funding sources—as papers in the WoS routinely do since late 2008.

A major part of the reason for the lack of further use of this tool is undoubtedly that it is difficult to extract the relevant information from the CPGs that would enable each reference to be tabulated with its salient information, such as the authors' addresses and the details of its funding. This is effectively a three-stage process. First, the relevant CPGs have to be found; usually (but not always) they are freely available on the Web. Their titles need to be translated, as those from most non-Anglophone countries will be in the local language. Second, their references have to be identified and collected from the CPG by means of a copying and pasting procedure to a spreadsheet, and then processed so as to give a series of standardized search statements that can be applied to the WoS. Third, they have to be sought in the WoS and their details have to be downloaded to file ready for analysis. This process is described in more detail in Sect. 22.2.1; it is inevitably rather labor-intensive because the different CPGs give references in different formats, and sometimes the references even within one CPG can differ in their format, although a certain degree of assistance can be provided with a visual basic application ( ) macro. Moreover, some of the references will not be to papers in the serial literature; these cannot be processed in the same way and are usually ignored. A series of VBA programs (written by Philip Roe of Evaluametrics Ltd.) were developed to assist with the extraction of the bibliographic details of the research papers. These VBA macros were specifically formulated to work with the WoS and this was particularly useful as the WoS permits the retrieval of grey literature including conference proceedings not covered in other databases, as well as providing options for accessing address details, country of publication, citation impact metrics, and other parameters in an analyzable format.

1.4 Newspaper Stories About Medical Research

In contrast to the paucity of papers about the use of the references on CPGs for research evaluation, there are over 50 papers in the WoS that concern how the mass media, particularly newspapers and magazines, report medical research, and their number has been increasing faster than the numbers with medical and research in their titles (Fig. 22.9).

Fig. 22.9

Numbers of papers in the WoS in each decade since 1960 that have medical and research in their titles (squares and solid line, left scale), or concern how medical research is reported in the news media (diamonds and dashed line, right scale)

Of these papers, the majority (31 out of the 52 with addresses) were from the USA, but this may reflect the bias in WoS coverage of social science journals, which accounted for half of the papers. Other countries that contributed papers were continental Europe (8), the UK (7), Canada (4), Japan (2), and China and Russia (1 each). A majority of the papers analyzed stories in newspapers (28) and/or magazines (12); only a handful were concerned with broadcast media, probably because archives of radio and television are less common and less easy to analyze. Many papers looked at how the general public, or sections within it, reacted to mass media coverage of a topic, and especially the effectiveness of public health campaigns [22.35]. Two papers showed that mass media coverage also led to more citations of the research publications as a result [22.36, 22.37]. The most popular disease area to be analyzed was cancer (25 papers), followed at some distance by mental health, including alcoholism (7), cardiovascular research (2), and arthritis and diabetes (1 each).

2 Methodology

A systematic approach was used to identify and download the CPGs, collect the cited evidence-base on these CPGs, download their bibliographical details, and conduct the analysis. Another search strategy was developed to identify the newspaper stories that reported medical research, enter their details to a spreadsheet, and then download the bibliographical details of those research studies for the analysis.

2.1 The References on Clinical Practice Guidelines

The first task, of course, was to find the CPGs in the various countries. Since most of them were in languages other than English, we called upon our European partners, other European collaborators, and the KCL graduate students to search for these on the Web, and to provide us with copies, normally in pdf format, with translations of their titles. Altogether we were able to obtain CPGs from 21 countries; some of the others did not appear to have any (e. g., Cyprus at the time, others may have had them, but lack of resources meant that we could not obtain them, e. g., Iceland and Norway). We compiled lists of all these guidelines, and it was immediately apparent that there were far too many for us to be able to process them all, since some had upwards of a thousand references, and many had several hundred, and they tended to be in different formats.

We therefore needed to make a selection, and decided to cover those CPGs that referred to diseases or disorders that were responsible for ${\mathrm{1}}\%$ or more of the disability-adjusted life years ( s) in EUR31 in 2010 as given by the Institute for Health Metrics and Evaluation at the University of Washington, USA [22.38]. These were as shown in Table 22.1, and included one or more disease areas from each of the five NCDs.

Table 22.1 List of 13 noncommunicable diseases or disorders selected for the analysis of their European clinical practice guidelines, with the estimated percentage European disease burden in DALYs in 2010

The next step was to find the references on each of these guidelines. Some CPGs had them neatly gathered together at the end (for example, those from SIGN); others had them at the end of each chapter or section. Naturally, their format varied according to the source of the CPG, and sometimes even within the same document. The normal format included the names of several authors, the title, the year, and the source (journal, volume, pages). However, the Finnish CPGs only gave the name of the first author and did not give the title of the cited paper, and special arrangements were needed for these references (see below).

The reference section was copied and pasted into an Excel spreadsheet. For some CPGs, the references were numbered sequentially, which allowed a specially-written VBA program to identify where each reference ended and the next began (because many ran on to two or more rows in the spreadsheet). For those that did not, the reference section was copied and pasted into MS Word, so that the numbers could be manually inserted before being transferred back to a spreadsheet.

The VBA program then parsed each reference into a standard form of WoS search statement that included up to three authors, plus the three longest words from the reference title, plus the year, plus the initial letter of the journal. (We could not use the journal name because this was often abbreviated in a non-standard format.) An example is given below.

2.1.1 Original Form of Reference

Jensen DM, Damm P, Sorensen B, Molsted-Pedersen L, Westergaard JG, Korsholm L, et al Proposed diagnostic thresholds for gestational diabetes mellitus according to a ${\mathrm{75}}\,{\mathrm{g}}$ oral glucose tolerance test. Maternal and perinatal outcomes in $\mathrm{3260}$ Danish women. Diabet Med 2003 Jan;20(1):51–57.

2.1.2 Format when Search Statement Prepared

AU=(Jensen, D AND Damm, P AND Sorensen, B) AND TI=(diagnostic AND thresholds AND gestational) AND PY=2003 AND SO=D*

These search statements were then grouped automatically into sets of a selected number, typically 20, that could be run against the WoS. References that did not appear to fit the format for journal papers were initially ignored by the program, but could be added later. Sometimes this was because the punctuation was not exact. (For example, the program expected journal references to have just three full stops: one after the list of authors, one after the title, and one at the end. In the example above, there are four because the title consists of two sentences, so this reference would initially have been rejected until the full stop after test was removed. Some other references were rejected because the title ended with a question mark.) It was also necessary to check that author names did not have accents, or other diacritical marks such as umlauts or Ø letters, as these are not used in the WoS.

For some sets of references where the numbers had been inserted manually in MS Word, they could each be separated by a paragraph mark, and then when they were transferred to Excel, each was on a single line. This enabled the different elements of the reference to be spread across to different columns, provided that the separators (usually a full stop) could be identified and they were correctly placed in the reference (see above). The CONCATENATE function could then be used to prepare search statements using the first author's name, the full title, and the year. The compound search statements were then run against the WoS for all years and all document types. However, sometimes these did not run if syntactic rules were inadvertently broken, such as the inclusion of terms such as and/or or the word near in the paper title.

Sometimes a compound search statement with, say, 20 individual statements yielded fewer than 20 papers; this was usually because one or more references were not in journals processed for the WoS in the given year. However, the reverse could also occur, with perhaps 22 or 23 papers identified. This was nearly always because the WoS had also recorded corrections or letters to the same journal about the original paper, together with the authors' reply, which would have had the same author(s), title and year, and so satisfied the search statement. We subsequently discarded items described by the WoS as corrections and letters as they would not have reported research results.

There was a particular problem with the Finnish CPG reference lists. The references were provided as a continuous list, such as this extract:

2014;23:39–46 80. Okin PM ym. Hypertension 2000;36:766–73 81. Koren MJ ym. Ann Intern Med 1991;114:45–52 82. Casale PN ym. Ann Intern Med 1986;105:173–8 83. Devereux RB ym. Hypertension 1994;23:802–9 84. Anavekar NS ym. N Engl J Med 2004;351:1285–95

The complete set of references on a CPG was first pasted into a Word document, and then paragraph marks were inserted before each reference number. They were then pasted into Excel, and the string ym (the Finnish form of et al) removed from each where it occurred. The reference was then spread across columns containing the first author, the publication year, the journal name as given, and the volume number and pagination. The journal name had then to be converted into the full journal name with a journal name thesaurus. The resulting search strategies were then run against the WoS, and the papers downloaded to file, up to 500 at a time. These were converted into an Excel spreadsheet by means of another VBA program which put the downloaded data into a standard format for analysis. However, this file contained many additional records by the named author in the given journal and year, and those not conforming to the given page numbers were subsequently removed by hand.

The analysis of the downloaded files was carried out by a further set of VBA programs that carried out the following functions:

• Characterization of each paper as clinical or basic or both according to the presence of one or more words from two lists of selected words in its title [22.39]

• Provision of the fractional counts of each country listed in the addresses

• Identification of the disease area (e. g., cancer site, such as breast or colon, or mental disorder, such as Alzheimer's or depression)

• Identification of the research domain (e. g., genetics, surgery).

This classification enables an understanding of the type of research, whether it is applicable to patients in the clinic, at the laboratory stage, or a mix of both. Research level (RL) is designated by a decimal number between ${\mathrm{1.0}}=$ clinical and ${\mathrm{4.0}}=$ basic. Each paper cited within a guideline was classified as clinical (1) or basic (4) or both (2.5) and these values were averaged to yield the research level of the set of cited papers, shown as RL ($p$). The same process was repeated to get the average research level value based on the journals in which these papers were cited as described above, shown as RL ($j$)

Information was also available on the gap (in years) between the date of the citing CPG and that of the cited paper [22.33], and on its funding (if it was published in 2009 or later). The latter topic, which was discussed in a recent paper [22.40], will be explored in a later paper.

The references on the clinical practice guidelines for each of the five NCDs were collected together in five separate spreadsheets, and the results are presented separately. Overall comments on them are brought together in the Discussion section. Countries' tendency to cite their own papers on CPGs is examined for all five NCDs.

2.2 The Newspaper Stories and the Research That They Reported

Our original intention had been to select one, two, or three newspapers from each of the 31 European countries, with the larger countries being represented by papers with different political outlooks, readership strata, and geographical origins. It was also necessary for the papers to have an easily searchable archive, or for their full texts to be available on the Factiva © Dow Jones database. However, it proved difficult to find researchers with all the language skills needed (we were eventually able to cover 18 European languages but these did not include Icelandic, Latvian, Lithuanian, Maltese, Norwegian, Slovakian, and Slovene). The time necessary to process newspaper stories for the 12 years of the study period (2002–2013) also prevented us from covering as many newspapers as we would have wished, as it turned out that our search strategies for capturing relevant stories from newspaper archives yielded a large number of false positives, which had to be read individually in order for irrelevant stories to be discarded. (Stories were only retained if they cited research from one or more papers from journals covered in the WoS.) Another limitation was that some newspaper archives did not go back as far as 2002.

Table 22.2 shows a list of the newspapers that were processed. Each newspaper was given a code, consisting of its country ISO2 code (Table 22.3) and one letter, normally the initial letter of its name. Most newspapers had their own searchable databases, but for some the researchers used the Factiva full-text database © Dow Jones. The search strategies used included the names of the relevant NCD diseases or disorders, and a set of terms indicative of research, thus:

(cancer OR leukaemi* OR melanoma* OR lymphoma*) AND (research* OR study OR scientist* OR expert*)

This search strategy, and the four others like it, were all used to search for relevant stories. They were translated into the 17 languages needed to search the non-English language newspapers.

Table 22.2 List of 31 European newspapers whose medical research stories were collected and used to create the file of news stories

Table 22.3 ISO2 codes for countries whose outputs are considered in this study

The researchers were all brought to KCL in groups for training. They were asked first to translate the search statements into their own languages, and then to check the selected newspapers' websites for their ability to be searched with the five search statements, and for how many years this could be done. Most of these archives were freely accessible, but for some we needed to subscribe to the newspaper for a short period in order to be granted access. The researchers were also taught about the use of short codes to connote the disease(s) or disorder(s) mentioned in the newspaper stories (three-character, or trigraphs) and the research domain (four-character, or tetragraphs). The NCD was to be connoted by its pentagraph code; some stories covered more than one of them.

The details of each selected newspaper story were then to be copied and pasted, or typed, into a spreadsheet containing 26 columns, as listed in Table 22.4.

Table 22.4 Columns on spreadsheet giving details from selected newspaper stories

The notes item (column W) was simply to assist the researcher to identify the cited paper in the WoS. This was the next task, and each paper that was identified was then downloaded with as identifier the index number of the story. (A few stories cited more than one research paper; these were given consecutive index numbers.) The full records were downloaded, and the details were converted into an Excel spreadsheet by means of the VBA macro used for the references on CPGs: they were then copied across to the spreadsheet of the stories with columns as in Table 22.5.

Table 22.5 Columns on spreadsheet giving details of cited papers from the Web of Science

The cited papers were than analyzed by means of VBA programs in a similar way to that used for the references on CPGs, see the previous section. This added many extra columns to the spreadsheet.

3 Results: Clinical Practice Guidelines

In total, we selected 413 CPGs in 26 European countries across the five NCD areas with an evidence-base of $\mathrm{47274}$ cited research papers. These were identified from the selected CPGs and their details identified and downloaded from the WoS for analysis.

3.1 Clinical Practice Guidelines—Cardiovascular Research and Stroke (CARDI)

We processed 74 CPGs in this disease area from 19 countries; 54 of the CPGs were for coronary heart disease () and 20 were for stroke (STR). (These were the two disease areas that had been selected for study within CARDI, see Table 22.1). There were $\mathrm{11762}$ references in total, of which $\mathrm{7447}$ were for heart disease and $\mathrm{4315}$ for stroke.

As is usual with the papers referenced on CPGs, the papers were very clinical, with mean RL ($p$) varying only slightly, from $\mathrm{1.12}$ for the papers cited by Austrian and Swedish CPGs to $\mathrm{1.03}$ for those cited by Spanish CPGs. However, the RL of the journals appeared less clinical, and the average RL ($j$) was $\mathrm{1.36}$. Some of the journals were more basic: $\mathrm{2110}$ papers (${\mathrm{18}}\%$) had RL ($j$) $> {\mathrm{1.5}}$ and 220 of them (${\mathrm{1.9}}\%$) had RL ($j$) $> {\mathrm{2.5}}$.

Some countries' research is much better cited on the European CPGs than others, and this is shown in Fig. 22.10. This figure shows that the spots for most European countries lie above the diagonal line, and those for East Asian countries lie below it. This is not surprising because the CPGs are all European and there is a tendency for research documents preferentially to cite papers by the countrymen of the document authors [22.33, 22.41, 22.42].

Fig. 22.10

Percentage presence of different countries in papers cited in European clinical practice guidelines for cardiovascular disease and stroke as a function of their percentage presence in CARDI papers in the Web of Science, 2002–2013, integer counts. For country codes, see Table 22.3. Dashed lines show percentage presence twice or half that expected on the basis of countries' presence in the WoS

Nevertheless, some non-European countries' research is well cited by these CPGs, notably that of Canada, Australia, Israel, and the USA. It might be supposed that this is just because the English-language UK CPGs cite a substantial proportion of the references ($\mathrm{2632}$ out of $\mathrm{11762}$, or ${\mathrm{22}}\%$) but this is not the case, as Fig. 22.11 demonstrates, where these four countries' papers appear to be at least as well cited by Austria + Germany, and by France + Poland.

Fig. 22.11

Percentage presence of USA, Canada, Australia, and Israel among the papers cited by CARDI CPGs from the UK ($n={\mathrm{2632}}$), from Austria and Germany ($n={\mathrm{2658}}$), and from France and Poland ($n={\mathrm{2411}}$), integer counts

The tendency of research documents to cite papers by their fellow countrymen is shown by Table 22.6, which shows the over-citation ratio ( ) for eight European countries and also these countries' presence in CARDI research. The OCR is a measure of the tendency of authors preferentially to cite papers by their fellow countrymen [22.41]. It is greater for countries with smaller scientific outputs, and has also tended to decrease with time as international communication has become easier and cheaper.

Table 22.6 Over-citation ratio for own country papers cited in eight countries' CARDI CPGs

Some over-citation ratios are somewhat smaller than expected, particularly from Spain and Portugal.

3.2 Clinical Practice Guidelines—Diabetes (DIABE)

These guidelines were not divided up by disease area, or by the sequelae that often result from diabetes, as the WHO and IHME data on disease burden do not distinguish between them. There was a total of 101 guidelines, see Fig. 22.3, from 25 countries, with a combined total of $\mathrm{5941}$ references. However, this total included many papers that were cited multiple times on these CPGs, with two papers being cited on as many as 17 of them. Figure 22.12 shows that the distribution of citations follows a logarithmic pattern.

Fig. 22.12

Distribution of citation scores for diabetes papers on 101 European CPGs

The gap between publication of the CPGs and of the references that they cite is shown in Fig. 22.13 with, for comparison, the distribution by year of synchronous citations (references) on a sample of diabetes papers from 2013, both as percentages of citations over a 19-year period. This suggests that the references on the CPGs are relatively recent: half appeared no more than five years previously. However, the CPG countries varied in how recent their cited references were: this is shown in Fig. 22.14. Finland, Germany, and the UK cite relatively old papers, but Croatia and Portugal relatively recent ones.

Fig. 22.13

Gap between dates of cited references on diabetes clinical practice guidelines and the guidelines (light brown line), and comparison with the time distribution of synchronous citations in 2013 from diabetes papers in the WoS (dark brown line)

Fig. 22.14

Mean gap between publication of a clinical guideline for diabetes and the references that it cited, for 15 European countries whose guidelines cited at least 75 references. Countries with $> {\mathrm{600}}$ cited references shown by brown bars; countries with $> {\mathrm{250}}$ cited references shown by gray bars; with $<{\mathrm{250}}$ references shown by beige bars

The next analysis was of the subject areas of the cited references and a comparison with the subject areas of European diabetes research in 2002–2013. The subject areas were connoted by trigraph codes, listed in Table 22.7 and the comparison is in Fig. 22.15.

Table 22.7 List of diabetes research subject areas, with trigraph codes

Fig. 22.15

Relationship between European diabetes research subjects, 2002–2013, and the evidence base of 101 European CPGs. For subject area codes, see Table 22.7

It appears that there is a reasonable match between the subject areas in which European diabetes research is being undertaken and those that are important in the provision of the evidence base. This is not the case for cancer, as we shall see. The subject areas that are of less utility in the provision of this evidence are genetics and effects on the liver. The latter may be due to there being few CPGs covering this subject area. Conversely, there is a lack of research on the effects of diabetes on the feet, which appears important for clinical practice.

The tendency for countries to cite their own papers among the references on their clinical practice guidelines is examined in Table 22.8. The ratios are higher than they were for the CARDI CPGs, and Spain (with Portugal) and Germany are again citing their own papers less often than do the other countries.

Table 22.8 Over-citation ratio for own country papers cited in ten countries' DIABE CPGs

3.3 Clinical Practice Guidelines—Mental Disorders (MENTH)

This NCD differs from the others because the individual disorders are so different from each other. The CPGs selected for analysis covered five disorders: drug use disorders (addiction) (ADD); alcohol misuse disorders (ALC); Alzheimer's disease and other dementias (ALZ); anxiety disorders (ANX); and unipolar depressive disorders (DEP). Altogether, we analyzed 100 CPGs from 20 European countries citing $\mathrm{12442}$ research papers and their division between the disorders, together with the numbers of references, the European disease burden, and the amount of European research, is shown as a chart in Fig. 22.16. There appears to be a reasonable correspondence between the four parameters for each disorder: depression attracts the most research activity, and there are more references per CPG than for the other disorders.

Fig. 22.16

Numbers of European mental disorders (MENTH) clinical practice guidelines, and the percentages of references, burden in DALYs, and research outputs for each of five disorders

The presence of the leading countries among the cited references on these CPGs is compared with their presence in mental disorders research in Table 22.9.

Table 22.9 Percentages of countries' papers in mental disorders research, 1995–2011, and among the cited references on European CPGs for mental disorders (CPG cites), and ratio between them. Country codes in Table 22.3. Cells with values: $> {\mathrm{2.0}}$ shaded bright green; $> {\mathrm{1.41}}$ shaded pale green; $<{\mathrm{0.71}}$ shaded yellow; $<{\mathrm{0.5}}$ shaded pink

The UK, Sweden, Finland, and the Netherlands show to advantage here, and the non-European countries in Asia and South America are less cited, including Israel. Country self-citation ratios are rather higher than they were for CARDI (Table 22.10). It is striking that Germany and Spain cite their own papers less than do the other countries, as was the case for the CARDI and DIABE CPGs.

Table 22.10 Over-citation ratio for own country papers cited in ten countries' MENTH CPGs

3.4 Clinical Practice Guidelines—Cancer (ONCOL)

The three most serious cancers—lung, breast, and colorectal—were the ones for which CPGs were selected. There were a total of 81 ONCOL CPGs: 31 for breast cancer (MAM, $\mathrm{3748}$ references), 30 for lung cancer (LUN, $\mathrm{4319}$ references), and 20 for colorectal cancer (, $\mathrm{1773}$ references). Figure 22.17 shows a log–log plot of 18 countries' presence among the cited references compared with their presence in cancer research in the WoS for 2002–2013.

Fig. 22.17

Percentage presence of different countries in papers cited in European clinical practice guidelines for cancer as a function of their percentage presence in ONCOL papers in the Web of Science, 2002–2013, integer counts. For country codes, see Table 22.3. Dashed lines show percentage presence four times, twice or half that expected on the basis of countries' presence in the WoS

The same pattern appears as with CARDI papers: most European countries' papers are relatively over-cited, and those from the three East Asian countries (China, Japan, Korea) are under-cited, here by a factor of about two. Belgian papers are the most cited relative to their presence in the WoS, followed by those of the Netherlands, Canada, and the UK. Each country's CPGs over-cite their own countrymen's papers by factors shown in Table 22.11.

Table 22.11 Over-citation ratio for own country papers cited in 12 countries' ONCOL CPGs

Although Germany has a low OCR value, as it does in the other NCDs, Spain and Portugal have much higher ones than usual, suggesting that their cancer research is of greater utility than their research on the other NCDs. Finland and Sweden are again quite reliant on their own research.

The next analysis was of the research domains of the papers that were cited on the CPGs, compared with those of European cancer research in 2002–2013. This is shown in Fig. 22.18, where the abscissa is the percentage presence of each of 11 research domains and the ordinate is their presence in the $\mathrm{9840}$ CPG references.

Fig. 22.18

Comparison of presence of each of 11 cancer research domains in EUR31 papers, 2002–2013, and corresponding percentages in the references on European ONCOL CPGs. Dashed lines show values twice and half the expected values. Domains: CHEM $=$ chemotherapy, DIAG $=$ diagnosis, EPID $=$ epidemiology, GENE $=$ genetics, PALL $=$ palliative care, PATH $=$ pathology, PROG $=$ prognosis, RADI $=$ radiotherapy, SCRE $=$ screening, SURG $=$ surgery, TARG $=$ targeted therapy

The main conclusions are two-fold. Genetics research, which is by far the most popular research domain, is of little importance to the development of most CPGs. Conversely, surgery and radiotherapy, which are the main means of curing cancer (as opposed to its palliation), are of great importance for CPGs but are less popular with researchers, are not well funded, and are not well cited in the literature [22.33, 22.43, 22.44]. Screening is also under-researched compared with its contribution to the evidence base of cancer CPGs.

3.5 Clinical Practice Guidelines—Respiratory Diseases (RESPI)

This was by far the smallest of the five NCDs that we investigated [22.45]. Most of the research was on just two diseases: asthma and chronic obstructive pulmonary disease (). Asthma was covered by 27 CPGs with $\mathrm{3334}$ references; COPD by 30 CPGs (two covered both diseases) and $\mathrm{4014}$ references. One CPG covered pulmonary fibrosis, with 94 references. Altogether, 57 CPGs from 19 countries were processed, with a total of $\mathrm{7289}$ references.

The comparison between research output from the different countries and their presence on the cited references is shown in Table 22.12.

Table 22.12 Comparison between countries' presence in RESPI research in the WoS and their presence in the references in European RESPI CPGs. Values in cells $> {\mathrm{2.0}}$ shaded bright green; if $> {\mathrm{1.41}}$ shaded pale green; if $<{\mathrm{0.71}}$ shaded pale yellow; if $<{\mathrm{0.5}}$ shaded pink

Once again, the Asian countries' outputs and ratios to their presence among the references on RESPI CPGs are all $<{\mathrm{0.5}}$ (and are shaded pink), and eight European countries have values $> {\mathrm{1.41}}$ (and are shaded bright or pale green).

The over-citation ratios for the countries whose CPGs have at least 100 references and at least one from their own country are shown in Table 22.13. The values for the larger countries are lower than for the other four NCDs, probably because European RESPI research has a bigger presence in the world (${\mathrm{56}}\%$) than its output in the other NCDs, which averaged ${\mathrm{40}}\%$.

Table 22.13 Over-citation ratio for own country papers cited in nine countries' RESPI CPGs

4 Results: Newspaper Stories

From the 31 newspapers covered across 22 European countries, there were $\mathrm{8596}$ cited research studies featured in newspaper stories for the five NCDs. The details of the stories and of the cited research papers were entered into a single spreadsheet for analysis.

4.1 The Five Noncommunicable Diseases

The file of stories and cited papers contained $\mathrm{8596}$ entries, so it was comparable in size with the sets of references on each of the CPGs. Of these, $\mathrm{3498}$, or nearly ${\mathrm{41}}\%$, concerned cancer, and the analysis of these stories and papers is treated in detail in the next section. Here we examine the distribution of the stories between the five NCDs, how it compares with the amount of research on each of them, and their relative disease burden in Europe. We also consider the geographical distribution of the cited papers and their research level.

The numbers of stories were very unequal between countries: the UK and Belgium, each with two newspapers, were much the most productive of data, and there were very few from Austria, Switzerland, and Cyprus (Fig. 22.19). It is not clear if these differences really reflect the amount of interest in medical research in the different countries, or are an artefact of the selection process.

Fig. 22.19

Numbers of newspaper stories about NCD research for 22 countries in 2002–2013. For country ISO codes, see Table 22.3

As mentioned above, of the five NCDs cancer was the disease area most often mentioned. Figure 22.20 shows that this reflects the amount of research into the disease in the EUR31 countries, but exaggerates its burden. This is also the case for diabetes, although the burden from this NCD is increasing. Conversely, cardiovascular disease research is under-reported, as is respiratory disease, though the latter is clearly seriously under-researched [22.45]. Mental disorders appear to be getting a fair share of news space.

Fig. 22.20

Comparative coverage by newspapers of research on five NCDs compared with the volume of European research (2002–2013) and relative disease burden (DALYs, 2012)

The research level of the papers cited by the newspaper stories was fairly similar to that of the European research in 2002–2013, except in CARDI and ONCOL where it is somewhat more clinical. This means that the newspapers are selecting research from the whole spectrum of RL for their stories, in contrast to the papers cited as references on CPGs which are almost entirely clinical (Fig. 22.21).

Fig. 22.21

Mean research levels of papers in five NCDs: cited by news stories, cited by CPGs, and published by EUR31 researchers. RL $={\mathrm{1.0}}$ is clinical observation; RL $={\mathrm{4.0}}$ is basic research

For all the newspaper stories, the over-citation ratios for the leading countries are shown in Table 22.14.

Table 22.14 Over-citation ratio for 13 sets of own country papers cited in newspaper stories

One of the features of the newspaper stories is that many of them included a comment on the significance of the results from an external expert. Altogether, $\mathrm{1520}$ stories mentioned a commentator (${\mathrm{18}}\%$), but the percentages were much higher in the UK (900 with commentators, ${\mathrm{32}}\%$), Denmark (60, ${\mathrm{30}}\%$) and Sweden (92, ${\mathrm{28}}\%$). There did not appear to be any in the German or Italian newspaper stories, but this may have been because the researchers simply did not record them. Table 22.15 lists the ones that were mentioned most often: the list is dominated by UK medical research charities, who are frequently invited to comment by the journalists on the Daily Mail and The Guardian. However, in other countries, most of the commentators are academics, many of them from the USA.

Table 22.15 List of commenting organizations in newspaper stories about NCD research

4.2 Mental Disorders Research Stories and Their Cited Papers

Within the subject area of mental disorders (MENTH), where there were $\mathrm{2175}$ stories and cited papers, the disorders of greatest interest to the journalists were Alzheimer's and other dementias, and depression. This accords with the volume of research, see Fig. 22.22, but it is not in accord with the burden (in DALYs in 2012).

Fig. 22.22

Percentages of news stories, of EUR31 research, and of the disorder burden within MENTH

The figure also reveals that alcohol misuse is a more serious problem than all the other disorders listed in the figure, but is relatively neglected both by researchers [22.46] and by the newspapers. There is also a lack of attention to suicide and self-harm, which accounts for almost ${\mathrm{10}}\%$ of all mental health problems.

The countries authoring the papers cited in the news stories about mental disorders were, as expected, mostly from European countries and the USA (Table 22.16). The comparator is the countries' output of MENTH papers in 2002–2013.

Table 22.16 Countries authoring papers cited in newspaper stories about mental health (newspaper stories (NS) cites) and percentage of MENTH papers from each in the WoS, 2002–2013. For country codes, see Table 22.3. Cells with ratio $> {\mathrm{2.0}}$ shaded bright green; for ratio $> {\mathrm{1.41}}$ shaded pale green; for ratio $<{\mathrm{0.71}}$ shaded pale yellow; for ratio $<{\mathrm{0.5}}$ shaded pink

It appears that the best-cited countries are the ones in Scandinavia, together with the UK, the Netherlands, and Belgium. (The value for Iceland is $\mathrm{11.6}$).

4.3 Cancer Research Stories and Their Cited Papers

We carried out a separate analysis of the cancer sites most often mentioned in the stories about cancer research, and also of the research domains. Figure 22.23 shows the leading sites mentioned in the stories, with, for comparison, the relative percentages of EUR31 cancer research and of the disease burden in Europe in 2012 in DALYs. There is clearly an imbalance in the selection of stories: breast and skin cancer (melanoma) get more coverage in the newspapers than they merit, but lung and pancreatic cancers get little coverage.

Fig. 22.23

Percentages of cancer newspaper stories, of ONCOL research, and of cancer DALYs, from eight leading body sites

Figure 22.24 shows a similar comparison between the different research domains (here, based on the cited papers rather than the codes given by the researchers) and the amount of research by the EUR31 countries. Coverage is again unbalanced: the main topic of the stories is epidemiology and some distance behind come genetics and prognosis.

Fig. 22.24

Percentages of cancer newspaper stories and of ONCOL research in the EUR31 countries, on 12 research domains. EPID $=$ epidemiology, PROG $=$ prognosis, GENE $=$ genetics, QUAL $=$ quality of life, PALL $=$ palliative care, SCRE $=$ screening, CHEM $=$ chemotherapy, SURG $=$ surgery, DIAG $=$ diagnosis, PATH $=$ pathology, RADI $=$ radiotherapy, TARG $=$ targeted therapy

However, there is little coverage by the news stories of the three main methods of treatment—chemotherapy, and especially surgery and radiotherapy—suggesting that the main interest of the journalists is in the prevention of cancer rather than its treatment. Of the treatment methods, chemotherapy and targeted therapy combined are covered in 230 stores, but radiotherapy and surgery combined in only 166. This may well give the public the false impression that cancer is cured by drugs rather than the latter treatments [22.43, 22.44]. To their credit, the journalists do provide better coverage of screening than the researchers do.

5 Discussion

In this research work, we evaluated two indicators of research, specifically citations in the clinical practice guidelines and in newspaper stories. This is the first pan-European study to our knowledge that aimed to inform the research gap between research publications and impact on clinical practice through CPGs or public health awareness through newspapers. Furthermore, through a systematic search methodology, this study covered five noncommunicable disease areas, which again, make it perhaps the largest of its kind through an assessment of two research indicators (clinical practice guidelines, newspapers), multicountry comparison, and different disease and subdisease areas pattern examination. Further research on how best the research evidence base or reporting in newspapers can correspond to the disease pattern affecting each European country can perhaps influence clinical practice as well as inform more effective health-policy practices.

5.1 Limitations of This Study

The first limitation of this study concerns the inevitable selection of sources, of both CPGs and newspapers. This was constrained by the time and the resources available for the study. We were not unduly constrained by language, as King's College London has graduate students from nearly all continental European countries, and we were able to employ them for the short periods needed for their assignments. However, the training provided to the researchers who were responsible for identifying and processing the newspaper stories, and the papers that they cited, was inevitably rather brief. Some of them may not have fully understood all the complexities of the coding system, or the need for a cited paper to be found in the WoS for its citing story to be included, or indeed for the details of commentators to be recorded. We were able to clean some of the data and this task enabled the results from a few countries to be much better as a result. Some of the countries' newspaper stories did seem rather few in number but much of the analysis presented here is based on results from those countries that appeared to have good coverage of medical research, notably the UK and Belgium.

Many of the results presented were compared with other outputs, notably the amount of research carried out on the five NCDs in Europe, and in other countries. We had to select a time frame for these outputs. For most of the comparisons, we used the 12-year study period (2002–2013). This is probably fair for the newspaper stories, as nearly all of these are written about new research that has just been published, but is more problematic for the papers cited on the CPGs, where some references go back many years. There is no right answer for the appropriate time frame to be used, and the use of the last 12 years meant that the outputs of east Asian countries, most of which have increased rapidly (but not those of Japan), may have put them at a comparative disadvantage.

We found a similar difficulty with regard to the disease burden. There are two main sources of data: the World Health Organization (WHO), and the Institute for Health Metrics and Evaluation at the University of Washington. Both have changed their data from time to time as a result of (presumably) better methods of analysis. In particular, there is inevitably dispute about the weights that should be assigned to disabilities that provide the basis of DALYs. A particular disability may prove much more burdensome in some countries than in others, and the methods used make international comparisons difficult. For instance, the mental disorder DALYs depend critically on the severity of the particular condition (e. g., depression), and this is quite hard to determine.

5.2 Advantages of This Study

Despite the above reservations, this study is the first to have compared the references on CPGs and in newspaper stories on five major disease areas and in over 20 countries, with many different languages. It was brought about by the award of a contract by the European Commission, and led to a multinational research activity to find out about the outputs and impacts of European medical research using a standard methodology in these different countries. We are now bringing out a series of papers [22.40, 22.42, 22.45] on the individual disease areas and on the methods of analysis that we have developed that will for the first time show the strengths and weaknesses of European medical research, which is inevitably very fragmented and needs to be better coordinated if it is to be efficient. We have also developed a methodology for the measurement of impacts in the real world that can be compared with the traditional evaluation criteria of citation counts [22.47]. These appear rather one-dimensional in comparison and because they are used so much for the allocation of research grants may distort research priorities.

The methodology described here can, in principle, be used on a much wider scale to provide research funders and research performers with information on how their outputs have influenced medical care through CPGs and the public through the mass media. We are considering how best to develop these information sources commercially. The difficulty is that the varied nature of the source materials makes it hard to automate the process of collection of reliable information, and therefore the cost of data collection would need to be spread across many potential subscribers.

5.3 Main Conclusions of the Study

Perhaps the most important conclusion is that the papers cited on CPGs and in newspaper stories are not the same ones as receive many citations in the serial literature, and that some relatively neglected areas are unexpectedly important for the practice of medicine or for the provision of useful information to the public. In cancer, surgery and radiotherapy have emerged as important areas for the guidance of physicians and surgeons.

In mental health, the public perception of depression as a subject that used to be kept under wraps has changed, and there is now a willingness to accept it as an illness that can and should be treated. Similarly, the treatment of Alzheimer's and other dementias is getting increasingly more attention, and this is aided (in the UK) by the prominence of the two Alzheimer's medical research charities as commentators on mental health stories. This will help them in their mission to raise funds to support new research.

The solicitation of comments from collecting charities by the UK media, which was noted earlier [22.37], occurs in other disease areas, as witness the prominence of Cancer Research UK, the British Heart Foundation, and Diabetes UK among the leading commentators (Table 22.15). It would surely be helpful to the corresponding charities in other European countries if journalists could call on them regularly (and expeditiously) to comment on the news stories that they were planning to write. This seems to be happening on a small scale in Denmark and the Netherlands, but very little elsewhere.

We also noted the over-citation of the research papers by their authors' fellow countrymen. Some countries, particularly in Scandinavia, had high observed-to-expected ratios of citations. This was associated with a small percentage presence in the world literature of a subject area.

Appendix

Table 22.B17 Organizations that publish CPGs in Europe

Table 22.B17 (continued)

Table 22.B17 (continued)