1 Introduction

In recent reform documents of science education such as Benchmarks for Science Literacy (AAAS 1993), the National Science Education Standards (NRC 1996), A Framework for K-12 Science Education (NRC 2012), the Next Generation Science Standards (Achieve Inc. 2012), the Shape of the Australian Curriculum (ACARA 2009), the New Zealand Curriculum Guides Senior Secondary (Ministry of Education 2012) and the Integrated Science Curriculum and Assessment Guide in Hong Kong (CDC and HKEAA 2007), much emphasis has been placed on aspects of NOS. Despite the inconsistencies in the components of NOS that should be included in the science curriculum, there is substantial agreement among science education organizations, scientists, science educators and researchers that the objective of developing informed conceptions of NOS should be included in the science curriculum (Abd-El-Khalick et al. 1998; Cobern and Loving 2001; Driver et al. 1996; Duschl 1990; Meichtry 1993). Among the five arguments presented by Driver et al. (1996) in explaining why understanding of NOS is an essential component of scientific literacy, the Democratic arguments lent support to the importance of NOS in making sense of socioscientific issues for informed decision making. Being able to read and critically evaluate science reports in the media is important for citizens living in an information-oriented society (Zimmerman et al. 2001). Informed conceptions of NOS are believed to support this important outcome of scientific literacy. By and large, science educators and researchers generally take on their arguments and see them as good reasons for developing students’ NOS understandings. Lederman (2007), though he concurs that these arguments are all important and noble reasons, cautions that they “are primarily intuitive, with little empirical support” (p. 832).

Instead of relying on the technical content of scientific subjects, non-specialists should make informed judgments based on the rubric of history, philosophy, and sociology of science (Fuller 1998). Learning about science is no less important than learning science for critical evaluation of science in the media. Scientific investigation takes place within a social community that can influence the choice of research questions, the interpretation of data and the acceptance of research findings and theory. Judgments about the validity of findings and conclusions largely depend on consensual agreement among researchers, which in turn can be influenced by whether a study has been replicated, and the degree to which results and conclusions fit with existing data and theory. Science is truly a human endeavour, and recognizing the social context of science is a prerequisite to evaluate science news reports. Not only scientists depend on the word of their peers for what they know, but non-scientists also depend even more on scientists for scientific information (Norris 1995). Without any sophisticated science background as that of scientists’, understanding the social context in which science knowledge is generated is particularly useful for nonscientists to evaluate science news reports (Korpan et al. 1997). The increasing importance of media as a source of continued science education creates great demand for “knowledge about the communication of science” (Norris et al. 2003, p. 126), and how well people are familiar with this kind of knowledge will determine how nonscientists “are able to engage critically with scientific information and science professionals” (Ryder 2001, p. 34). While learning science alone is far from enough for critical evaluation of science in the media, NOS understandings are useful to equip non-specialists for evaluating scientific information in media reports. This is supported by Norris (1995) who wrote that:

[…] the proper attitude for nonexperts to have toward scientific experts is not to feel a need to judge the claims that they [the experts] make in their area of expertise, but, rather, to be disposed to judge the grounds for their claim to expertise before willingly consenting to recognize that expertise. (p. 213)

The above implied that it is more reasonable to expect non-specialists to evaluate scientific claims based on “the grounds for their claim”, which require an understanding about science rather than directly on “the claims”, which in turn require a sophisticated understanding of science content knowledge.

There has been dispute over the past decades on whether NOS conceptions would pose any influence on the evaluation of scientific information. Within the research area of educational psychology, students with more sophisticated views about knowledge and knowing were more capable of comparing alternative perspectives and weighing evidence in evaluating arguments rooted in controversial issues (Kuhn and Weinstock 2002). Findings from Mason and Boscolo (2004) suggested that an epistemological understanding would influence the critical interpretation of controversial issues. A logical question would then be: Would a sophisticated understanding about science, i.e. NOS conceptions, also foster critical interpretation of controversial issues, in particular, those with a science component? Within the research area of socioscientific issues (SSI), some studies support that understandings of NOS are of relevance to the examination of scientific information (Kolstø 2001; Ryder 2001; Sadler et al. 2004) while other studies have indicated that NOS was weakly or even not at all associated with decision making regarding SSI (Bell 2003; Bell and Lederman 2003; Zeidler et al. 2002). Kolstø et al. (2006) argued that decision making on SSI could be influenced by various aspects such as moral and political perspectives, and suggested that NOS understandings are of relevance to the examination of the science dimension in SSI. Scarce research has been performed in the context of news articles reporting scientific issues to demonstrate whether NOS understandings are of relevance to the evaluation of science in the media.

2 Theoretical Framework

Given that science is truly a human endeavour, it can hardly be evaluated holistically by relying on science perspectives alone without taking into account the social and other dimensions; in other words, its evaluation requires multiple perspectives. This is supported by Kolstø et al. (2006) who argued that a range of criteria for peer review were emphasized by well known journals and institutes such as Nature, Science, the American Institute of Physics, American Geophysical Union, and Institute of Physics, including technical failings, adequately established main claim, availability of data sets, whether the study was set in the context of previous work, whether all relevant works were cited, and disclosure of affiliations and funding.

Among various NOS aspects, the peer review process, the community of practice, the social and cultural embedded NOS and the tentative NOS were chosen as the target NOS aspects for their relevance with the evaluation of science in the media. The selection was based on the list of components under knowledge of science which were identified as important for critical engagement with science news by twenty-six specialists from science education, media education, science communication and journalism from the UK, Ireland, the USA, and Canada (McClune and Jarman 2010). Science is an enterprise run not only by scientists, but also by members of the society, who reflect on and shape social perspectives and values (Boersema 1998), leading to two types of cultural influences affecting science, namely science-as-society and science-in-society. The target aspects embraced both cultural influences. The peer review process and the community of practice were intrinsic to science-as-society whereas the social and cultural embedded NOS addressed the cultural influence of science-in-society. These, together with the reasons for selecting the four target aspects, are further discussed in the following part. Since both the peer review process and community of practice are closely related to the cultural influence of science-as-society, they are discussed together.

2.1 Peer Review Process and Community of Practice

Science-as-society roots within the science community. It refers to the interactions among scientists, and the culture of the science enterprise itself. Judgments about the validity of findings and conclusions for science knowledge generation largely depend on consensual agreement among science specialists. The science enterprise establishes its own rules of practice such as playing a crucial role in minimizing subjectivity through processes such as peer review, collaboration between science research teams, and debates between scientists in disagreement. These play an important part in the process of science, for instance, the peer review process for determining whether findings are to be accepted for publishing or not, and the community of practice through which rules of practice are established within multiple communities of the science enterprise. As suggested by McClune and Jarman (2010), understanding the methods that science experts use to develop new knowledge and being able to identify procedures such as the peer review process would give reason for people to place confidence in the work by scientists.

2.2 Social and Cultural Embedded NOS

The other type of cultural influence, science-in-society, is beyond the science community. It refers to the interactions between scientists and outsiders of the science community, and relates to the influence of societal factors, such as politics, economics, and religion, which affect the kind of science that is investigated (Lederman et al. 2002), for instance, research proposals written in response to funding agents, and collaborations between scientists at universities and the industry. Science is practised in the context of a larger culture. It is not only affected by, but also affects various elements of the society such as social fabric, power structures, politics, socioeconomic factors, philosophy, and religion (Lederman et al. 2002). Being aware of the interactions between science and society could be useful in order for nonscientists to evaluate science news reports. For example, understanding that science is increasingly becoming commercialized and the implications of how science is funded could be crucial for critical engagement with science in the media (McClune and Jarman 2010). Social and cultural embedded NOS was therefore chosen as one of the target aspects.

2.3 Tentative NOS

Studies have shown that most young people and adults view science knowledge as emerging from observation and experiment, being fixed for all time and of certain status (Driver et al. 1996). Science reported in the media is usually science-in-the-making for its high news value, and is typically tentative as brought about by the advancement in technology and the inferential, creative, imaginative and socio-culturally embedded NOS, even though it is, at least partially, supported by empirical evidence. There is no absolute certainty within frontier science, which may be due to “empirical uncertainty” caused by the lack of data, “pragmatic uncertainty” caused by the lack of means to investigate the problem, and “theoretical uncertainty” caused by the lack of scientific mechanisms for explanation (Driver et al. 2000, p. 300). Uncertainty is a feature of science-in-the-making where the “right answer” is unavailable to everyone including science specialists (Gregory and Millar 1998, p. 243). Being aware of the tentative and contested nature of science-in-the-making was reported as important for critical engagement with science in the news (McClune and Jarman 2010). This coincides with Schommer-Aikins and Hutter (2002) who suggested that one of the thinking dispositions associated with critical thinking about controversial issues was that students should be able to address the evolving nature of knowledge. This conjecture was supported by Scharrer et al. (2012), who found that undergraduates with various majors being less aware of the complexity and tentativeness of scientific knowledge appeared to have more readily formed the impression that perceiving information as comprehensible would qualify them to decide about the persuasiveness of a scientific claim. Findings by Mason (2000) on middle school students suggested that students viewing knowledge as changeable in nature were more readily to accept evidence against their prior beliefs regarding controversial issues. These findings suggested that the conception of tentative NOS can influence the evaluation of science-in-the-making which often appears in the media. Thus it would be meaningful to include tentative NOS as a target aspect in this study to examine its possible association with the evaluation of science news by non-science majors.

2.4 Aims and research questions of the study

Previous studies have consistently reported weak performance of students in the evaluation of science in the media. Studies indicated that both high school students (Norris and Phillips 1994) and university students (Norris et al. 2003) tended to overestimate the degree of certainty expressed in science news reports. Participants in both studies interpreted statements in science-related reports with what the researchers described as “biased towards truth ascription” by attributing to the statements a higher degree of certainty than that expressed by the authors (Norris et al. 2003, p. 125). This could have suggested that science discourse is typically viewed as authoritative, making non-specialists feel unnecessary or unqualified to critically reflect on the issues. In the same studies, it was reported that students had difficulty in distinguishing between roles and status of statements, for example, the explanations of phenomena from the phenomena themselves, and university students performed in almost the same manner as the high school students (Norris and Phillips 1994) despite having received more advanced science education.

These research findings indicated that university students in these studies did not outperform high school students, and Norris et al. (2003, p. 139) sadly concluded that “science education seemed to have very little to do with these important tasks associated with life-long learning of science and democratic citizenship”. It may sound too radical to deny the value of science education in serving these important goals so perhaps it would be fairer to conclude that further science education beyond high school level may have little to do with those important tasks. Little research in this area has been focused on non-science majors at tertiary education. This group is representative of the major consumers of science, who are mostly non-specialists with limited science background. It is unlikely for non-science majors at tertiary education to receive further formal science education after graduation. If they show weak performance in the evaluation of science in the media, there would be little that formal science education could do to prepare them for critical engagement with science in the media.

The overarching aim of this study is to examine the correlation between NOS understandings and multiple perspective evaluation of science in the media by non-science majors. If there is correlation between understandings of NOS and multiple perspective evaluation of science in the media, findings from this study will lend support to the Democratic arguments by Driver et al. (1996). If the result is to the contrary, this study will raise the concern of identifying neglected factors other than, or in addition to, NOS understandings that should be considered for enhancing the ability of evaluating science in the media by non-science majors. This study was guided by the following research questions:

  1. 1.

    How well do non-science majors evaluate science in the media based on multiple perspectives?

  2. 2.

    Is there any correlation between NOS understandings and multiple perspective evaluation of science in the media?

3 Methodology

A mixed methods approach was employed in this study to examine non-science majors’ evaluation of science in the media and their understandings of NOS so as to look for the possible correlation between them. It is “a combination of qualitative and quantitative methods for pragmatic reasons” (Lederman and Lederman 2013, p. 1074) for it can balance the strengths and weaknesses of each other to best answer the research questions in this study. Modes of data collection in the current research study were open-ended questions in form of questionnaires and semi-structured interviews. Open-ended responses allowed the researchers to obtain rich data and to understand how the world is perceived by the participants. By using open-ended questions, respondents were given the opportunity to answer the questions from their own frame rather than being forced to choose from a fixed response predetermined by the researchers. Follow-up interviews were carried out in a semi-structured format, which served three purposes. First, written responses made by respondents might be limited by their writing skills. Interviews served as a way of ensuring that participants’ written responses were aligned with their NOS understandings and their views about the scientific claims reported in the news articles. Second, it helped clarify language use to ensure that the written responses were not misinterpreted. Third, it helped seek for more examples and probe deeper understanding of participants’ conceptions of NOS.

In order to capture the relationship between evaluation of science in the media by non-science majors and their understandings of NOS, NOS instruction was part of the course structure in this study in order to develop a wide range of NOS understandings among participants. A pre- and post-test was not adopted in this study since it would be hard to ensure that differences, if any, identified between the pre and post tests were attributed to the NOS instruction in the course or other factors at play.

3.1 Participants

Thirty-eight students enrolled in a general education course named Food and the Body held by a local community college participated in the study. All of them were full-time students either in their first year or second year of studies pursuing an Associate Degree in Arts, Business Administration, Legal Studies or Social Science for acquiring broad-based general education in preparation for employment or further studies. They were enrolled in the course to satisfy the graduation requirement in general education for science. The course had no science prerequisites. Students enrolled in the course were non-science majors, and had diverse backgrounds in science ranging from having taken science classes up till Form 3 (9th grade) to Form 7 (13th grade). They were chosen for the reason that they were representative of the public who generally possess limited science background while a reasonable literacy level could have been assumed. NOS instruction was part of the course structure, and the students were actively taught how to apply this knowledge to the evaluation of news reports in general. This could help develop a widespread of NOS understandings among participants for capturing the relationship between NOS understandings and the evaluation of science in the media.

3.2 Data Collection

Data from different sources were used to address the questions of interest in this study. These included (1) the Health News Evaluation Questionnaire, (2) the Views about Science Questionnaire, and (3) a follow-up interview. The Health News Evaluation Questionnaire contained three science news articles for participants to evaluate, and the Views about Science Questionnaire was composed of open-ended questions for getting insight into participants’ understandings of NOS (“Appendix”). The follow-up interview was conducted after questionnaire administration for participants to clarify and further elaborate their NOS understandings and their evaluation of science news articles.

3.2.1 Health News Evaluation Questionnaire

Three science news articles from the area of health were selected for evaluation by participants. News articles were screened and selected according to the following selection criteria. First, the news articles should carry health-related claims, e.g. those about food and diet. According to Wellington (1993), food and diet are among the most frequent topics covered by the press. It is those scientific claims of high news values that are more likely to be reported by the media. Timeliness, relevance, conflict/controversy and impact are among the criteria for determining news value. In other words, it is those controversial scientific claims from science-in-the-making potentially affecting a lot of people that are more likely to be reported in the media. News reporting health claims other than food and diet, e.g. the impact of mobile phones on health, also readily fulfilled these requirements for high news value. Two of the three news articles selected for this study reported claims about food and diet while the other one reported a health-related claim. Using these articles also tied in well with the subject nature of the course that the participants attended. This promoted a sense of relevancy to the course, and helped achieve a better response rate and obtain richer data from participants.

Second, as this research study aimed at exploring the relationship between understandings of NOS and the evaluation of science in the media, the three selected news articles should provide opportunities for participants to evaluate by including the target aspects of NOS.

Third, it was necessary to limit the length of news articles while providing a reasonable coverage of content and opportunities to evaluate by making reference to the target NOS aspects, so that participants would not be overwhelmed by the lengthy news articles and meaningful data could be obtained. Thus the number of words in the selected news articles was limited to around 300–700 words.

Lastly, science news are not only limited to those which are purely science in nature, but also those of socioscientific nature. Socioscientific issues are often uncertain and complex, with connections to science content, and of social significance (Eastwood et al. 2012). The chosen articles covered both contexts for a fuller account of the evaluation of science in the media by non-science majors. Text-based articles, instead of television or other forms of news, were chosen for the reason that the incorporation of visual forms such as video clips, illustrations or graphical representations might impose unintended thoughts about the news to readers.

Here are the descriptions of the three news articles:

Article 1: The Tofu Claim The first article from Telegraph Media Group reported that tofu might harm memory in elderly people (Khan 2008). This was a study carried out by researchers at Loughborough and Oxford universities with Indonesian colleagues to assess the effects of high soy consumption among elderly Indonesians living in urban and rural areas of Java. No causal linkage was identified to support the claim. Professor Eef Hogvervorst, who led the research, admitted that the effect of phytoestrogens in soy products was not entirely clear.

Article 2: The Calorie Claim The second article from Cable News Network (CNN) reported findings about older adults who cut down the amount of calories they consumed and as a result got the benefits of weight loss and better memory (Harding 2009). The research was carried out by Dr. Agnes Floel and her colleagues from the University of Munster in Germany. Independent researchers from relevant fields all supported and agreed with the reported findings. The figures were commented as statistically significant, and the article ended with two proposed theories on how caloric restriction might slow aging. A high consensual agreement was reached within the science community on this claim.

Article 3: The Cell Phone Claim Unlike the first two articles, Article 3 reported a socioscientific issue. This article was reported by Canadian Broadcasting Corporation (CBC) News (The Associated Press 2008) regarding a warning of limiting cell phone use because of the possible risk of cancer by Dr. Ronald B. Herberman, the director of the University of Pittsburgh Cancer Institute. His warning was based on early unpublished data, and was supported by a former health adviser in the US federal government as well as authorities in England, France and India. However, it was opposed by an analysis by the University of Utah in 2008, the Food and Drug Administration (FDA), a massive ongoing research project named Interphone, and Joshua E. Muscat of Penn State University.

At the end of each article in the questionnaire, participants were asked to rate the extent to which they agreed with each of the reported claims from the choices of ‘strongly disagree’, ‘disagree’, ‘tend to disagree’, ‘tend to agree’, ‘agree’ and ‘strongly agree’. They were then asked to justify their choice by providing as many reasons as possible. The three news articles were validated by two science education specialists to ensure that they would potentially elicit the use of the target NOS aspects. Follow-up interviews were conducted for participants to clarify and further elaborate their viewpoints.

3.2.2 Views About Science Questionnaire

Question items in the Views about Science Questionnaire were mainly extracted from Views of Nature of Science-Form C (VNOS-C) (Lederman et al. 2002) and Views of Scientific Inquiry (VOSI) (Schwartz et al. 2008) questionnaires. It takes as long as one and a half hours for teacher respondents to just complete the VNOS-C (Lederman 2007), and can be overwhelming if participants are required to complete both the VNOS-C and the VOSI questionnaires. Specific items targeting particular NOS aspects were selected from each of the questionnaires in order to gain sufficient data without overwhelming the participants. This also encouraged deeper and richer responses. Question items addressing the target aspects, social and cultural embedded NOS and tentative NOS from VNOS-C and community of practice from VOSI, were extracted. As there was no question item available from these two well-established questionnaires for probing participants’ understanding of the peer review process, a separate question item was developed for such purpose (as below).

  1. (a)

    Do you know about the peer review process in the publication of scientific findings? Please provide a general outline of the peer review process.

  2. (b)

    Please comment on whether the peer review process is objective or not. Give examples to support your view.

In each question item, participants were asked to give examples for justification and further elaboration for revealing the depth of their NOS understandings. Follow-up interviews were conducted to validate the interpretations of responses from the current pool of participants, and to establish the validity of the newly added question item.

3.2.3 Administration of Questionnaire

The Health News Evaluation Questionnaire was administered in Week 2 of the semester after the implementation of decontextualized NOS teaching. Participants were given 1 week for completing the Health News Evaluation Questionnaire. The questionnaire was distributed in class for participants to take home and finish at their convenience so that as much time as needed was given due to the open-ended nature of the questionnaire. Giving participants ample time for questionnaire completion also encouraged participants to answer the questions in a detailed manner. This also mimicked what happens in everyday life that readers may or may not take the initiative to look up for further information in trying to know more about a science news article. The course content did not touch on the topics reported by the selected news articles in the Health News Evaluation Questionnaire. Given that the participants were non-science majors having taken science classes up till Form 3 (9th grade) to Form 7 (13th grade), it was reasonable to assume that their science background, in terms of the technical dimension of the three chosen articles, would be representative of the general understanding of science by the public.

After the Health News Evaluation Questionnaire was collected in Week 3, participants were given another week to complete the Views about Science Questionnaire and return it in Week 4. The Health News Evaluation Questionnaire and the Views about Science Questionnaire were administered separately for minimizing the effect of referring to NOS understandings for the evaluation of the science news articles. After administering the questionnaires, time was allowed for the analysis of responses. Volunteers were invited for follow-up interview in Week 5 and the interviews were conducted from Week 6 to Week 8 depending on the availability of the volunteers. Eighteen out of 38 respondents agreed to participate in the follow-up interviews.

3.2.4 Follow-Up Interviews

Upon the return of questionnaires from participants, preliminary data analysis was conducted. Volunteers were then invited for interview to ensure the validity of item interpretation and responses, and to further probe their views about science and their evaluation of the news articles. During semi-structured interviews, participants were asked to explain their responses to the questionnaire. During the interview, participants were shown or reminded of their written responses in the questionnaire. Then they were asked if their beliefs were different from what was represented in their written responses. The researcher asked the interviewees to further elaborate on their answers and provide examples so as to obtain a better understanding of their NOS conceptions and their evaluation of the selected news articles. Each interview lasted for approximately 20 min and was audio-taped, transcribed verbatim and translated. Through the follow-up interview, ambiguities were clarified and any misinterpretation of the responses to the question items was identified so as to faithfully represent respondents’ views. The follow-up interviews were useful for clarifying the viewpoints of participants and triangulating data for confirming findings.

3.3 Data Analysis

Upon receiving the questionnaires from participants, their written responses were read carefully. The Views about Science Questionnaire and the Health News Evaluation Questionnaire were analyzed separately before being put together to examine the possible relationship therein. Data were coded for the stance adopted in evaluating science news articles and participants’ understandings of the peer review process, community of practice, tentative NOS, and social and cultural embedded NOS to check for patterns across participants and individual units of data.

3.3.1 Stance Adopted in the Evaluation of Science in the Media

The term stance was borrowed from Phillips and Norris (1999) in describing the interaction between the reader’s world and the world as it is represented in popular reports of science in the media. Stance in the current study refers to the ability of readers to interact with science in the media with multiple perspectives, which include but are not limited to scientific and social dimensions. The written responses of justification put forth by participants in each of the three science news articles were analyzed and categorized into various levels of stance by making reference to rubrics generated by the argumentation quality rubrics (Sadler and Fowler 2006) and the rubrics of socioscientific reasoning (Sadler et al. 2007). The argumentation quality rubrics were developed based on Toulmin’s Argument Pattern (TAP) (Toulmin 1958). The rubrics focused on analyzing the justification of claims rather than categorizing argument structures into claims, data, warrants, backings and rebuttals. The argumentation quality rubrics were chosen as they could minimize the problems encountered due to the ambiguous nature of argument structures identified by TAP (Erduran et al. 2004), and identifying the argument structures was not the focus of this study. Grounds, in the words of Sadler and Fowler (2006), “corresponded to a variety of possible supports or details regarding a justification. Statements that would have been classified as data, warrants, or backings in Toulmin’s scheme were considered grounds” (p. 993).

Level 1, the lowest level of justification, captured those cases with justification limited to simple grounds. Responses at this level were either vague or with just a superficial justification of the viewpoint. Some were a more or less direct citation or paraphrasing from news articles, as referred to as “text-based” position by Phillips and Norris (1999). Level 2 and Level 3 required justifications based on elaborated grounds, which should include data, warrants and backings. Neither simplifying an issue by focusing on one factor nor solely considering an issue by simple cause and effect was preferred (Sadler et al. 2007). Responses at Level 2 were confined to evaluation by a single perspective only whereas those at Level 3 differed in that they required the consideration of a reported claim through multiple perspectives.

Perspectives adopted by participants included but were not limited to methodology, sampling, proposed mechanism, consistency between proposed claim and evidence, scientific knowledge/personal understanding, other variables, compatibility with general phenomenon, consideration of counter position, existence of vested interest etc. Perspectives differed from grounds in that it was possible to develop more than one ground based on the same perspective. For instance, by looking into an issue through the perspective of methodology, justifications could be made on grounds such as research design, lack of methodological details or use of experimental and control groups. From the perspective of proposed mechanism, justifications could be built based on grounds in terms of whether the proposed mechanism was logical or how well it fitted into existing belief. Analyzing based on multiple perspectives was desirable and was regarded as more advanced (Sadler et al. 2007). This corroborates with Kolstø et al. (2006) that the complexity involved often made it impossible to encompass the whole issue from a single perspective.

Viewing scientific issues as subject to ongoing inquiry and being skeptical towards potentially biased information were desirable for advanced practices in decision making (Sadler et al. 2007). Instead of taking the issues as absolute truth, readers should remain open-minded and be prepared for forthcoming changes upon the emergence of new evidence. Informed decisions based on the best of knowledge today might be changed in the future. Advanced evaluation of science in the media should consider the possibility of uncertainties and urge for more information as appropriate. Apart from exhibiting skepticism, considering the social context where scientific knowledge was generated was also essential for the critical evaluation of science in the media. Due to their limited science background, it would be far from the reach of the lay public to evaluate science in the media solely from a science dimension. Even peer reviewers do not only evaluate findings from a science dimension, but also from a social dimension such as the disclosure of affiliations and funding (Kolstø et al. 2006). The public might rely even more on the social dimension due to their limited science background. Readers should recognize that both the dynamics of science as an enterprise and the interactions between science and society could influence the advancement of science.

Evaluating a scientific claim through scientific and social dimensions and being aware that it is subject to ongoing inquiry were necessary for a more advanced evaluation of science in the media. Responses indicating elaborated grounds from multiple perspectives encompassing both scientific and social dimensions coupled with addressing the uncertainties associated with a scientific claim would be categorized as Level 5. Failure in encompassing scientific and social dimensions or addressing the characteristic of ongoing inquiry in scientific claims would be rated as Level 4. The classification of stance into 5 levels is summarized in Table 1.

Table 1 Rubrics of stance categorization
Full size table

3.3.2 Views About Science

Results from the Views about Science Questionnaire and follow-up interviews were used to generate four categories to describe participants’ understandings of each of the target NOS aspects, according to McDonald (2010) and Schwartz et al. (2008). Responses were categorized based on a scale of one to four. A score of “1” was assigned to responses with strong evidence of naïve understandings, a “2” was assigned to responses indicating limited understandings, a “3” for responses showing partially informed understandings, and a “4” for responses demonstrating informed understandings where no contradictory answer was identified in both written responses and follow-up interviews. Should there be any inconsistencies between participants’ views about science and their written responses on the questionnaire, they were to be identified during follow-up interviews, and priority would be given to interview data (Lederman et al. 2002).

The Views about Science Questionnaire and the Health News Evaluation Questionnaire were analyzed separately before being brought together for the analysis of correlation with the use of SPSS software 16.0 (SPSS Inc., Chicago, IL, USA). This helped provide answers to the possible correlation between NOS understandings and the evaluation of science in the media among non-science majors.

4 Results and Discussion

In this section we present the results of the assessment of participants’ views of the peer review process, community of practice, tentative NOS, and social and cultural embedded NOS using the Views about Science Questionnaire. This is then followed by the results of the stance adopted by participants in the evaluation of each news article. Lastly, the relationship between NOS understandings and the stance adopted in the evaluation of science news articles is discussed.

4.1 Views About Science

Data collected from the questionnaires were confirmed and triangulated with the follow-up interview transcripts. No discrepancy was identified between written responses and interview transcripts, except on one scenario regarding the item assessing the understanding of the tentative NOS. Student 23 did not explicitly point out in his written response that theories could be changed due to technological development or re-interpretation of old evidence, but his verbal response (as below) still suggested an informed view by stating that people from different centuries would hold different beliefs:

Yes, of course, since in the scientific process many scientists will check and investigate a theory after it has been published, and people living in different centuries and years have different attitudes and beliefs, so scientists will overthrow or change some theories.

The informed view of Student 23 was confirmed in the follow-up interview where he further elaborated his response and suggested technological advancement as one of the reasons for the changes in theories (excerpt as below). The use of the follow-up interviews as above was useful for confirming findings by providing a platform for participants to further elaborate their written responses.

I think it will change. The attitude to theories may also change, for example, in the time of Galileo, the Church disapproved what he proposed. But then now it has been proved correct. Belief regarded as false in the old times may actually be proved right in the modern world with the advancement in technology.

Of the four target aspects, the peer review process and the community of practice were not covered by VNOS, and the rubrics for categorizing participants’ understandings of these two aspects were not available, thus this requires further discussion.

4.1.1 Understanding of the Peer Review Process

Understanding of the peer review process is rooted at the social dimension of science, in particular, the culture of science-as-society. According to the analysis by McComas (1998) on the consensual view of NOS objectives extracted from eight international science standards documents, it has been agreed upon that the peer review process is among one of the important NOS elements. An informed view of the peer review process requires one to understand that it is an evaluation process in which experts critique the work of authors or groups seeking recognition and publication, and is never straightforward. This process serves as a gatekeeper to the publication of new findings. While it provides a comparatively fair judgment by taking measures aiming at objectivity, sometimes social or personal values may influence judgment as suggested by Student 26 in her written response that “The peer review process may not be very objective. It can be affected by the background knowledge and personal beliefs of scientists…” In other words, the peer review process is infused with subjectivity to a certain extent.

Participants possessing a partially informed view of the peer review process failed to provide a detailed account of the process. They agreed that both objectivity and subjectivity were involved, but failed to substantiate their arguments with valid explanations. For instance, Student 31 stated in the questionnaire that “… the process is objective, since the whole process has to pass through many scientists and have lots of checking. However, there may be some biases in the process, and [so this can be] not really objective at all.” Student 31 noted the objectivity and subjectivity involved in the process, and pointed out that bias could make the process subjective. She could have substantiated the argument by listing possible factors accounting for the objectivity (e.g. the quality of data set and the validity of empirical evidence), and possible reasons for causing the bias therein. Since Student 31 failed to put forward a comprehensive explanation for this, she was categorized as possessing a partially informed view. Participants classified as having a limited view failed to describe the process properly, or suggested that the process was either completely subjective or objective. As shown in the written excerpt from Student 34, “Firstly, peer reviewers would discuss together to check if research findings are new, authentic, without plagiarism involved. If the above is satisfied, findings can then be announced.” The student held the misconception that peer reviewers would discuss the findings together, and insofar as the findings were new, authentic and without committing plagiarism, they would be accepted for publishing. In reality, the peer reviewers rarely discuss the submitted findings together. Participants with a naïve view, failed to give any descriptions of the process, leaving the answer sheet blank or giving “I don’t know” answers.

4.1.2 Understanding of the Community of Practice

Similar to the peer review process, community of practice is also an aspect rooted at the social dimension of science. It specifically focuses on how scientists work together and on its possible impact on research findings. Possessing an informed view requires one to understand the social dynamics within the science community as manifested in being aware that scientists working together may still get different conclusions, as well as that consensus may be reached through discussion no matter whether the same procedures are followed or not. Consensus will not necessarily be reached, but is likely due to the common goal shared by members of the same research team (Schwartz et al. 2008). Student 37, an informed participant of the community of practice, was aware of the social interactions among scientists working together, making consensus more likely to be reached as indicated in the written excerpt “The reason is that, the scientists working together would share their observation and viewpoint on the question, and finally they would reach a conclusion to the questions.” Participants categorized as having a partially informed view were able to point out that it was the interactions and discussions among scientists working together that could make consensus more likely to be reached. However, they held the belief that if different procedures were applied, consensus would no longer be more likely to be reached even by working together as demonstrated in the excerpt by Student 35 that “… as they use different procedures to collect data, so the problem with reaching consensus may exist.” Participants having a limited view noted that it would be more likely for scientists to reach consensus by working together for reasons other than the social interactions and common goals shared by scientists, e.g. through the use of the same methodology as indicated by the following excerpt from Student 38:

If the scientists are working together using the same procedures, then it is more likely for them to obtain the same conclusion. It is because it is easier to control the experiment, and minimize the number of variables that they may come across.

Finally, participants with a naïve view held the belief that working together or not would not pose any influence on whether consensus was more likely to be reached, in other words, the social interactions among scientists were ignored.

4.1.3 Overview of participants views of NOS

More than 50 % of the participants achieved at least a partially informed view for all the target NOS aspects as shown in Table 2. Participants performed best with regards to the tentative NOS aspect, with 68.4 % of them possessing an informed view. This was followed by the peer review process aspect, with 50 % of the participants possessing an informed view. For both the social and cultural embedded NOS and community of practice 31.6 % of the participants were found to hold an informed view. However, 34.2 % of the participants possessed a partially informed view with regards to the social and cultural embedded NOS, compared to 21.1 % for the community of practice, suggesting that participants had the weakest understanding of the community of practice, and they were not quite aware of the social dynamics among scientists working together. Overall speaking, a wide range of NOS understandings for each target aspect was identified among participants, which was important for capturing the relationship, if any, between NOS understandings and the evaluation of science in the media.

Table 2 Views about science held by participants
Full size table

4.2 Stance Adopted in Evaluation of Science News Articles

As shown in Table 3, it was encouraging that at least 42.1 % of the participants were able to achieve a stance of Level 4 or above in the evaluation of each of the three science news articles, suggesting that most of the non-science majors were able to evaluate the science news articles from multiple perspectives encompassing both scientific and social dimensions or addressing the uncertainty involved. Below is a representative excerpt by Student 20, ranked as Level 5 in the evaluation of Article 1:

The reported claim is well supported, since it is a “to be published” journal article with a large sample size. Since it is ready to be published, it means that the study has gone through several steps… In addition, the large sample size increases validity as well. Nevertheless, the statement does not truly explain how soy products “worsen memory”, it only explains that there are damages to the functions of the brain.

There is one setback, however, to the trustworthiness of the article. It is the statement by Professor Eef Hogvervost of Loughborough University; he stated that “it is not entirely clear what their effect on the ageing brain is”. Thus, even to an expert, there isn’t a sure guarantee that phytoestrogens (from soy products) brings damage to the brain…

The credibility that Student 20 gave to the large sample size in the reported research and the criticism he raised regarding the inadequacy of the explanation for the reported phenomenon were factors rooted in the social dimension of science. This refers to issues related to social aspects such as considering whether findings had been published or not, and the professional recognition of the scientists involved and the universities they work for. Student 20 also discussed the uncertainty associated with the reported claim as admitted by the chief investigator. His evaluation of the claim through multiple perspectives transcending science and social dimensions coupled with the consideration of uncertainty associated with the reported claim placed his evaluation at a stance of Level 5.

Table 3 Stance adopted in the evaluation of each of the science news articles
Full size table

The evaluation of Article 1 by Student 21 was categorized as a stance of Level 4.

… the study funded by the Alzheimer’s Research Trust only tells us the conclusion but did not mention anything about the underlying mechanism. That is, the reason for which oestrogen damages brain functioning is missing… the study only focuses on one group of people, the Javanese people. This is again not persuasive enough as the researchers do not provide any explanation why only Javanese people are subject to test… the conclusion that concentrates on only one particular ethnic group is too narrow to get a generalized or an accurate result… even though the article has been published in the journal… it does not necessarily imply that the article is totally objective… social or personal values may affect scientific judgment in the peer review process.

Student 21 questioned the lack of reference to an underlying mechanism and the generalizability of findings by selecting Java people as the only subjects. These were factors associated with the social dimension of science. Her response also encompassed the social dimension by discussing the possible impact of peer review process on the trustworthiness of the reported findings. Given that her response did not address the characteristic of ongoing inquiry in the reported claim, her evaluation of Article 1 would place her at a stance of Level 4.

Student 22 was categorized as having a stance of Level 3 by building her arguments based on aging as a confounding variable, i.e. taking the view that aging might also be responsible for memory loss among the elderly but was not reported in the news article as a factor being considered in the research, and the lack of explanation for the reported claim in Article 1 (excerpt as below). These were convincing, but were limited to social dimension of science only.

… The article proposes that eating tofu would cause lower level of memory functioning of the elderly. However, the article does not show any evidence that there is any relationship between the intake of tofu and the reduced level of memory functioning as the elderly tend to have a declining level of memory functioning. The elderly, especially those in sixties, tend to suffer from loss of memory more than other age groups. This is a common phenomenon… The article does not elaborate why the intake of soy products would cause the lowering level of memory functioning…

Student 30 was categorized as having a stance of Level 2 in her evaluation of Article 2. She kept justifying her viewpoint based on the argument of the effect of calorie cutting on brain without considering other factors in concern. Given that her argument was well elaborated, the evaluation of Article 2 would place her at a stance of Level 2 (excerpt as below).

As we know that cutting calories would make the body more sensitive to the blood sugar-regulating hormone, insulin, and would result in a drop in the C-reactive protein… so that [would] increase the flow of blood sugar to our brain, making the brain more active… the article said that reducing calorie intake keeps cells under constantly low level of stress… when the reducing calorie intake keeps cells under a constant low level of stress, they become easy to cope with… and can prove the brain’s memory increase.

Student 34 justified her viewpoint in the evaluation of Article 1 based on the small proportion of subjects having their memory influenced by tofu consumption. However, she failed to further explain its influence on the trustworthiness of the reported claim. She did make an attempt to present another justification based on trypsin inhibitor, but failed to substantiate her argument, making it unconvincing. Her evaluation was limited to simple grounds, and was classified as a stance of Level 1.

… it was only found that those who eat tofu more than once a day had a 20 % lower level of memory functioning than those eating very little of the product. I think this reflects just some of the people suffering from Alzheimer’s. I know that Tofu maintain trypsin inhibitor, and it will obstruct digestion and absorption of protein.

Among the three news articles, participants performed least satisfactorily in the evaluation of Article 2, and best in that of Article 3. Only 5.3 % of the participants reached a stance of Level 5 in the evaluation of the calorie claim in Article 2, compared to 23.7 % in Article 3 (see Table 3). This could be attributed to the consensual agreement among scientists which led to a higher level of trustworthiness identified by the participants in Article 2, making them less inclined to raise concerns about the uncertainties associated with the reported claim. Participants performed best in the evaluation of the cell phone claim in Article 3, which was of socioscientific nature. Findings suggested that SSI more readily aroused awareness of the uncertainty associated with the reported claim, and elicited more responses in making reference to the social dimension of science in the evaluation. This could be due to the complexity of SSI, which makes it difficult to address the issue through a single perspective (Kolstø et al. 2006), and thus typically initiates more than just scientific perspective for its evaluation. Together with a wide range of opinions from the science community, government officials and authorities reported in Article 3, the importance of evaluating through a social perspective was made prominent.

4.3 Correlation Between NOS Understandings and the Stance Adopted in the Evaluation of Science News Articles

The correlation between the target NOS understandings and the stance adopted in the Health News Evaluation Questionnaire is shown in Table 4. The Pearson’s correlation analysis showed that the level of understanding of the tentative NOS was significantly correlated with the stance adopted in the evaluation of the cell phone claim reported by Article 3 (r = 0.434, p < 0.05). This significant correlation suggested a moderate association between informed understanding of the tentative NOS and a more advanced stance by considering a reported issue through multiple perspectives. This echoes previous findings (Liu et al. 2011; Schommer-Aikins and Hutter 2002) that participants holding the belief that knowledge was complex and tentative were more likely to take multiple perspectives when making decisions about controversial issues. Although no similar correlation was observed in the other two articles, almost all participants whose responses were classified as Level 5 in the evaluation of these two articles also possessed an informed understanding of the tentative NOS, with the exception of Student 16. Even if the student was classified as adopting a sophisticated stance in her evaluation of Article 1, the way in which she pointed out the uncertainty associated with the claim remained in a more or less a direct copy from the text as indicated in the excerpt, “According to research done by Loughborough University, it is not entirely clear what effect soy products have on the brain.” This observation was consistent with the moderate correlation identified in Article 3, suggesting that a better understanding of the tentative NOS potentially enhanced the evaluation of science in the media. The cell phone claim in Article 3 was based on contested scientific findings. Its socioscientific nature could have made the claim more complex. It was in this kind of context that an informed understanding of the tentative NOS would be particularly useful for enhancing the evaluation of science in the media. No other correlations were identified between NOS understandings and the stance adopted in the evaluation of the science news articles, indicating that other target NOS aspects had no significant impact on the stance adopted in the evaluation of science news reports.

Table 4 Table showing the correlation relationship between the target NOS understandings and the stance adopted in the Health News Evaluation Questionnaire
Full size table

5 Conclusion

It was encouraging that more than half of the participants were able to achieve a stance of Level 4 or above in the evaluation of the three science news articles, showing that non-science majors were able to evaluate science news articles from multiple perspectives encompassing both scientific and social dimensions or addressing the uncertainty involved. In terms of multiple perspective evaluation, participants tended to perform better in evaluating articles of socioscientific nature, and less well in articles being rated as more trustworthy. This could be due to the complexity typically associated with SSI, making it unlikely to encompass the issue through a single perspective (Kolstø et al. 2006). Thus it typically initiated evaluation beyond the scope of the scientific perspective, in other words, science news of socioscientific nature more readily aroused a sense of awareness to the uncertainty associated with the reported claims, and elicited more responses in making reference to the social dimensions in the evaluation.

Another finding in this study was the correlation between the understanding of tentative NOS and the stance adopted in the evaluation of SSI (r = 0.434, p < 0.05). This moderate correlation suggested the possibility of informed understanding of the tentative NOS in making participants more likely to evaluate science news articles through multiple perspectives, from both scientific and social dimensions and recognizing the uncertainty involved in a claim. This correlation was only observed in Article 3, which was based on contested scientific findings of socioscientific nature. It was particularly prominent in news articles of such nature that informed understanding of the tentative NOS was associated with multiple perspective evaluation of science in the media. This was possibly a result of informed understanding of the tentative NOS that raised the awareness towards the effect of technological advancement and/or the creativity and imagination embedded in scientific investigation. This mirrored previous findings by Liu et al. (2011) that non-science majors holding beliefs about scientific knowledge being tentative were more likely to recognize the complexity and take multiple perspectives in the decision-making process. Similar results were reported by a study about decision making on the controversial SSI—genetically modified food by ninth-grade students (Khishfe 2012). The study reported that participants were more able to relate their discussion to the tentative NOS after receiving explicit NOS instruction, suggesting an association between understanding of the tentative NOS and the interaction with SSI. The relationship between understanding of the tentative NOS and the evaluation of science news reports was limited to the article of socioscientific nature, and was not observed in the other two articles. The complex and controversial nature could have made the uncertainty associated with SSI become more prominent. For other target NOS aspects, no correlations were identified with the stance adopted in the evaluation of science in the media. Despite this result, it cannot be concluded that understanding the social dimensions of science does not pose any influence on the evaluation of science in the media, without looking into whether participants did not consider this at all or considered this but concluded this was irrelevant or not helpful in supporting their justification. Future work on identifying the reasons for why and why not NOS understandings are applied in the evaluation will move this field forward.

This study only reflected the evaluation of science news by non-science majors in the area of health-related issues which were chosen for the reason of readily scoring high at news value. It cannot be concluded with certainty whether the yielded findings are only applicable to health-related news articles or can be generalized to other science news. Findings in the current study were only limited to text-based articles, whereas science news also comes in visual forms such as television news and articles with the incorporation of images and graphical representations. Similar to text-based articles, these visual forms may also mislead readers or intend to manipulate their thoughts (Jarman et al. 2012), making readers fall prey to media’s agenda. For example, the use of graphics for explanation may over-simplify complicated scientific ideas (Winn 1989), making readers overestimate its certainty. Using open-ended questionnaires and interviews to explore the correlation between NOS understandings and evaluation of science in the media has its merits as discussed earlier but it comes with the limitation of transforming qualitative data into quantitative. This limitation may also be extended to similar studies. Moreover, there are aspects other than the stance adopted for studying how science in the media is evaluated. Future work to explore the relationship between NOS conceptions and the evaluation of science in the media (1) reporting findings from other domains of science and (2) of formats other than the text-based ones as well as (3) focusing on ways other than stance adopted to study the evaluation of science in the media, would be useful for a better understanding of the relationship between the two.