Introduction

The number-one attraction for students who take atmospheric science classes is weather forecasting (Knox and Ackerman 2005). Besides its popularity with students, active learning through repeated practice is the best way for students to gain experience and improve (Roebber and Bosart 1996). Also, tomorrow’s forecast is not found among the answers in the back of the book! Weather forecasting also encourages critical-thinking skills, involving the highest levels in the cognitive domain of Bloom’s taxonomy (e.g., application, analysis, synthesis, evaluation) (Cervato et al. 2009). In addition, forecasting contests and discussions of the current weather in the classroom can motivate student learning (Harrington et al. 1991; Skeeter 2006), inspire better grades (Hilliker 2008), improve their written communication skills (Market 2006), and result in better forecasts (Market 2006). Finally, weather discussions help close the gap between the knowledge-seeking professors and the goal-seeking students (Roebber 2005) because an otherwise sterile set of equations illustrating complicated physical concepts comes alive when being used to illustrate the current weather in the news or out the window.

Teaching weather in the classroom can also stimulate educational and scientific research. The results from weather forecasting contests have motivated published journal articles on the forecast process (e.g., Sanders 1963, 1967, 1973, 1979, 1986; Bosart 1975, 1983; Gyakum 1986; Bosart and Landin 1994; Hamill and Wilks 1995; Roebber et al. 1996; Bond and Mass 2009; Decker 2012). Weather discussions at the University at Albany deriving from undergraduate and graduate laboratory research courses have motivated student-led published research articles (e.g., Schultz et al. 1997; Bals-Elsholz et al. 2001), and weather education at universities has even led to commercial ventures (Roebber et al. 2010; www.weatherquest.co.uk).

Even for students who are not majoring in meteorology, students in other disciplines who take a meteorology course may develop an appreciation for how the weather affects society and the economy, an understanding of what is happening outside and how it affects their lives, and a practical knowledge about the weather and how it evolves. Yet, whether these students would benefit from similar forecasting exercises as meteorology majors is less well known. Specifically, Hilliker (2008) showed that students in a general education course and geology majors who participated in a forecast contest could improve as they developed more experience, although Seuss (2011) did not find such improvement among students in a survey course for nonmajors. In addition, students’ performance in the contest may (e.g., Hilliker 2008) or may not (e.g., Bond and Mass 2009) be correlated with their academic performance.

An opportunity to test the validity of these ideas in a different context arose in 2010. A new meteorology course (EART 30551) was being developed at the University of Manchester for about 45 third-year earth- and environmental-science students. As there is no degree program specific to meteorology at Manchester, all students had little previous exposure to meteorology before this class. The goals of the course were for the students to see the practical application of the lecture material, to look at the weather and weather forecasts outside of the lecture period, and to engage their critical-thinking skills. The course was designed to have a 20- to 30-min weather discussion presented by the instructor in lecture, coupled with a weather forecasting contest. The weather discussion explored the recent and present weather in the context of the week’s lecture material and then discussed the forecast. With the support of the Faculty eLearning Team, an application was built to collect the student forecasts, score the forecasts, and display the results. This contest was run for two consecutive years: autumn semesters 2010 and 2011. The performance of the students was assessed, and student evaluations were collected at the end of the semesters asking about the utility of the weather discussions and the forecast contest to their learning. The purpose of this article is to describe the contest and discussions and to assess whether students not majoring in meteorology benefit from such activities. In particular, do students gain critical-thinking skills, are students more motivated to learn about the weather, and did the forecasting contest close the gap between the instructor and students?

The Contest Rules and the Weather Discussion

The eLearning Team led the effort to design and build MetCast, a Web site that the students could log into to enter their forecasts, check the results of the previous forecasts, and see their rankings in the contest. The Web site was accessible either directly or through the University of Manchester virtual learning environment Blackboard. The contest was run in two consecutive autumn semesters (2010 and 2011) to 40 and 44 students, respectively.

The rules of the contest were given to the students in the course syllabus. The rules from the 2011 syllabus follow.

The purpose of the forecasting contest is to get you to think about the weather at least once a week in more detail than “Do I need my wellies today?” During weeks 2–11, you will be responsible for making a forecast once a week for two quantities:

  • the maximum temperature (in whole °C) at the official Met Office observing site at Woodford east of Manchester (WMO code = 03348) for the 24-h period starting 0000 UTC Saturday, and

  • your assessment of the likelihood of precipitation at Woodford for the same 24-h period. Your assessment of the probability of precipitation (POP) will be determined by an integer between 0 and 10, where 0 means zero chance of precipitation and 10 means a certain (100 %) likelihood of precipitation.

Participation in the forecast contest is mandatory and is included as part of your evaluation (10 % of the final class grade). Quality of the forecasts will not be part of the evaluation. However, the game will be scored. The best six scores will receive the following bonus points added onto their final course grades (ranked from first to sixth): 6, 5, 4, 3, 2, and 1. (In other words, if you have the best score in the forecast contest and received a 65 in the class assignments, your final score will be increased to 71.) [In 2010, this section read, “The student with the best score in the class will receive an additional 8 marks, the second best score will receive an additional 6 marks, the third best score will receive 4 marks, and the fourth best score will receive 2 marks.”] Any student who beats the score from the average of the class forecasts (called consensus) will receive an additional 2 marks (in addition to the marks above if you are one of the top scores).

Error points will be assigned to individual forecasts as the sum of the temperature error [abs (observed temperature − forecast temperature) × 8] and the precipitation error [(observed POP − forecast POP)2], where the observed POP is 0 if no precipitation is measured, 5 if a trace of precipitation is measured, and 10 if precipitation is measured (0.1 mm or more). The sum of the error points from all ten individual forecasts over the class will be summed to produce the total score. The student with the lowest score will be declared the winner.

If you do not submit a forecast by 2300 UTC, you will receive 100 error points for the week’s forecast. This error will likely be damaging to your ranking, but may still keep you in the game. Furthermore, missing a forecast will affect your mark for the 10 % assessment.

In making your forecast, you may consult any source of information that you wish: your own local knowledge of weather in Manchester, Internet, Met Office, TV or radio weather forecasts, other students, tarot cards, etc. A basic set of URLs will be provided within Blackboard. Enter your forecast from the Web site http://metcast.seaes.manchester.ac.uk, which is also available through Blackboard.

To start, login to the Metcast Web site http://metcast.seaes.manchester.ac.uk with your university account and create a username/pseudonym (does not have to be your real name or your university account name, but must be a clean and appropriate name).

The students were also given these two hints about how to make a good forecast.

  1. 1.

    Because the precipitation error is squared, you should only forecast 0 or 10 if you are absolutely certain it will not or will precipitate. If you are not expecting it to rain, make a forecast of 1 or 2. It is better to take 1 or 4 error points if it does not rain than to take 81 or 64 points if it does rain.

  2. 2.

    Climatology (the average conditions at Manchester or CLIMO) is worth knowing. The average monthly high temperatures and POPs are listed below.

    1. a.

      October: 14 °C, 5

    2. b.

      November: 9 °C, 6

    3. c.

      December: 7 °C, 6

Some changes were made to the contest after 2010. First, only the four best students in 2010 received bonus points added onto their final course grades (8, 6, 4, and 2 bonus points, respectively). These bonus points were changed in 2011 to allow more students (six) to benefit from the contest. Second, the number of error points from missing a forecast was dropped from 150 points in 2010 to 100 points in 2011. This decrease was to keep students who miss one forecast from being knocked too far out of contention. These weights between temperature and precipitation errors are based on the lead author’s experience from past forecast contests.

For comparison, a no-brainer forecast of climatology each week would have yielded 346 points in 2010 and 380 points in 2011. A student forecasting climatology would have ended up in 11th place overall in 2010 and 35th place overall in 2011, neither of which would have placed the student in a position to win bonus marks.

In both years, at the end of each lecture, the instructor gave a 20- to 30-min weather discussion. The discussion was meant to elaborate upon the recent lecture material and to show the practical application of the lecture material to the real atmosphere. The weather discussion involved a presentation of the recent past, present, and future weather. Discussion of the future weather involved providing the Met Office forecast, showing some forecast weather maps, and giving some general guidance to the students in making their forecast for this week’s contest.

Description of the Forecast Contest Application

To register students for the contest, collect the forecasts, calculate the results, and display of the rankings, a Web-based forecast contest application called MetCast was especially developed for the class (Figs. 1, 2). The Web application was written using PHP, and data were stored in a MySQL database. The open source jQuery UI JavaScript library was used to provide user interface enhancements on top of well-formed XHTML and CSS. Much effort was given to the graphic design of the Web application to increase its appeal to students. The software rotated through four seasonal themes designed for MetCast by the Faculty eLearning Team.

Fig. 1
figure 1

Administrator management page of MetCast

Full size image
Fig. 2
figure 2

Competition rankings page of MetCast. Red numbers in the last column are the number of missed forecasts

Full size image

eLearning application is thought to be most effective when learners can associate the object of learning directly rather than trying to understand how to use the system (Constabile et al. 2005). Therefore, the user interface for the students was kept as simple as possible. The Web application authenticated users through the university’s Central Authentication System so that it integrates better with other university systems. In this way, students can access the site via the university’s virtual learning environment more seamlessly. The registering of students for the contest was controlled via release of an entry password as this avoided the additional technical effort in integrating with the university’s student management system.

The Web application was developed to manage multiple competitions at the same time, although in practice only one competition ever ran at a time. Students could enter or edit forecasts up to the deadline. They could also review their past performance by comparing their previous forecasts against the observed readings. A student’s ranking against the other participants in the contest was presented as a simple bar chart, which also displayed the class consensus (Fig. 2).

These efforts in the design and implementation of MetCast appeared to pay off. At the end of 2010, the students were asked to comment on the MetCast application, whether there were technical issues or aspects that were confusing. Only a few students noted minor technical problems, but none were persistent. No serious problems were noted in 2011 either.

Results of the Contests

There were some differences between the forecast contest results from 2010 and 2011. One difference was that the 2011 students outperformed the 2010 students. In 2011, six students beat consensus, which had a score of 128 points. By comparison, only one student in 2010 beat consensus, which had a score of 275 points. (Recall that receiving fewer points is a better score.) Anecdotally, the weather situations in 2010 seemed to have been harder to forecast for than in 2011. The second difference is that students were more likely to take greater risks in 2011 than in 2010 (e.g., only forecasting 0 or 100 % probability of precipitation). The third difference is that fewer students missed forecasts in 2011 than in 2010. Finally, the students in 2011 seemed to have been more engaged with the course than the students in 2010. The reasons for these last three differences are unknown.

The errors in maximum temperature forecasts as a function of week of the contest for 2010 and 2011 are shown in Fig. 3a,b. The temperature errors are the values derived from subtracting the forecasted value minus the observed value. Thus, positive numbers represent overforecasts and negative numbers represent underforecasts (Fig. 3a,b). (Recall that this error is multiplied by 8, and then the absolute value is applied to compute the temperature error points for the contest.) The students tend to underforecast the high temperature during most weeks in both 2010 and 2011, with a standard deviation averaging about 1 °C. There is some indication that, after an initial week or two, the forecasts settle down after the students have a better understanding of how the contest works and what typical errors in weather forecasts are like.

Fig. 3
figure 3

Forecast errors in the contest by week. a Temperature errors (°C) by week in 2010 (observed minus forecast): minimum forecast temperature error made by any student (thin solid blue line), maximum forecast temperature error made by any student (thin solid red line), mean forecast temperature error (thick solid black line), mean forecast temperature error plus and minus the standard deviation (dashed purple lines). b Same as in (a) except for 2011. c Error points in probability of precipitation forecasts by week in 2010: minimum forecast error points received by any student (thin solid blue line), maximum forecast error points received by any student (thin solid red line), mean error points (thick solid black line), standard deviation of error points (dashed purple line)

Full size image

The errors in probability of precipitation forecasts as a function of week for 2010 and 2011 are shown in Fig. 3c,d. On average, the smaller errors in 2011 are indicative of the tendency for students to have taken greater risks (and to have it pay off) in the probability of precipitation forecasts mentioned earlier (Fig. 3c,d). Otherwise, there is little indication that the forecasts showed improvement over the 10 weeks. Instead, the higher errors were associated with more difficult forecasts that did not agree with the majority of the students’ (and the instructor’s) forecasts.

The results from the forecast contest were compared to the scores that the students received in the five tests (Fig. 4). The forecast contest results bore little relationship to the scores that the students received on five tests administered during the semester, with only a correlation coefficient of about −0.2. (Recall that low scores represent good forecasts, so the correlation should be expected to be negative.) Consistent with Hilliker (2008) and Bond and Mass (2009), the students who did not make good forecasts also scored poorly on the tests, although there is quite a spread in the data. The weak relationship between these two performance measures, although somewhat surprising at first, indicates that the forecast contest possibly assesses a different set of skills than the tests. In fact, three of the four students in 2010 who received the bonus marks and three of the six students in 2011 who received the bonus marks were below the class average, demonstrating that the forecast contest allowed students to excel in ways different from the other assessments in the course (four-page essay and five 30-min tests). Another possible explanation is that the best students may not have been motivated for the bonus points, whereas the poorer-performing students were more highly motivated. (One difference between the two years is that any one missed test in 2010 was replaced by the average of the remaining four scores, whereas the students could drop their lowest test score in 2011.) These results differ from those of Bond and Mass (2009), who found a correlation of 0.4 between forecasting performance and test scores among fourth-year meteorology majors (here, high grades in forecasting are associated with good forecasts, hence the positive correlation coefficient). Likely, the 10 forecasts made by the students falls short of the 20–25 forecasts needed for students to reach a minimum standard of skill (e.g., Sanders 1973; Bosart 1975; Gedzelman 1978; Bond and Mass 2009), a result confirmed by the relatively flat trends in errors over the 10 weeks in Fig. 3. Instead, our results are more similar to those of Seuss et al. (2011) showing relatively little improvement after the first few forecasts.

Fig. 4
figure 4

Scatterplot of student scores in forecast contest (x axis, where low scores are good) versus average test score (y axis, where high score is good). The test score was the average of all five tests in 2010, but the average of the best four tests in 2011. Linear regression lines for 2010 and 2011 are plotted, and the corresponding equations and correlation coefficients are shown

Full size image

Evaluation

At the end of 2010 and 2011, the students were asked for their opinions about the contest and its contribution to the course. Changes implemented in 2011 (to be discussed below) improved the ratings that the students gave the discussion and the contest.

In 2010, most students felt that the contest was a good use of their time (56 % agreed or strongly agreed, with another 26 % in between; Fig. 5), although a minority felt that the contest helped them understand the lectures (only 28 % agreed or strongly agreed, with another 38 % in between; Fig. 6). The students were split about whether they felt their forecasts improved over the 10 weeks (55 % felt they improved; not shown), a result roughly consistent with the mean errors in Fig. 3. Most students felt that the weekly discussions of the weather in lecture helped them understand the lecture material (75 % agreed or strongly agreed, with another 18 % in between; Fig. 7), so these student responses indicated a pathway to improve in future years: better integration of the forecast contest, weather discussions, and lecture material.

Fig. 5
figure 5

Students’ answers to the question “The forecast contest was a good use of my time.” There were 28 respondents in 2010 and 36 respondents in 2011

Full size image
Fig. 6
figure 6

Students’ answers to the question “The forecast contest helped me understand the lectures.” There were 28 respondents in 2010 and 36 respondents in 2011

Full size image
Fig. 7
figure 7

Students’ answers to the question “The weather discussions in lecture helped me understand the lectures.” There were 28 respondents in 2010 and 36 respondents in 2011

Full size image

Indeed, in teaching the course in 2011, a greater effort at integration was made. More detailed notes (including annotated graphics) from the forecast discussion were made available online after lecture for students to study from. These actions seemed to be responsible for the improvement of students’ learning. A larger proportion of students felt that the contest was a good use of their time (73 % agreed or strongly agreed, with another 24 % in between; Fig. 5) and felt that the contest helped them understand the lectures (50 % agreed or strongly agreed, with another 25 % in between; Fig. 6). Nearly all students said that the weekly discussions of the weather in lecture helped them understand the lecture material (85 % agreed or strongly agreed, with another 11 % in between; Fig. 7). Although these improvements are heartening, more work remains in integrating the forecast contest with the lecture material and the weather discussions.

How Students Made the Forecasts

Given that the students had no prior experience in weather forecasting, how did they do it? What resources did they use? How did the best students win?

In an online survey at the end of the 2010, most students reported using the BBC, the Met Office Web sites, or both to aid in determining their forecasts. A few other students used other Web sites. Some students even evaluated this information critically in making their own forecasts, as indicated by the quotes below.

  • Sometimes I was prompted to read the more detailed sections on the websites about the weather. It certainly helped my awareness of how accurate/inaccurate forecasting is and made me more interested in the weather.

  • I just used the BBC website and the Met Office website, and if those were vastly different I would use ‘weatheronline.co.uk’, then just decided upon an average each week.

  • I found the map discussions very interesting regarding the forecasts we had just done and for instance why unpredicted events occurred.

  • I mainly used the Met office website. But then I noticed their temperature results were often a couple of degrees out. So then I started looking at other news sources like the BBC. … In the final few weeks I came to the rough conclusion the Met Office would probably predict the temp to be around one or two degrees lower than the actual temperature so then I put the temperature higher taking into account the other sources’ predictions.

An email survey of the winners from both years yielded similar results. Students who did well examined at least two Web sites for forecasts, and two mentioned waiting until the deadline before submitting their forecast to capture any last-minute changes to the weather that may affect their forecast. A couple of students mentioned that, before this course, they followed the weather regularly anyway because of a personal interest in it, including one who was an outdoor enthusiast.

Conclusion

The forecast contest and weather discussion implemented in this course demonstrated that even non-meteorologists could appreciate the fun and the challenge of weather forecasting. The contest also encouraged the students to look at the weather outside of the lecture period, look at weather forecasts from sources outside the classroom (Met Office, BBC, internet sites), and be more conscious of how to obtain this information. The results showed that the best students academically were not the ones who necessarily performed well in the forecast contest, contrary to some previously published results. Specifically, students who may not necessarily be book-smart can be competitive.

Based on the feedback from the 2010 students, the course was altered for 2011. Changes included more students receiving bonus marks, more integration between the lecture material, forecast contest and weather discussion, and annotated notes from the weather discussion made available online. Student evaluations indicated that the forecast contest and weather discussions helped the students grasp concepts from lecture and was a good use of their time.

Thus, our results support the conclusions from the previous literature that forecast contests help develop students’ critical-thinking skills from the highest levels in the cognitive domain of Bloom’s taxonomy. Specifically, students gather weather information from several sources and apply it to the construction of their forecast, students analyze the quality and the source of the weather information in creating their own forecast, students synthesize information from these disparate sources, and they critically evaluate the information they collect in constructing their forecast. The student evaluations showed that the weather discussion and the forecast contest motivated the students to learn about the weather in order to succeed in the contest, although better integration is still needed. Finally, because the answers to tomorrow’s forecast are not in the back of the book, the contest helps turn students into knowledge-seekers rather than goal-seekers, closing the gap between instructors and students.