Examining the Use of Audience Response Systems in Secondary School Classrooms: A Formative Analysis

Overview

An Audience Response System (ARS), also known as a personal response system (Hinde and Hunt 2006), electronic voting system (Kennedy and Cutts 2005), student response system (Kaleta and Joosten 2007 and clickers (Bergtrom 2006), permits students to click in answers to electronically displayed multiple choice questions using a remote control device. After students select an answer, the results are instantly aggregated and displayed in chart form for the entire class to review. Responses are usually anonymous, but a teacher can assign individual remote devices to specific students for purpose of evaluation. Once the feedback is presented, a teacher can alter the course of instruction or students can work out misconceptions and difficulties through peer or classroom discussion. Widespread use of ARSs in secondary schools is a relatively new phenomenon beginning in 2003.

Several researchers have completed extensive reviews on the use of ARSs (Caldwell 2007; Fies and Marshall 2006; Judson and Sawada 2002), however only one paper could be found examining the use of ARSs in secondary school classrooms (Penuel et al. 2006). The purpose of the current study was to conduct a detailed formative analysis of the potential benefits, challenges, and use of ARSs from the perspective of secondary science students.

ARSs in Secondary Schools

Research on the use of ARSs in secondary schools is sparse. Penuel et al. (2006) have conducted the only study of ARSs in a K-12 environment. They noted that teachers used ARSs for two principal reasons: formative assessment (to improve learning and instruction) or summative assessment (to deliver and grade tests). They also observed that training increased frequency of use and that teachers rarely used ARSs to promote discussion. Teachers who adopted the view that students should play a significant, active role in learning were more likely to use ARSs for formative assessment. Overall, the main impact of ARSs in K-12 was to increase motivation and improve student learning.

Benefits of Using ARSs in Higher Education

Because there is limited research on the use of ARSs in the secondary school domain, it is worth reviewing the impact of ARS in higher education, particularly since the majority of ARS use has occurred in science related subject areas (e.g., Allen and Tanner 2005; Beatty et al. 2006; Dufresne and Gerace 2004; Fagan et al. 2002; Preszler et al. 2007; Sharma et al. 2005). A detailed review of the ARS literature in higher education revealed four key areas of focus: overall attitudes toward ARSs, student involvement, assessment, and learning. Each of these areas will be discussed in detail.

Overall Attitudes Toward ARS

Extensive data has been collected indicating that student attitudes toward ARSs are very positive (Caldwell 2007; Fies and Marshall 2006; Judson and Sawada 2002; Simpson and Oliver 2007). Students have also reported that ARSs are easy to learn and use (e.g., Hinde and Hunt 2006; Pelton and Pelton 2006; Pradhan et al. 2005; Sharma et al. 2005; Siau et al. 2006).

Student Involvement

When an ARS is used in a higher education classrooms, students are more engaged in the content presented (e.g., Bergtrom 2006; Preszler et al. 2007; Simpson and Oliver 2007), participate more (Caldwell 2007; Draper and Brown 2004; Greer and Heaney 2004; Siau et al. 2006; Stuart et al. 2004; Uhari et al. 2003), and pay more attention to concepts presented (e.g., Bergtrom 2006; Draper and Brown 2004; Jackson et al. 2005; Latessa and Mouw 2005; Siau et al. 2006; Slain et al. 2004). In addition, a number of researchers have observed that ARSs stimulate discussion, especially when they are used with a peer instruction strategy (Beatty 2004; Brewer 2004; Jones et al. 2001; Nicol and Boyle 2003). With this strategy, students felt they were better able to discuss and improve their understanding of higher level concepts (Draper and Brown 2004).

Assessment

Evidence suggests that using an ARS improves the feedback cycle between instructor and students with the rapid, anonymous, collection and presentation of all student responses to questions asked (Abrahamson 2006; McCabe 2006; Pelton and Pelton 2006). Students also enjoy seeing how well they are doing relative to their peers (Burton 2006; Caldwell 2007; Draper and Brown 2004; Hinde and Hunt 2006; Simpson and Oliver 2007). In addition, considerable data supports the use of ARSs for providing effective formative assessment of student knowledge (Beatty 2004; Bergtrom 2006; Brewer 2004; Caldwell 2007; Draper and Brown 2004; Dufresne and Gerace 2004; Greer and Heaney 2004; Jackson et al. 2005; Siau, et al. 2006; Simpson and Oliver 2007; Stuart et al. 2004). Finally, some indirect evidence suggests that students do not like using ARSs when participation is associated with a grade (Caldwell 2007).

Learning

Many higher education students report that they learn more when ARSs are used (Greer and Heaney 2004; Nicol and Boyle 2003; Pradhan et al. 2005; Preszler et al. 2007; Siau et al. 2006; Slain et al. 2004; Stuart et al. 2004; Uhari et al. 2003). Some students prefer discussing ARS-based questions with their peers who have a similar language and therefore can articulate solutions more successfully than the instructor (Nicol and Boyle 2003; Caldwell 2007). Other students maintain answering ARS questions forces them to think more about the important concepts (Draper and Brown 2004; Greer and Heaney 2004) resolve misconceptions (D’Inverno et al. 2003). Finally, there is substantial qualitative and quantitative evidence to suggest that learning performance increases as a direct result of using ARSs (e.g., Carnaghan and Webb 2007; Fagan et al. 2002; Kaleta and Joosten 2007; Kennedy and Cutts 2005; Pradhan et al. 2005; Preszler et al. 2007; Schackow et al. 2004).

Challenges Associated with Using an ARS

Three categories of challenges have been reported by students and instructor using ARSs: technology, time, and method. With respect to technology, students who forgot to bring their remote devices to class were unable to fully participate in ARS-based discussions (Caldwell 2007; Reay et al. 2005). Another more critical technological issue occurred when the remote devices did not work, a particularly stressful situation during a formal evaluation (Sharma et al. 2005; Siau et al. 2006).

Several challenges associated with ARS are related to time. There are very few classroom-ready, subject-specific collections of ARS questions available, so instructors have to develop original questions. Many find this task extremely time consuming (Allen and Tanner 2005; Beatty et al. 2006; Fagan et al. 2002; Freeman et al. 2007; Paschal 2002). In addition, the time required to set up an ARS, hand out the remote controls at the beginning of the class, and collect remote controls at the end of the class can be significant (Hatch et al. 2005, Stuart et al. 2004). Furthermore, some discussions, if not moderated properly, can consume considerable class time by drifting away from the main concepts (Nicol and Boyle 2003, Reay et al. 2005).

A final set of challenges involves students who react unfavorably to the use of an ARS because the method of learning has been changed. They are accustomed to lectures and a switch in teaching format can lead to stress, frustration, and resistance at first (Beatty 2004; Fagan et al. 2002). In addition, using ARSs require more cognitive energy and cooperation from students and this extended effort may not suit students who are accustomed to passive lectures (Trees and Jackson 2007). Finally, some students doubt their ability to direct their own learning when ARSs are used (Allen and Tanner 2005).

Summary and Purpose of Study

Previous research on the use of ARSs in classrooms has been conducted almost exclusively in the domain of higher education. Benefits reported when ARSs are used have included greater student involvement, a positive reaction to formative assessment, improved quality of learning, and increased learning performance. Challenges identified when using ARSs have consisted of malfunctioning technology, increased demands on teaching time, and students having to adjust to a new way of learning. Since the use of ARSs is increasing rapidly in secondary schools (Abrahamson 2006), it is prudent to broaden the scope of current ARS research in the area of science where a majority of higher education instructors have used ARSs. Therefore, the purpose of the current study was to examine benefits, challenges, and strategies associated with using ARSs in secondary school science classrooms. Three questions were addressed:

  1. 1.

    What were the perceived benefits of using an ARS in a secondary school science classroom?

  2. 2.

    What were the perceived challenges in using an ARS in a secondary school science classroom; and

  3. 3.

    How did teaching strategy influence the use of ARSs in a secondary school science classroom? (summative vs. mixed vs. formative assessment).

Method

Sample

Students

The student sample consisted of 213 students (107 males, 105 females, 1 missing data), enrolled in grades 10 (n = 50), 11 (n = 81), and 12 (n = 82). Subject areas where an ARS was used included biology, chemistry, physics and general science. Eight-six percent of the students claimed that they were comfortable or very comfortable with technology (n = 182). Sample population data was collected from seven classrooms in six different secondary schools. All students were selected through convenience sampling and had to obtain signed parental permission to participate.

Teachers

The teacher sample consisted of seven teachers (5 males, 2 females), with 8–26 years of teaching experience (M = 18.0, SD = 7.0). Six out of seven the teachers reported that they were comfortable or very comfortable with technology.

Procedure

Teachers were emailed by an educational coordinator and informed of the ARS study. Participation was voluntary and subjects could withdraw from the study at any time without penalty or prejudice. Each teacher received two half days of training in November and February on how to use the ARS software and possible strategies for using an ARS in the classroom. Teachers were then asked to use an ARS in their regular classrooms, although frequency of use was controlled by the individual teacher. Each teacher had access to a laptop computer, an LCD projector, and one ARS system (32 clickers and a receiver) from E-Instruction (see http://www.einstruction.com/ for details of the specific ARS brand used). This equipment was purchased independently by the school board in where the study was conducted. All teachers used the ARS for a three month period, however, data collected for this study focussed on the last month. During the final month in which ARSs were used, all of secondary school science students reported using the ARS one to two times.

Data Sources

Student Survey

Based on the last month in which an ARS was used, students completed the ARS Attitude Survey for Students (Kay 2008a). This survey consisted of 11, seven-point Likert scale items that focused on the four key areas reported in previous ARS research: overall attitudes, student involvement, assessment, and learning. Cronbach’s reliability coefficient for the ARS survey was 0.88. However, since this was a formative analysis of student attitudes toward the use of ARSs in secondary school science classrooms, items on the scale were analyzed individually to gather as much information as possible.

Student Comments

Students were asked the following open-ended question: “What was the impact of clickers on your learning in the past month”? A coding scheme was developed to categorize 255 student comments made by 175 students (Kay 2008b). Some students made more than one comment, while other students offered no response. No student made more than 3 comments. Each comment was rated on a five-point Likert scale (−2 = very negative, −1 = negative, 0 = neutral, 1 = positive, 2 = very positive). Two raters (the authour and a trained research assistant) assessed all comments made by students based on category and rating value. After round one, inter-rater reliability was 83% for categories and 93% for ratings. Comments where categories or ratings were not exactly the same were shared and reviewed a second time by each rater. After round two, an inter-rater reliability of 98% was reached for categories and 99% for the rating values.

Teacher Questions

Teachers were asked to indicate their gender, years of teaching experience, comfort level with computer technology, and whether they used an ARS for formative and/or summative assessment. Finally, teachers were asked to respond to open ended written questions about the overall impact of ARS on their classroom and suggestions for future teachers.

Results

Benefits to Using an ARS

Overall Attitudes

Students were somewhat positive about using ARSs with 62% agreeing that they preferred to use an ARS (M = 5.0 ± 0.26, possible range was 1–7) and 42% indicating that their class was better when an ARS was used (M = 4.5 ± 0.21, see Table 1). General comments offered from the open-ended survey question were mixed with the majority being neutral (n = 13). Representative neutral comments about ARS were “It didn’t affect me” and “It did not make much of a difference”. Typical negative comments (n = 5) were “Clickers are a waste of time” and “Horrendous—takes up too much time”. Characteristic positive comments were “It was good” and “Amazing”.

Table 1 Summary of ARS quantitative student survey questions (n = 213)
Full size table

Of the 17 comments made about student reaction to using a different method of teaching, 11 were positive (see Table 2). Sample positive comments were as follows:

Table 2 Summary of student comments about audience response systems (n = 255)
Full size table

It’s fun to use and adds a certain aspects to the class that you wouldn’t have in a regular class.

I learned better because it promoted friendly competition within the class. Also it made reviewing easier and more fun. It is better than reading questions out of a textbook.

I learn better than sitting and listening and it improves my multiple choice skills.

Student Involvement

Students indicated that they were more involved in science class when an ARS was used. Based on the survey data, 70% of the students agreed that they were more engaged (M = 5.2 ± 0.21), 63% agreed they were more motivated (M = 5.0 ± 0.23), 62% agreed that they participated more than they normally would (M = 5.0 ± 0.21), and 53% agreed that more class discussion was generated (M = 4.6 ± 0.20).

Student comments regarding engagement were universally positive (M= 1.24, scale ranged from −2 to +2). Typical comments were as follows:

It motivated me and helped me a little with constantly paying attention.

Using the clickers made the lesson more enjoyable.

Keeps me awake and helps to keep me interested in the subjects we are learning.

Fewer comments were made about participation than engagement (n = 17), although the mean rating was high (M = 0.94) and most comments were positive (n = 15). Typical comments about participation included:

I’ve only used them once but I really liked them. I thought it was a great way to be involved. Everybody was doing something.

It forced people who normally don’t feel comfortable participating in class discussion to participate.

Only 12 comments were made about paying attention, but the mean rating was high (M = 1.00). Sample attention-based comments were as follows:

I am forced to pay more attention in class which helps me learn better.

I was more inclined to listen in class because I knew that the lesson would be shortened and therefore I could focus more on the lesson.

Assessment

Students liked using ARS for formative assessment. Just under 75% of students agreed that using ARSs was a “good way” to test their knowledge (M = 5.3 ± 0.20). Comparing answers with other students (M = 4.8 ± 0.20) was rated somewhat lower, with 56% agreeing that they liked to see the responses of their peers. Students were split on how they felt when they selected a wrong answer and the rest of the class chose a correct answer—35% felt bad while 39% did not feel bad. Finally, only 33% of the students agreed that they liked using an ARS for formal tests or summative assessment (M = 4.1 ± 0.24).

Comments about formative assessment were rated high (M = 1.00), however few comments were made about comparing responses or not getting a correct answer. The following are typical comments made about formative assessment:

Helped me by testing my knowledge and getting to know where the class and I are at.

Clickers were fairly good in a normal class and fairly good way to test our knowledge. However, on tests, it is very inefficient and can waste time.

We found out what common mistakes were made more commonly, then discussed why that was the wrong answer.

Learning

Only 43% of the students agreed that they learned more when an ARS was used. The mean score for this item was 4.3 (±0.20), the second lowest of all survey questions. However, open-ended comments revealed a different picture. Ratings for comments about learning process (M = 0.52), reviewing previous concepts (M = 1.20), helping to improved memory (M = 0.62), and teacher explanations (M = 1.00) were relatively high. However, the mean rating for learning performance comments was negative (M = −0.71). It is important to note, though, that all of the negative comments about learning performance were made when ARSs was used for formal testing, as opposed to formative assessment. Comments made about the overall learning process (n = 29) and reviewing concepts (n = 15) were the most common. Typical remarks made about the learning process were:

Helped me figure out how much I learned.

Using the clickers helped me get a better understanding of some of the concepts that I did not understand before.

Since clickers were used to review for a test, I was able to obtain answers to test-like questions easily giving me insight as to what more studying was needed.

Representative comments about using an ARS to review concepts included,

It reinforced the idea of physics just before the test.

However, I found the practice test we did with the clickers very helpful.

It was great review of the units and topics we just covered, and I find I remember things better if I get them wrong first on a test and this system always seems like a test for me so I feel I learn more this way.

Comments made about an ARS helping to improve memory were less frequent (n = 8) but worth noting.

It actually helped me remember facts we have been learning better.

Because we could see the correct answer immediately after it helped me remember the answer and get my questions answered.

Challenges Experienced Using ARSs

Technology

Attitudes toward ARS technology were not assessed using the survey data, but the mean rating for comments was negative (M = −.1.00), albeit only for eight students. The majority of student complaints were targeted toward whether the remote devices actually worked.

They are too difficult to use and it was difficult to tell when the correct answer was chosen.

Took more time to learn to use them than time spent learning; caused confusion; didn’t like pointing at the receiver; not good in a test situation.

Overall Attitudes

While most “overall” attitudes toward ARSs were neutral, comments from five students were exceptionally negative (e.g., “Clickers are a waste of time” or “Horrendous—takes up to much time”). Other comments were focussed using a new method to learn.

I prefer to listen to the teacher—using the clickers constantly would become tedious.

It went slowly and I don’t like the time limit questions.

Student Involvement

As stated earlier, most students were enthusiastically involved when an ARS was used, however some students were stressed. Typical comments referred feeling pressure during a test.

Using the clickers on a test is very stressful. Not only are we stressed because of the test conditions, but it’s difficult to work the clickers.

The clickers have made me nervous in test situations and have made me lose confidence in situation where immediate feedback was given. The clickers have had an altogether negative feedback on my learning experience in biology class.

Assessment

The survey indicated that 35% of secondary students “felt bad” selecting an incorrect response when almost everyone else in the class selected a correct response (see Table 1). Most comments about assessment (see “Benefits”) and ARSs were positive.

Learning Performance

Even though reviewing concepts, improving memory, and obtaining explanations from the teacher were rated as positive consequences of using an ARS, a small group of students (n = 23) were adamant about the negative impact of the ARSs on learning performance. Representative comments included:

Frustrating, stressful to use in test situations.

The time constraints compared to a normal test were very annoying since the slow students miss out and the speedy students get bored.

The clickers are NOT good to use on long tests; they’re frustrating and hard to check your answers.

Teaching Strategy Used with ARSs

Comprehensive information was not collected on how ARSs were used in secondary science classrooms, however, three general assessment-based strategies were reported by teachers: using an ARS for formative assessment only (formative, n = 130), using an ARS for both summative tests and formative assessment (mixed, n = 45), using an ARS for summative assessment only (summative, n = 38). Means for all items on the ARS student attitude scale as a function of strategy selected are presented in Table 2. Mean values for all attitude scale items were lowest when a summative assessment strategy was used and highest when a formative assessment strategy was employed.

A MANOVA was run to compare formative, mixed, and summative approaches to using the ARS (Table 3). Using an ARS for formative assessment was rated significantly more positively than using an ARS for summative assessment on all 11 Likert scale items in the ARS attitude scale. Using an ARS for formative assessment also resulted in significantly higher scores on most survey items when compared to a mixed approach (formative & summative) (Table 4).

Table 3 Mean ARS survey item scores as a function of strategy used (n = 255)
Full size table
Table 4 MANOVA examining attitude toward ARS as a function of teaching strategy
Full size table

Teachers’ Comments

When science teachers were asked about the overall impact of ARSs, two main themes emerged: learning and motivation. With respect to learning, comments focussed on the value of feedback, not having to write everything down, and examining areas of weakness.

The immediacy of the feedback permits me to identify difficulties and address them quickly.

My class really struggles with written work so the clickers were very popular.

By using clickers as an assessment tool, I can focus lessons on areas of weakness.

Regarding motivation, using an ARS appeared to increase the overall energy level in the science classroom.

Clickers have made my review sessions very exciting and motivating for my students.

They benefited—the positive energy was good. For example they would say yeah we’re using the clickers today instead of your usual boring lesson!

When prompted to give suggestions to future teachers about using ARSs, science teachers in this study offered the following observations. First, set up of ARS is time consuming, as is the creation of good questions. Second, keep the clicker sessions short to avoid losing the focus of a lesson. Finally, sometimes the remote devices do not work and students have difficult registering responses—using the more reliable radio frequency (as opposed to infrared) remote devices was encouraged. It is important to remember that these comments were collected from only seven secondary school science teachers.

Discussion

The purpose of this study was to conduct a formative ARS use in secondary school science classrooms. Three areas were examined including benefits, challenges, and teaching strategies.

Benefits

Overall Attitudes

The survey data suggests that science students were moderately positive about using ARSs, a result that is partially consistent with previous studies conducted in higher education. Part of the problem with assessing overall attitudes is that positive and negative attitudes can cancel each other out leading in a neutral result. As Fies and Marshall (2006) suggest, general impressions and anecdotal comments about attitudes toward ARSs need to be supplemented by a more detailed, focussed analysis in order to provide a more in depth analysis.

Student Involvement

Relative to assessment and learning, student involvement was rated highest by students on the survey data and open-ended comments. Data collected strongly suggested that ARSs helped increase engagement, participation and, to a lesser extent, attention paid in class. These results are consistent with those observed in higher education (e.g., Caldwell 2007; Fies and Marshall 2006; Judson and Sawada 2002; Simpson and Oliver 2007). It is unclear, though, whether increases in student involvement are a superficial reaction to the excitement of using TV-like remote devices in a game like atmosphere, or a reflection of a more profound change in student learning and cognitive engagement. The next stage of research that needs to be done with respect to student involvement is to examine what in particular is engaging about ARSs, the quality of the participatory efforts by students, and why students are more focussed when an ARS is used.

Assessment

A majority of secondary school science students appreciated the use of ARSs for formative assessment particularly with respect to checking their understanding and reviewing for tests. Comparing answers with other students and getting general feedback played a secondary role. In addition, most teacher comments enthusiastically endorsed the use ARSs for formative assessment. These results are consistent with those observed in higher education (Beatty 2004; Bergtrom 2006; Brewer 2004; Caldwell 2007; Draper and Brown 2004; Dufresne and Gerace 2004; Greer and Heaney 2004; Jackson et al. 2005; Siau et al. 2006; Simpson and Oliver 2007; Stuart et al. 2004). The success of using formative assessment with ARSs stands in marked contrast to students’ resistance to using summative assessment.

Learning

A number of students commented that using ARSs to review concepts improved the overall learning process. The impact of using ARSs for improving recall of material presented was noteworthy, but commented on by only a few students. While challenges will be discussed in the next section, it is important to note that perceptions of improved learning process did not necessarily lead to increased learning performance. At least 23 students felt that learning performance was hampered, sometimes appreciably, when ARSs were used to deliver summative assessment. Simply using an ARS does not guarantee success and choice of pedagogy clearly effects students perception of learning.

Challenges

Overall Attitudes

It is evident that a few students had difficulty adjusting to the newness of using an ARS. While this type of problem was experienced by less than 2% of the student sample, it is challenge that has also been noted in higher education. The suggested remedy has been to engage in fun practice sessions before use ARSs for teaching, clearly explain the rationale for using an ARS, and identify the intended benefits for students (Crouch and Mazur 2001; Trees and Jackson 2007).

Student Involvement

While a majority of students were more involved when ARSs were used, some students were decidedly stressed, particularly in a summative testing situation. The comments about increased stress were not frequent, but they were passionate. Given the student frustration level when an ARS was used for summative tests (see “Teaching Strategy Used with ARSs”) it might be prudent to use this technology for formative assessment purposes only. Student anxiety might also be reduced by emphasizing the learning versus having to get the right answer.

Learning

The most noteworthy learning challenge when using ARSs was the negative impact on learning performance perceived by students who were asked to use this tool to complete graded tests. Overall rating of this practice was the lowest rated survey item. However, this study was a preliminary investigation of the use of ARSs in secondary schools science classrooms, and more thought and analysis needs to be directed toward the use of summative assessment.

Teaching Strategy Used with ARSs

Only one previous study reviewed the use of ARSs in a K-12 environment (Penuel et al. 2006). These researchers observed that teachers naturally migrated to one of two strategies when using ARSs: formative (instructional) or summative (testing) use. A similar pattern was observed in the current study with secondary school science teachers. Most of the science teachers in the current study chose to use an ARS as a formative assessment tool. Based on the survey results, students overwhelmingly preferred formative over summative use. Overall attitude, student involvement and the value of ARSs as effective learning tools were all rated significantly lower when students experienced summative as opposed to formative assessment. No research to date has been done comparing formative and summative approaches, so further research is needed to confirm these initial findings.

Recommendation to Educators

The current study is formative analysis of the use of ARSs in secondary school science classrooms, so it would be inappropriate to offer unequivocal advice about their use in the classroom. That said there are several tentative suggestions that may be worth noting.

First, many secondary school science students did not respond well when an ARS was used as a test-taking tool. Student involvement and learning performance were perceived as decreasing. In the interest of minimizing the negative impact of this tool, at least in its initial implementation in secondary schools, it might be safer to limit the use ARSs to formative assessment.

Second, it may be important to explain why an ARS is being used in the classroom. This practice might help the small, but vocal group of students who resisted this new method of learning. Caldwell (2007) and Trees and Jackson (2007) note that if you expect to gain full student acceptance, you need to explain why you are using ARSs and what you expect to achieve with the technology.

Third, teachers should be aware of the time needed to set-up an ARS and more importantly, to create effective questions. While only two teachers commented on this issue, previous research in higher education suggests that question development is very time consuming (Allen and Tanner 2005; Beatty et al. 2006; Fagan et al. 2002; Freeman et al. 2007; Paschal 2002).

Finally, because stress was experienced by students when the technology did not work, teachers should test the ARS equipment ahead of time in order to ensure that all remote devices respond properly. Batteries, for example, might need to be changed.

Caveats and Future Research

This study investigated the use of ARSs in secondary school science classrooms. Two principal data collection tools were used—survey questions and open-ended comments. The design and analysis of the data collection tools were based on a thorough review of the literature examining ARS use in higher education. Nonetheless, at least three caveats need to be considered when interpreting the current results.

First, the data are intended to provide a starting point for investigating use of ARSs in secondary school science classrooms. More detailed examination is required in key areas such as indentifying sources of student engagement, the effect of different teaching strategies, and why long-term memory might be enhanced with the use of ARSs.

Second, the results are based on limited use of an ARS once or twice over a one month period. The impact could be distinctly different if an ARS was used regularly. For example, engagement may be reduced when students become more familiar with ARSs. On the other hand, the impact of an ARS on learning may be more significant with increased use.

Finally, learning performance needs to be examined in more detail. While systematic data on the perceived quality of learning was collected for the current study, evaluation of student performance is needed to firmly establish the actual learning impact of ARSs.