Introduction

The U.S. Department of Education (2002) has defined technology integration as “the incorporation of technology resources and technology-based practices into the daily routines, work, and management of schools” (Ch. 7, para. 3). As digital technologies have become more ubiquitous and as more stakeholders have taken an interest in technology for education, we have seen a corresponding growth of interest in establishing models and methods for improving teaching and learning with technology. Beginning in 2007, the International Society for Technology in Education (ISTE) released national educational technology standards (NETS) for teachers, students, and administrators, with the purpose of standardizing expectations for teaching and learning with technology (NETS 2012). The NETS paved the way for establishing standardized ways of thinking about technology integration, which many states have followed when drafting their own standards for student learning and teacher licensure (e.g., Idaho Pre-Service Teacher Technology Standards). The Common Core State Standards are also an example of how technology has been infused into the education system (CCSSI 2010). Though there is no separate technology section to the Common Core State Standards, technology literacy and the use of technology devices are written throughout the standards, suggesting that effective teaching and learning now requires not only technical competency, but also an understanding of how technology interacts with content knowledge and teaching practices.

In light of these developments, teacher education programs have sought to provide learning experiences to aspiring teachers that will help them to incorporate technology effectively in the classroom. Though there have been some assumptions that new teachers will bring a certain baseline understanding of technology that will improve their ability to teach with it, such assumptions are generally based upon non-empirically validated narratives of new teachers as members of the millennial, net, or digital native generation (Parette et al. 2004; Selwyn 2009). Such assumptions are problematic for three reasons. First, they do not take into account the diverse experiences of pre-service teachers and varying levels of technological competency amongst them. Second, they assume that literacy surrounding consumer use of technology can easily translate into effective learning and teaching practices with technology. And third, they do not account for systemic and contextual factors that influence teachers’ access and utilization of technologies in their classrooms (cf. Donnison 2007).

Further, as we more fully recognize the importance of technology literacy in terms of how students and teachers use technology versus merely whether or not they are using technologies, we begin to see that problems associated with technology integration may have much less to do with access and much more to do with effective use in practice (cf. Al-Awidi and Alghazo 2012; Jenkins et al. 2006) or barriers to effectively using certain types of technology, such as cloud computing or social networking sites (Donna and Miller 2013; Veletsianos and Kimmons 2013).

For teacher education programs, this poses a significant difficulty, because programmatic work must be able to help pre-service teachers to not only use technologies competently but to use them in ways that effectively connect with practice and employ critical thinking to support effective learning experiences in real-world contexts. In this study, we explored one teacher education program’s approach to supporting pre-service teachers to improve technology literacy by asking the following two questions:

  1. (1)

    Are some technology-specific performance tasks more likely to lead pre-service teachers to think about technology integration in specific ways (e.g., replacement, amplification, transformation)?

  2. (2)

    Does the way a pre-service teacher thinks about technology integration (in terms of replacement, amplification, or transformation) impact their self-assessed technology integration competence?

Theoretical framework

Various research studies have sought to ascertain the value of diverse technologies in educational contexts. As one example, student response systems (or clickers) have been utilized to improve learner engagement and achievement (Blood and Neel 2008; Gauci et al. 2009; Shaffer and Collura 2009) and may increase student willingness to express opinions in class (Stowell et al. 2010). As another example, interactive whiteboards have been identified as providing multiple pedagogical opportunities for supporting flexible instruction and improving student achievement (Marzano 2009; Roblyer and Doering 2013). Both of these technologies are relatively new to education, and using each in a classroom setting may require teachers to fundamentally rethink some aspects of their practice and to respond to technological benefits and limitations. Laptops, smartphones, digital cameras, digital audio players, e-book readers, and tablet computers are other examples of technologies that have educational potential but that should be implemented in ways that are critical and evaluative of actual learning outcomes.

As new hardware and software technologies are introduced to classrooms on a daily basis, the current state of teaching with technology in classrooms has become a moving mark for teacher education programs to hit. It is difficult to know what technologies are available in a given classroom today, let alone what will be available 5 or 10 years down the road and what expectations will be placed on teachers with regard to technological fluency. Understandably, teacher education programs have struggled to prepare teachers to teach effectively in classrooms with existing technologies and have had difficulty articulating what technology literacy looks like in a world where technology is always changing (Stanford and Reeves 2007).

Further, there is much more to teacher technology education than mere facility with technology tools. Just as access to technology does not equate to literacy with technology (Jenkins et al. 2006), the ability of novice teachers to use technology for personal purposes does not readily translate into the ability to effectively teach and learn with technology. The prominent technological pedagogical content knowledge (TPACK) model of technology integration illustrates this well and suggests that technology is only one of three essential components for thinking through what effective technology integration looks like (Mishra and Koehler 2006). Just as a teacher must understand how technology works, that teacher must also understand the content area and pedagogy and the nuanced and interconnected relationships between all three areas (Walker et al. 2012). According to the TPACK model, teaching well with technology is something that is “difficult to do well” and can only be addressed by confronting “wicked problems” of practice in highly contextualized situations (Borko et al. 2009; Mishra and Koehler 2006, 2007). For this reason, we do not provide an operationalized definition of “effective technology use” in this paper, because truly effective technology use is complex, contextual, and multi-faceted.

This complexity requires teachers to become self-regulated learners when it comes to incorporating technology by developing cognition, metacognition, and motivation (Barak 2010). Without such self-regulation, teachers will be ill-equipped to develop and apply strategies of technology integration in their classrooms. Developing self-regulation, however, is itself a complex task that requires forethought, performance, and self-reflection (Zimmerman and Schunk 1989), and as such, teacher education programs need to engage pre-service teachers in processes that require them to plan, execute, and reflect upon their own technology use (Romano and Schwartz 2005). Such processes of reflection and critical inquiry are not new and have been proposed as essential in teacher education for almost a century (Dewey 1933; Yost et al. 2000), but technology integration is not always approached in this manner. Through reflective processes, pre-service teachers must make a shift from thinking about technology as just another content area toward thinking of technology as a tool for supporting student learning (Niess 2005; Vannatta et al. 2001), and these experiences must help them to develop a sense of self-efficacy with technology in order to be effective in the classroom (Al-Awidi and Alghazo 2012; Darling-Hammond et al. 2002).

Hofer and Grandgenett (2012) describe a teacher education program that focuses on integrating technology throughout all courses but also provides a technology centered course that presents opportunities for the development of technology-enhanced, curriculum-based teaching and learning. In such a course, students complete technological application assignments and reflect on their learning, concluding with a capstone technology-integrated lesson plan. Through such experiences it is expected that students will be able to connect technological skill to essential content knowledge and pedagogical practices, thereby developing applicable practices for using technology in their teaching contexts.

Such contextualized approaches, however, do not necessarily address the issue of preparing teachers to operate in a world where technology is ever-changing unless they are also coupled with a strong critical component that leads teachers to reflectively evaluate their own technology integration practices in terms of outcomes and effects (cf. Dawson 2006). To illustrate, one model of technology integration called RAT (Replace, Amplify, Transform) considers the role that technology plays in any teaching and learning context and asks whether the technology “replaces” previous practice, “amplifies” current practice, or “transforms” practice into something altogether new (Hughes et al. 2006). The creators of RAT viewed technology use in the classroom “as a means to some pedagogical or curricular end” and focused upon the reasoning and goals of teachers that direct their choices toward adopting technologies with an end in mind (Hughes et al. 2006, p. 1616). In this view, technology serves a pedagogical or curricular role in the classroom, and building upon empirical studies of teacher behaviors and beliefs in the classroom (Hughes et al. 2006), RAT seeks to identify that role and to make it explicit.

RAT was developed “in consultation with past research,” “theories about technology integration,” and “analysis of classroom observations and teacher interviews” (p. 1616). Though RAT authors do not provide an operationalized definition of “technology,” they applied it to a wide variety of tools and associated practices, including such diverse examples as word processing, video games, and Web 2.0 tools as long as they have “explicit subject matter connections” in how they are being used (p. 1617). In this manner, RAT focuses less on what technologies are being used and more upon what role those technologies are fulfilling in specific classroom settings, conceivably making it applicable to any instantiated artifact used in a classroom context, be it digital, analog, etc.

RAT authors theoretically defined use case categories as replacement, amplification, and transformation, and proposed a method for systematically assessing particular technology uses based upon the instructional methods being used, student learning responses, and curriculum goals (Hughes et al. 2006, p. 1616). Transformation draws upon Pea’s (1985) concept of technology “redefining” possibilities and Cuban’s (1988) concept of “second-order” change, wherein technologies are used to fundamentally change student mental processes and instructor instructional processes, allowing teachers to teach “skills and content in ways impossible in the traditional classroom” (Mason et al. 2000, p. 107). From the same foundations, amplification considers improving efficiencies and reach with technology (Pea 1985) or “first-order” change (Cuban 1988), thereby streamlining the learning process while leaving it unchanged in form. Replacement, on the other hand, emerged as a new construct to account for instances wherein technology “serves merely as a different means to the same instructional end” (p. 1617). Such a lens provides a focused view by which a teacher or researcher can evaluate prospective technologies for integration against existing practices (Hughes et al. 2006), and priorities can be placed on those technologies worthy of investment due to their transformational potential or technologies that may be ignored because they only replace what is already done well through a lower tech means.

Though each technology integration model is different in that it tends to suit users in some roles better than others (i.e. IT director, teacher, or administrator), it seems that model adoptions are rarely based on a clear rationale of model value but rather out of habituation (Kimmons and Hall in press). For our research, the RAT model embodied a parsimony that is desirable for teacher use, and in this study, we utilize the RAT model to analyze pre-service teachers’ critical thinking about technology integration and apply it as a useful lens for understanding their ways of thinking that emerge through a technology integration course.

Context

This study was conducted in conjunction with sections of an established technology integration course at a public university in the United States. As such, contextual background about the course, national and state expectations, and design decisions are now explained in more detail. In many areas within the U.S., there is a push to embrace educational technologies that can enhance instruction. In our context, the state of Idaho is making progress toward mandating technology integration in pre-service teacher courses and has adopted a set of required pre-service teacher technology standards. The Idaho State Department of Education (ISDE 2011) has also recently made the following statement:

All applicants for initial Idaho certification … from an Idaho approved teacher education program must demonstrate proficiency in relevant technology skills and practices to enhance classroom management and instruction. Each Idaho public higher education institution shall be responsible for the assessment of pre-service teachers in its teacher preparation program. The assessment must measure understanding and the ability to apply strategies and beliefs about the integration of technology based on current research and best practices congruent with the International Society for Technology in Education professional teaching standards, the National Council for Accreditation of Teacher Education standards, and state accreditation standards.

Through such rulings and the tangible need to prepare teachers to appropriately use technology in educational contexts, a course was developed in the target university that addresses the identified standards.

The course Technology, Teaching and Learning is delivered through face-to-face or online instruction and is designed to prepare pre-service teachers to effectively integrate technology in support of their instruction. Students enrolled in the course are typically college juniors and seniors who are approaching their student teaching experience. Students are education majors who will receive an endorsement in a content area (e.g. Mathematics) along with certification in K-8 teaching or secondary education. Students who take this course have generally completed much of their coursework in methods and pedagogy, which allows them to have some theoretical grounding for classroom application of technologies that pre-service teachers at an earlier stage of development might lack.

The main goal of the course is to support students in the integration of technology into classroom instruction to enhance teaching and student learning of core content (concepts, skills, attitudes) in specific contexts. Course objectives include:

  1. (1)

    Using a variety of software applications applicable to a classroom setting;

  2. (2)

    Using various technologies effectively to deliver a lesson;

  3. (3)

    Discussing how technology allows students to represent and communicate what they learn;

  4. (4)

    Planning classroom instruction that integrates technology that students understand using high quality pedagogy;

  5. (5)

    Developing an ability to critically evaluate technology through a personal understanding of the benefits and drawbacks of individual technologies and the barriers that may be encountered with implementation;

  6. (6)

    Developing strategies to learn technical skills, troubleshoot, and manage technology use within a classroom;

  7. (7)

    Developing a vision for teaching with technology.

Since it is impossible to cover all the technologies pre-service teachers will encounter in a school setting, this course engages the idea of suites of technology, which is designed to give students a concrete introduction to commonly used technologies. This idea of suites groups technologies that have similar foundational purposes or benefits. For example, cloud-based technologies afford users collaborative spaces where content can be created, edited, and shared in a commonly accessible cloud platform. Another example would be concept-mapping tools or digital storytelling software. By providing students with an opportunity to explore a suite of technology, the conversations in class and accompanying reflection documents are rich with exploring the pros and cons of various technologies within a suite as well as an evaluation of the technology given a particular use. The two-credit face-to-face course meets once a week for approximately 2 hours; the online course is entirely asynchronous. Throughout the course, students are given select readings from peer reviewed journal articles, salient learning technologies blog posts, and other sources representing preeminent thinkers in the field that align with required state standards. Students demonstrate their technological performance competence through specific suites of technology that culminate with six performance tasks and reflections.

Methods

This study sought to understand the relationship between the course performance tasks and pre-service teachers’ technology integration learning outcomes. In our teacher education program, we believe that it is valuable to both increase pre-service teacher self-efficacy with technology and also to lead our pre-service teachers to think critically about technologies for educationally valuable application. The two guiding research questions for this study were:

  1. (1)

    Are some technology-specific performance tasks more likely to lead pre-service teachers to think about technology integration in specific ways (e.g., replacement, amplification, transformation)?

  2. (2)

    Does the way a pre-service teacher thinks about technology integration (in terms of replacement, amplification, or transformation) impact their self-assessed technology integration competence?

The first research question is important, because if it is the goal of teacher educators to lead pre-service teachers to think and apply technologies in specific ways, then it is important for us to recognize which performance tasks and technologies lend themselves to specific ways of thinking about technology integration in educational contexts so that we can focus our attentions on those that yield the most valuable learning outcomes. The second research question is also important, because as many teacher education programs rely upon self-assessments to determine aspects of pre-service teacher competence in technology integration, teacher educators should be able to determine whether or not pre-service teacher self-assessed competency reflects candidates’ thinking of technology in valuable ways. These research questions build upon the current literature in this area by examining how pre-service teachers think about technology integration and how this thinking impacts self-assessment. Exploring these issues is essential, because most studies on pre-service teacher technology integration rely heavily upon self-assessment and whether or not technology is used rather than upon how technology use and technical competence relate to pre-service teachers’ instructional goals. To answer these questions, we employed a mixed methods research design (Creswell 2003). We describe our participants, data collection methods, and analysis in the following sections.

Participants

Participants in the study were undergraduate students in an education program at a medium sized university in the western United States. For the purposes of this paper, we refer to these students as pre-service teachers because they were preparing to enter the teaching profession. As a component of the teacher education program, pre-service teachers were required to take the aforementioned two-credit course, Technology, Teaching and Learning, and the course served as a teaching moment for students to elicit their critical thinking about technology integration. Only these students were invited to participate in the study, because they were the population being studied and because the studied performance tasks were products of the course. Opening participation up to others would have required additional participants (1) matching students enrolled in the course in terms of teacher preparation experience, (2) completing the four performance tasks, each of which consisted of many hours of work, (3) self-reflecting on those performance tasks, and (4) self-evaluating with the pre- and post-surveys. Given the research questions, it did not seem prudent or necessary to expand recruitment beyond the students enrolled in the course.

Participants were enrolled in one of four sections of the course during the 2012–2013 academic or summer terms. Three of these sections were taught in person and one section was taught online. The online section was included, because there was no evidence-based reason to assume that learning outcomes would be different as a result of the medium when instructional resources, assignments, and other course elements remained unchanged (Bernard et al. 2004; Jaggars and Bailey 2010; Means et al. 2013; Tallent-Runnels et al. 2006; Zhao et al. 2005). Three different instructors took part in teaching these four courses, and across the four sections, thirty-four pre-service teachers agreed to participate in the study (n = 34).

Data collection

Data sources for the study include a pre- and post-survey on self-assessed technology competence and written reflections on four performance tasks (Content Management Systems [CMS], Cloud Computing [Cloud], Digital Stories/Videos [DS], and Concept Mapping Software [CMAP]).

Pre- and post-survey

The survey instrument was developed to incorporate course objectives, professional standards, and an assessment of participants’ self-assessed competence with various hardware and software components pre-service teachers would likely experience in a classroom setting. This survey was administered at the beginning of the 16-week semester and again at the end of the course. The survey consisted of three sections, representing different theoretical constructs: Technology Goals, Hardware and Software, and ISTE Standards. The Technology Goals section included general statements about technology integration fluency and vision like “I am able to troubleshoot technical problems students may have with educational technology hardware.” The Hardware and Software section required students to rate their technology-specific proficiencies with tools like “Microsoft Word,” “Video Editing Software,” and so forth. The ISTE Standards section required students to self-assess on the NETS for Teachers standards (NETS 2012). The purpose of this instrument was to document growth in pre-service teacher competence through the course.

The survey was designed specifically for this study based on Dillman et al. (2009) tailored design method. The three sections in the survey were written purposely for the pre-service teacher audience. The items in Technology Goals were based explicitly on the course goals and objectives as designed by faculty at the onset of course implementation at the university. Hardware and Software focused on technologies pre-service teachers would encounter in the course or those that were commonly used in the teaching profession. The verbiage in ISTE Standards corresponds to the ISTE Standards for Teachers (ISTE-T). For example, ISTE-T Standard One states, “Facilitate and inspire student learning and creativity” and is accompanied by four specific sub-standards. To address this standard and to gather the essence of this goal for teachers, the survey item focused on ISTE-T Standard One as follows: “I am able to design or adapt relevant learning experiences that incorporate digital tools and resources to promote student learning and creativity [emphasis added].” All other survey items in this section were structured similarly.

Reliability testing on each of the survey constructs yielded highly reliable Cronbach’s alpha values as follows: Technology Goals (8 items, α = .89); Hardware and Software (13 items, α = .91); and ISTE Standards (7 items, α = .94). Overall survey results on all items were also highly reliable, yielding a Cronbach’s alpha of .96 for all aggregated survey responses, with α = .91 on the pre-test and α = .92 on the post-test administration. All survey items may be found in Table 1.

Table 1 Technology survey items
Full size table

Threats to the validity of this pre- and post-test survey method were minimized for several reasons. First, survey questions focused upon self-assessment of current knowledge and skill level rather than upon knowledge recall, thereby reducing recall bias. Second, social setting, survey question composition, and expectations of participants were the same in both administrations, and results were used comparatively, not generally. Therefore, any results potentially impacted by social desirability bias would be expected to be replicated in both pre- and post-test administrations, and comparative results would remain unchanged. And third, though response shift bias might occur as a result of students’ exposure to new ways of thinking about technology in the classroom, these shifts would be expected to be uniform across participants, due to their participation in the same learning experiences (e.g., all students learned about TPACK), and would therefore not impact between-groups comparative analyses.

Performance task written reflections

Performance tasks in the course are designed so that students have opportunities to engage with technology that is applicable for their future profession as teachers. For example, the Content Management System performance task requires students to make a classroom website in a Content Management System or web hosting platform (e.g., Google Sites, Wordpress, Wix, Weebly) including information about their background, their teaching philosophy, audience specific sections, calendars for parents, and links to websites for students. Through this assignment, students are required to incorporate multiple forms of media, such as photos, videos, and audio as components to their website. Other performance tasks are similar in that students are tasked with demonstrating competent performance as they engage with various technologies in ways that will be useful for teaching in their content areas. Performance tasks were selected by the research team and were based on similar assignments in previous sections of the course. Students enrolled in the course had not had prior exposure to the performance tasks. After completing each performance task, participants submitted a written reflection responding to the following questions:

  1. (1)

    Why would you use this technology to enhance your students understanding of a subject (conceptual understandings, skills of inquiry, processes, attitudes, nature of science, etc.)?

  2. (2)

    How could you see your students using this technology to enhance their understanding of a subject? Please provide a specific, detailed example.

  3. (3)

    How do you see yourself using this technology (to guide discussions, organize your classroom, improve your teaching knowledge, assessment, etc.)?

  4. (4)

    What issues do you see with this technology (concerns, drawbacks, barriers towards implementation, etc.)?

  5. (5)

    What could help you address these issues?

Responses to these items were submitted as a graded component in the course and generally took the form of a one paragraph reply to each question.

Data analysis

Data were analyzed using a mixed methods triangulation design approach, wherein quantitative and qualitative data were gathered together, analyzed separately, and then integrated for comparison (Creswell 2003). We began by analyzing the performance task reflections to determine how students were thinking about technology integration in accordance with the RAT model (Hughes et al. 2006). Upon initial reading of the reflections, we found that many students, as prompted, talked about hindrances of technology in classroom settings, and we recognized that this did not fit well within the RAT model, which seems to assume that technology has either a neutral or positive effect in a classroom, but not a negative effect. We also found that researchers initially disagreed on some interpretations of student reflections and whether they exemplified Replacement, Amplification, or Transformation. As a result, we developed a systematic process for assigning codes to each student thought chunk (i.e. statements representing a complete thought) in a reflection by asking four questions, which are provided in Table 2.

Table 2 Process for analysis of replacement, amplification, and transformation
Full size table

In this process, reflections on each performance task were coded based on the described activity or view of technology as thought chunks using the following coding list: hindrance, replacement, amplification, or transformation. We first asked if the outcome of the described activity was clearly negative in terms of student learning outcomes or classroom culture (e.g., confusion, unnecessary disruptions). If the description was clearly negative, we assigned the thought chunk a hindrance code and did not apply the RAT model. Otherwise, we asked whether the described activity might have been done without the technology or via a lower tech means (e.g., index cards, chalkboard). If this was not possible, we assigned the thought chunk a transformation code, because the technology was allowing for a transformational change in practice. Otherwise, we asked if the technology fundamentally changed the described activity so that the outcome was something new, different, or previously impossible (e.g., synchronous collaboration at a distance). If so, we assigned the thought chunk a transformation code, because the thought chunk reflected transformational practice. Otherwise, we finally asked if the outcome was clearly positive (e.g., improved efficiency). If so, we assigned the thought chunk an amplification code, because it was improving existing practice, and if not, we assumed that the described technology use was having a neutral effect and labeled the thought chunk as replacement, because it was not transforming or improving existing practice.

Through this process, our goal was to highlight instances when technology was used to “replace and, in no way change instructional practices, student learning processes, or content goals,” instances where technology amplified “current instructional practices, student learning, or content goals,” or transform the “instructional method, the students’ learning processes, and/or the actual subject matter” (Hughes et al. 2006, pp. 1617–1618). Each thought chunk was coded independently by two researchers, and then researchers compared codes and came to consensus.

To answer the first research question of whether some performance tasks (i.e. CMAP, Cloud, CMS, and DS) were likely to be applied in specific ways (i.e. transformative, replacement, or amplification), McNemar tests for comparing dependent proportions with a continuity-correction were examined (Agresti 2013). A continuity-correction was used because low cell counts makes the Chi square approximation of the McNemar statistic inappropriate (Edwards 1948). For transformation, replacement, and amplification, we examined all possible pairwise comparisons. For transformation, we examined whether CMAP was used more or less than Cloud, CMS, or DS; whether Cloud was used more or less than CMS or DS; and whether CMS was used more or less than DS. This resulted in a total of 6 tests. We examined these comparisons for replacement and amplification as well. As a result of the pairwise testing, we ended up with 18 (6 × 3) non-independent comparisons. In order to control for multiple comparisons, we used the Bonferroni adjustment (i.e. we multiplied the p value from all tests by 18).

To examine the second hypothesis that the way a student thought about technology integration in the classroom would impact their self-assessed competence of technology integration, we first took the technology goals and ISTE standards sections from the post-survey of self-assessed technology integration competence and created an average score for each participant, ranging from 1 to 5, to create a general competence score. We did not include the second section of the survey on specific technologies in the analysis, because many of these technologies were not addressed in the course. A series of regression models were then fit. First, students’ general competence scores were regressed on to their average amplification, transformation, and replacement scores. Second, each post-survey score from technologies addressed through course performance tasks (i.e. CMAP, Cloud, DS, and CMS) was regressed on to whether amplification, transformation, or replacement was associated with the technology. Initially, linear regression models were considered for the four performance task post-score regressions. However, the post-score for these measures was inherently ordinal. Given the low sample size, post-scores were examined using an exact logistic regression (Forster et al. 2003) as a sensitivity analysis. An exact logistic regression model is an MCMC-based logistic regression model that is appropriate when sample sizes are small or there are low or no observations for each level of a predictor. Post-scores were dichotomized collapsing the two lowest categories into one category (e.g. low CMAP post-survey) and the two highest categories into the other (e.g. high CMAP post-survey). In addition, for each linear regression model, residual plots and Cook’s distance were examined (Cook and Weisberg 1999) to identify outliers and influential points that might affect the findings.

Findings

For the first research question, results of three binomial logistic regressions on the coded variables of replacement, amplification, and transformation revealed that there was a significant effect of the performance task upon whether it was applied in a transformative manner, but that no such overall effect existed for amplification and replacement.

All performance tasks yielded amplification codes with a high frequency and in a uniform manner, with 92–94 % of all participants exhibiting amplification between performance tasks. For replacement, Cloud (31 %) was applied more often than CMAP (27 %), CMS (19 %), or DS (0 %). However there was no statistical difference between any of the tasks after correcting for multiple comparisons. (p > .05) in spite of DS never being used in a replacement manner (Fig. 1).

Fig. 1
figure 1

Proportion of pre-service teachers’ RAT reflection by performance task

Full size image

Examining the data descriptively, the pre-service teachers generally exhibited a high level of amplification in how they applied technology in their thinking and rarely referred to technology use that did not show some clear benefits in their classrooms (i.e. replacement). We take this to be a positive indicator of our course’s value.

With transformation, on the other hand, means varied from medium to low application. Significant differences between performance tasks revealed that the Cloud and CMS performance tasks (with means of .55 and .52, respectively) were each significantly higher in instances of transformation than were the CMAP and DS performance tasks (with means of .04 and .03; p = .02 for both Cloud against CMAP and DS performance; p = .04 for CMS against CMAP; and p = .03 for CMS against DS).

For the second research question, results of the regressions are summarized in Table 3. Because the results were similar between the linear regression, and the exact logistic regression, the results presented in Table 3 and below are from the linear regression model. Additionally, while Cook’s distance and examination of the residual plot showed influential points and outliers, their exclusion from the model did not change the findings (except where noted) and all students were left in the analysis.

Table 3 ANOVA table for the regressions examining whether the way students think about technology integration in the classroom (replacement, amplification, and transformation) impacts their self-assessed competence in technology integration (post-survey)
Full size table

Prior to fitting the linear regression models, we examined the correlations between our predictors to assess the potential for multicollinearity. The largest, in magnitude, point-biserial correlation was between those that used replacement and transformation on the cloud task (r = −0.47). Our findings did not change based on the exclusion of either predictor from our model. Therefore, models were fit and examined with all predictors present.

Though pre-service teachers generally reported a high-level of competence on general items (average of 4.3 out of 5), there was no relationship between a participant’s general competence score and their amplification, replacement, or transformation scores for reflections (p >.05 for all comparisons). This means that overall student competence did not reflect any particular way of thinking about technology integration (replacement, amplification, or transformation).

For the CMAP post-survey, there was not enough variability associated with amplification and transformation to include these predictors in the model as nearly everyone used amplification (all but one) and nearly everyone did not use transformation (all but one). Therefore, the CMAP post-survey was regressed only on replacement and no association between the use of replacement and CMAP post-survey score was found (p >.05).

For the CMS post-survey, there was again not enough variability associated with amplification, so CMAP was regressed onto replacement and transformation only. There was no relationship between transformation in the CMAP post-survey score (p >.05), however, there was a potential relationship between the CMAP post-survey score and replacement (p = .055). This relationship was not robust to the more sophisticated, and appropriate, statistical techniques (i.e. the exact logistic regression model) and when influential points were removed, this marginal relationship went way. This moderate relationship can be seen as a result of the violation of the model assumptions described above and it can be concluded that there was not a relationship between the CMAP post-survey score and replacement.

For the Cloud post-survey there was again no variability associated with amplification, and the Cloud post-survey was regressed onto replacement and transformation. No relationship between the Cloud post-survey and replacement or transformation were found (p >.05).

There was not enough variability in the predictors to run the DS model.

Discussion

Though it is not the purpose of this study to provide a complete view or guide for technology integration, these findings have implications for how we train teachers and prepare them to apply technology competencies in their classrooms. These findings extend understanding of how pre-service teachers use technology versus merely whether or not they are using technologies (Al-Awidi and Alghazo 2012; Jenkins et al. 2006). Results from the first research question suggest that the types of performance tasks we used only had an impact on how pre-service teachers applied their understanding of technology integration in their educational contexts if our goal was for pre-service teachers to think transformatively about their practice. On the other hand, if we only wanted pre-service teachers to apply their technology competencies to replace or amplify existing practice, then the performance task that we used to teach this did not seem to matter. This means that if we are only seeking to replace or amplify existing practice, then the types of performance tasks that we utilize in our teacher training programs may not matter. We may be able to prepare teachers to use a technology instead of a more traditional method or to think about the efficiencies of using certain technologies in the classroom with little thought about the nature of the technologies we are using. However, if our goal is to empower pre-service teachers to utilize technology in their practice in transformative or disruptive ways, then the types of technologies and tasks we employ with pre-service teachers become increasingly important.

To illustrate, another prominent model of technology integration focuses on first- and second-order change with accompanying barriers to effective technology integration for each (Ertmer 1999, 2005). In this model, first-order change means that technology is applied in a way that is consistent with existing norms and patterns of behavior (i.e. replacement and amplification), while second-order change means that technology is applied in a way that fundamentally changes those norms (i.e. transformation). Based on our findings, it seems that second-order change is also influenced by the nature of the tasks and technologies that we employ with pre-service teachers.

In discussions of these two orders of change, barriers to first-order adoption are generally categorized as extrinsic to the adoptee (e.g., lack of resources, adequate training), while barriers to second-order adoption are categorized as intrinsic (e.g., pedagogical beliefs, attitudes). However, from our findings it seems that there is an additional barrier to second-order change that is embedded in the technology and learning experience itself. That is, some technologies and tasks lend themselves more to thinking transformatively (i.e. second-order adoption) than others, which means that technology integration should be seen as a three-way, negotiated relationship between the intrinsic factors of adoptees, the extrinsic factors of their schools and social contexts, and also the nature of the technologies they employ (cf. Fig. 2).

Fig. 2
figure 2

Transformational technology use as a negotiated outcome of intrinsic factors, extrinsic factors, and the type of technology used

Full size image

In our study design, it is difficult to separate the performance tasks from the technologies used to perform them, but it is interesting that the two least transformative tasks in our study (i.e. concept mapping and digital storytelling) are widely used amongst teacher education programs and were implemented in a manner that allowed for student autonomy (i.e. pre-service teachers were given options of which applications to use to complete the tasks). This suggests that these commonly used tasks may not lend themselves to helping pre-service teachers apply technology in second-order ways and that, more generally, the types of tasks and technologies we use in teacher preparation may either improve or negate pre-service teachers’ abilities to make the transition to second-order technology integration.

Results from our second research question are also important for teacher education, because we were unable to find any relationship between self-assessed technology competence on common indicators (e.g., ISTE standards) and pre-service teachers’ critical thinking about technology integration. In our study, this was true for both general competence and competence in technology-specific performance tasks (e.g., CMS). This means that pre-service teachers who might self-assess as being proficient in a variety of technology competencies may actually only exhibit replacement views of technology integration that do not emphasize amplifying or transforming existing practice for the better but that they may be assessing themselves on mere facility with a tool and not meaningful application of that tool for teaching and learning. So, though pre-service teachers may exhibit confidence in their abilities to integrate technology into teaching, they may actually only do so in ways that have no intended impact on student learning and classroom culture (i.e. replacement).

These findings suggest that if teacher education programs value the types of meaningful technology integration practices that may be categorized as amplification and transformation, then both self-assessed competency on specific technologies and general competency on adopted standards (e.g., NETS) are not sufficient to ascertain this. This is likely because the language used in general competency items are rarely connected to actual outcomes that have a positive effect on student learning and classroom culture, which means that pre-service teachers may merely be self-assessing on their technical fluency (e.g., knowing which buttons to press) or responding with general technophilic attitudes. Disconnected from specific positive classroom outcomes and goals, however, such competencies and attitudes may never move beyond mere replacement of existing practice with a neutral, albeit technology-based, alternative. Practitioner notes are provided for reference purposes (Fig. 3).

Fig. 3
figure 3

Practitioner notes

Full size image

Conclusion

By exploring student reflections on performance tasks and self-assessment on technology competency within a teacher preparation technology integration course, this study has attempted to understand (1) relationships between technology-focused performance tasks and pre-service teachers’ critical thinking of technology integration; and (2) relationships between how pre-service teachers are critically thinking about technology integration and their self-assessed competence in technology integration. Results from this study suggest that careful selection of technologies used in performance tasks for teacher education programs are important to support transformative classroom uses of technology and that the manner in which students self-assess on technology competency does not reflect critical thinking about meaningful applications of technology in educational contexts. We suggest that teacher education programs carefully consider how they expect students to apply technology literacies in classroom contexts (e.g., in transformative ways) and strive to utilize technology suites in their learning experiences that support these specific critical thinking patterns. We also suggest that self-assessment on technology competency should be intertwined with reflective practices on how technology might be applied in classroom settings and with what outcomes. Otherwise, it seems that technology integration courses might effectively teach pre-service teachers how to use technology from a technical perspective but not how to think about technology integration in ways that improve, transform, or meaningfully disrupt existing practice.